Tag Archives: blog

CloudFlare API v4 and Fail2ban: Fixing the unban action

In January 2017, I wrote an article about how to protect your WordPress blog using the WP Fail2Ban plugin, fail2ban on your Linux/ FreeBSD host, and CloudFlare. Back then, the fail2ban was using the CloudFlare API V1, which was already deprecated since November 2016.

Free-Photos/ pixabay.com/ Creative Commons CC0

Although the actions were updated later to use CloudFlare API V4, I still had problems with the unbaning of IP addresses. IP addresses were banned, but the unban action failed. 

This is the unban action, which is included in fail2ban (taken from fail2ban-0.10.3.1 which is shipped with FreeBSD 11.1-RELEASE-p10):

And this is the unban action, which finally solved this issue:

I found the solution at serverfault.com. The only difference is an additional tr -d '\n'  in the last line of the statement. Kudos to Jake for fixing this!

To prevent the action file to being overwritten, you should copy the original cloudflare.conf  located in the  action.d  directory, e.g. to mycloudflare.conf , and use the copied action file in your fail definition.

Using WP fail2ban with the CloudFlare API to protect your website

The downside of using WordPress is that many people use it. That makes WordPress a perfect target for attacks. I have some trouble with attacks, and one of the consequences is, that my web server crashes under load. The easiest way to solve this issue would be to ban those IP addresses. I use Fail2ban to protect some other services. So the idea of using Fail2ban to ban IP addresses, that are used for attacks, was obvious.

From the Fail2ban wiki:

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs — too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc).

That works for services, like IMAP, very good. Unfortunately, this does not work out of the box for WordPress. But adding the WordPress plugin WP fail2ban brings us closer to the solution. For performance and security reasons, vcloudnine.de can only be accessed through a content delivery network (CDN), in this case CloudFlare. Because CloudFlare acts as a reverse proxy, I can not see “the real” IP address. Furthermore, I can not log the IP addresses because of the German data protection law. This makes the Fail2ban and the WordPress Fail2ban plugin nearly useless, because all I would ban with iptables, would be the CloudFlare CND IP ranges. But CloudFlare offers a firewall service. CloudFlare would be the right place to block IP addresses.

So, how can I stick Fail2ban, the WP Fail2ban plugin and CloudFlares firewall service together?

APIs FTW!

APIs are the solution for nearly every problem. Like others, CloudFlare offers an API that can be used to automate tasks. In this case, I use the API to add entries to the CloudFlare firewall. Or honestly: Someone wrote a Fail2ban action that do this for me.

First of all, you have to install the WP Fail2ban plugin. That is easy. Simply install the plugin. Then copy the wordpress-hard.conf from the plugin directory to the filters.d directory of Fail2ban.

Then edit the /etc/fail2ban/jail.conf and add the necessary entries for WordPress.

Please note, that in my case, the plugin logs to /var/log/messages. The action is “cloudflare”. To allow Fail2ban to work with the CloudFlare API, you need the CloudFlare API Key. This key is uniqe for every CloudFlare account. You can get this key from you CloudFlare user profile. Go to the user settings and scroll down.

Cloudflare Global API Key

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Open the /etc/fail2ban/action.d/cloudflare.conf and scroll to the end of the file. Add the token and your CloudFlare login name (e-mail address) to the file.

Last step is to tell the WP Fail2ban plugin which IPs should be trusted. We have to add subnets of the CloudFlare CDN. Edit you wp-config.php and add this line at the end:

The reason for this can be found in the FAQ of the WP Fail2ban plugin. The IP ranges used by CloudFlare can be found at CloudFlare.

Does it work?

Seems so… This is an example from /var/log/messages.

And this is a screenshot from the CloudFlare firewall section.

Cloudflare Firewall Blocked Websites

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Another short test with curl has also worked. I will monitor the firewall section of CloudFlare. Let’s see who’s added next…

Important note for those, who use SELinux: Make sure that you install the policycoreutils-python package, and create a custom policy for Fail2Ban!

A strong indicator are errors like this in /var/log/messages:

You will find corresponding audit messages in the /var/log/audit.log:

Make sure that you create a custom policy for Fail2Ban, and that you load the policy.

The Linux OOM killer strikes again

As a frequent reader of my blog, you might have noticed that vcloudnine.de was unavailable from time to time. Reason for this was, that my server was running out of memory at night.

Running out of memory is bad for system uptime. Sometimes you have to sacrifice someone to help others.

It is the job of the linux ‘oom killer’ to sacrifice one or more processes in order to free up memory for the system when all else fails.

Source: OOM Killer – linux-mm.org

The OOM killer selects the process, that frees up the most memory, and that is the least important to the system. Unfortunately, in my case it is Apache or MySQL. On the other hand: Killing these processes have never brought back the system to life. But that is another story. Something has consumed so much memory at night, that the OOM killer had to start its deadly work.

Checking the logs

The OOM has started its work at ~5am, and it killed the httpd (Apache).

While checking the Apache error_log, this log entry caught my attention.

The next stop was the Apache access_log. At the same time as in the error_log, the Apache logged a POST request wp-login.php in the access_log.

And there were a lot more attempts… I did a short check of older log files. It was not the first OOM killer event, and the log entries were smoking gun. Especially the POST for wp-login.php.

The number below the command is the number of the POST requests logged in the access_log. The current access_log starts on Jan 08 2017. And since start, there are alreay 876 POST requests to wp-login.php. Looks like a brute force attack.

So there is nothing wrong with the sever setup, it simply breaks down during a brute force attack.

2016 – How did it go?

The year 2016 is coming to an end (thank god…). 2016 was a difficult year. One of my goals for 2016 was to write more PowerShell code and to learn Python. I missed both goals. But hey, at least I cleaned up my Git account.I renewed my VCP by passing VCP6 in the first attempt, and I took the VCP7-DTM beta exam (no results yet). Since I am not the guy who attends conferences, I was not attending VMworld in Barcelona or HPE Discover in London. In fact: I was no at no conference or vendor roadshow, because I had an ass full of work.

Let’s look at some good things.

My blog got round about 1/3 more page views than in 2015. That way my goal for 2016. I’m very happy about that. Thank you!

I definitely missed my goal to write more blog posts. As you can see, I have written a bit more than last year. But as long not as much as I wanted. I always have enough posts in my draft folder. But I’m constantly wondering if it is worth writing about this or that topic. I usually write about things that happen to me in the job. No product reviews, nothing about architecture or C-level things, less about development, design or how something works. Maybe that’s my problem. Let’s see if I can change this in 2017.

I constantly optimized vcloudnine.de in 2016. The result is quite good. Thanks to Eric Siebert for his input.

vcloudnine_performance_gtmetrix

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

These are some of my most popular blog posts (according to Google). Fun fact: Only one is related to VMware and today obsolete.

  1. WSUS on Windows 2012 (R2) and KB3159706 – WSUS console fails to connect
  2. Load VMware PowerCLI snap-in automatically in PowerShell ISE
  3. Reset the HP iLO Administrator password with hponcfg on ESXi
  4. Users on Exchange 2013 can’t open public folders or shared mailboxes on an Exchange 2007/ 2010
  5. Replace HP iLO security certificates

I don’t want to make predictions for 2017. Let’s see how this blog will develop.

How to dramatically improve website load times

Over the last weeks, I’ve tried to improve the performance of my blog. The side was very slow and the page load times varied between 5 and 10 seconds. Much too long! I’ve reduced time consuming plugins, checked the size of pictures, checked CSS and HTML for misconfiguration/ slow clode and tuned the database. The page load times have not really improved.

Yesterday, I checked the httpd.conf on my webserver and found a little typo (accidentally commented line). After a restart of the Apache webserver, the page load times have dramatically improved (down to 2 – 3 seconds). What had happened?

HTTP keep-alive

HTTP keep-alive, sometimes also called “HTTP persistent connection”, was designed to transfer multiple HTTP requests and responses over a single TCP connection. This is much better as opening a new connection for every single request/ response pair. The benefits of HTTP keep-alive are:

  • lower CPU usage
  • lower memory usage
  • reduced latency due to reduced requests/ handshaking

These benefits are even more important, if you use HTTPS connections (and vcloudnine.de is HTTPS-only…), because each new HTTP connection needs much more CPU time and round-trips compared to an unsecure HTTP connection. This little picture clarifies the differences.

HTTP_persistent_connection

Wikipedia/ wikipedia.org/ Public domain image resources

If you’re using Apache, you can enable HTTP keep-alive with a single line in the httpd.conf.

Further information can be found in the documentation of Apache (Apache webserver 2.2 and 2.4).

Top vBlog 2015: vcloudnine.de placed on #133

What a great show by Eric Siebert, David Davis, Simon Seagrave and their special guests Scott Davis from Infinio and John Troyer from TechReckoning! If you missed it, watch the recording!

First, I want to thank Eric for his work. If you read tweets like these, you will get a bad conscience.

This is the seventh year that Eric has organized and conducted the annual Top vBlog contest. He put so much work into this contest and this should be be recognized. I also like to thank the sponsor Infinio for supporting this contest.

2015 was the second year in which I partaken the Top vBlog contest, but vcloudnine.de was on the voting list for the fist time. I started this blog in 2014 so I was on the “Newcomer” list of the contest. I’m always trying to create valuable content. This isn’t easy and often a draft is thrown to trash. I hope vcloudnine.de was chosen because of valuable content and not because voters like me. ;) This year’s Top vBlog poll brought us a lot changes. Eric has leaked some details in a blog post short before the announcement:

  • 60% more votes than 2014
  • 30% more blogs on the voting list
  • 7 changes in the top 10
  • 4 blogs in the top 25 that were not in there last year
  • 2 blogs in the top 25 that were newcomers this year
  • 1 blog new to the top 10

Congratulations to…

“Out of competition”: Duncan Epping (VCDX #007) and yellow-bricks.com for “defending” 1st place. Does anyone doubt it? Not really, right? ;) Congrats Duncan!

I am particularly happy for Derek Seaman (VCDX #125). His blog is a gold mine of content and he’s generating more and more (read his vSphere 6.0 series). Congrats Derek, #7 is totally deserved!

Congrats to Melissa Palmer for winning the “Best new blog” category. Keep on blogging, Melissa!

Congrats to Chris Wahl (VCDX #104) for winning the “Best indipendant blogger” category. Reading his blog is always a pleasure!

Also well deserved: Brian Madden has won the “Best VDI blog” category. His blog is an awesome resource if you deal with VDI!<in/p>

Honestly: That William Lam has won in the category “Best scriping blog” and Cormac Hogan in the “Best storage blog” category was no suprise for me. Totally deserved, guys!

I am very happy to see that some bloggers that I have on my reading list, ranked up in the list. You can find the results of the Top vBlog 2015 contest here. Congrats to all participant and thanks again to Eric Siebert!

To make the long story short…

I’m happy and disappointed at the same time. vcloudnine.de landed on place 133. Not the worst placement for a new blog. But I have missed my personal goal to be placed under the top 100. I’d like to thank all, that have voted for vcloudnine.de. This is a great motivation to work harder and to create more valuable content. Thank you all!

Top vBlog 2015 Contest has started

If you are a frequent reader of virtualization blogs, then you may have heard about the vLaunchPad. It lists hundreds of VMware & virtualization blogs, as well as links to resources and other material. The vLaunchPad is managed by Eric Siebert (@ericsiebertvsphere-land.com) and he organizes year for year the annual Top vBlog voting contest. This year the Top vBlog contest is sponsored by Infinio.

In the 2014 voting my “old” blog was voted on place 292 of 320. I should mention that blazilla.de had only german-language content. In a community, where english is the predominating content language, this result may not surprise. If you are interested in last year’s results, you can find them here. In 2014 I have started vcloudnine.de, but I didn’t nominated it for the 2014 voting. Instead, I nominated blazilla.de for the Top vBlog 2014 contest. This year the tables turned and I have nominated vcloudnine.de for the categories:

  • Best new blog (Blog started in 2014), and
  • Best independent blogger (Can’t work for VMware or a hardware/software vendor)

As always all blogs that are listed on the vLaunchPad are included in the general voting. I don’t have a goal for the voting, but a place between #49 and #100 would be nice. ;)

Some short sentences about vcloudnine.de:

vcloudnine.de is the personal blog of Patrick Terlisten. The site has a strong focus on virtualization, storage, networking and IT infrastructure in general. The main driver of this blog is to share knowledge and write about topics, that I think is worth mentioning. The views expressed anywhere on this site are mine and not the opinions and views of my employer or a vendor.

The predominating topics on vcloudnine.de are VMware, HP Storage, HP Data Protector, networking in general and Microsoft Exchange.

Andreas Lesslhumer (@lessi001running-system.com) has created a nice statistic for 2014: Virtualization blogs 2014 by numbers. The statistic is based on the blogs, that are listed on the vLaunchPad. vcloudnine.de was one of the 28 blogs, that published more than 100 blog posts in 2014. In 2015 I published 13 blog posts so far. But to be honest: It’s not about the number of posts you publish – the content matters! So if you vote for a blog, vote for the content, not the number of published posts or the author.

Check out the Top vBlog 2015 landing page and don’t forget to vote for your favorite blogs! The voting will start soon!

End-of-year review 2014

The days at the year’s end are usually the time to look back and to draw a resume.

I blogged on blazilla.de for nearly 7 years, but in December 2013 I decided that it was time for a new beginning: New blog platform, new content, another language, but the same topics. In January 2014 I started vcloudnine.de.

Some statistics

Until today, I wrote about 100 blog posts. According to Google Analytics, these are my Top 10 most visited blog posts for 2014:

  1. VMware ESXi 5.5 host doesn’t mount VMFS 5 datastore
  2. Load VMware PowerCLI snap-in automatically in PowerShell ISE
  3. Event ID 4625 – Failure Reason: Domain sid inconsistent
  4. Windows Server 2012 Cluster with VMware vSphere 5.1/ 5.5
  5. Configuring HP StoreOnce VSA and HP Data Protector for HP StoreOnce Catalyst
  6. Deploying HP StoreOnce VSA with HP Data Protector – Part III
  7. Exchange 2013: Event ID 2937 MSExchange ADAccess after public folder migration
  8. Some thoughts about HP 3PAR Adaptive Optimization
  9. Users on Exchange 2013 can’t open public folders or shared mailboxes on an Exchange 2007/ 2010
  10. Trouble with Broadcom NetXtreme II and VMware ESXi

Since January vcloudnine.de was visited about 90.000 times. I think this isn’t a bad score for a new blog and I hope that it will be much more in 2015. Most visitors came from the US. These are the Top 5 countries:

  1. United States
  2. Germany
  3. United Kingdom
  4. France
  5. Netherlands

Most of them used Google Chrome to access vcloudnine.de (this has really suprised me…). And someone regulary visited vcloudnine.de with a BlackBerry. These are the Top 5 used browser to access vcloudnine.de:

  1. Chrome
  2. Firefox
  3. Internet Explorer
  4. Safari (including in-app)
  5. Opera

I’m curious how vcloudnine.de will develop in the future.

Final words for 2014

This blog deals with my work. Because of this, the final words for 2014 are mainly written from the job perspective. 2014 was also the first full year back in the office, after working for more than four years mainly for a enterprise customer as an Infrastructure Architect. 2013 was a transitional year, but 2014 has surprised and disappointed me in different ways. From the career perspective, I can’t say much good about 2014. I hope 2015 will be better.

On the other hand, 2014 was a successful year: I finished my a degree at the FernUniversität Hagen. I’m a bit proud of this. The majority doesn’t finish the study at the FernUniversität Hagen. Many give up because of the double burden of study, job and family. 2014 was also the year of recertification. Between September and October I passed five exams and recertified my HP Master ASE, my DataCore DCIE, VMware VCP and I also achieved my first Juniper certification. Since April 2014 I’m a vExpert. This award has made me incredibly proud. I got so many positive feedback because of this and it has motivated me in many ways.

I look forward to your feedback. Drop me an email or a comment. I wish you and your family all the best. Enjoy the holidays and we will see us again in 2015.

Yet another blog…

… about virtualization, storage, networking and IT infrastructure in general. Is that really necessary? Yes! To be honest, this is my second attempt. I’m blogging for nearly seven years on blazilla.de. It’s a well-frequented german blog with a strong focus on HP, Storage, DataCore, VMware and other technical topics. Blogging about technology you’re working with is a good way to share, recap and internalize knowledge. Sharing knowledge is a main driver of the scientific and IT community. If you’re writing about something and you share it, you can give something back to the community. So why the hell do I start a second blog? Because I can. ;) blazilla.de is a blog that is mainly focused on the german IT community. This blog is my attempt to get more visible to the international community and give something back to it. I’m not a native English speaker. But I will try my best. I hope you enjoy this blog. :)