The Linux OOM killer strikes again

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

As a frequent reader of my blog, you might have noticed that vcloudnine.de was unavailable from time to time. Reason for this was, that my server was running out of memory at night.

Running out of memory is bad for system uptime. Sometimes you have to sacrifice someone to help others.

It is the job of the linux ‘oom killer’ to sacrifice one or more processes in order to free up memory for the system when all else fails.

Source: OOM Killer – linux-mm.org

The OOM killer selects the process, that frees up the most memory, and that is the least important to the system. Unfortunately, in my case it is Apache or MySQL. On the other hand: Killing these processes have never brought back the system to life. But that is another story. Something has consumed so much memory at night, that the OOM killer had to start its deadly work.

Checking the logs

The OOM has started its work at ~5am, and it killed the httpd (Apache).

While checking the Apache error_log, this log entry caught my attention.

The next stop was the Apache access_log. At the same time as in the error_log, the Apache logged a POST request wp-login.php in the access_log.

And there were a lot more attempts… I did a short check of older log files. It was not the first OOM killer event, and the log entries were smoking gun. Especially the POST for wp-login.php.

The number below the command is the number of the POST requests logged in the access_log. The current access_log starts on Jan 08 2017. And since start, there are alreay 876 POST requests to wp-login.php. Looks like a brute force attack.

So there is nothing wrong with the sever setup, it simply breaks down during a brute force attack.

Follow me

Patrick Terlisten

vcloudnine.de is the personal blog of Patrick Terlisten. Patrick has a strong focus on virtualization & cloud solutions, but also storage, networking, and IT infrastructure in general. He is a fan of Lean Management and agile methods, and practices continuous improvement whereever it is possible.

Feel free to follow him on Twitter and/ or leave a comment.
Patrick Terlisten
Follow me

2 thoughts on “The Linux OOM killer strikes again

  1. BerndH

    Hi,
    unfortunately, this is a common issue with the Linux in General. Instead of implementing a clean an proper panic procedure… someone had the Idea to implement the OOM Killer Feature. But talking about the Linux Kernel and Unix Kernels is a different kind of beast.
    So.. to avoid your Problem… give the following Procedure a shot.

    vi /etc/sysctl.conf
    vm.overcommit_memory = 2
    vm.overcommit_ratio = 80

    systctl -p
    or init 6 ;)
    Your problem should be solved, in 99 Percent of all Cases.

    Take Care

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

I accept!