Category Archives: Operations

Security: If it doesn’t hurt, you’re doing it wrong!

The Informationsverbund Berlin-Bonn (IVBB), the secure network of the german government , was breached by an unknown hacker group. Okay, a secure government network might be a worthy target for an attack, but your network not, right? Do you use the same password for multiple accounts? There were multiple massive data breaches in the past. Have you ever checked if your data were also compromised? I can recommend haveibeenpwned.com. If you want to have some fun, scan GitHub for -----BEGIN RSA PRIVATE KEY-----. Do you use a full disk encryption on your laptop or PC? Do you sign and/ or encrypt emails using S/MIME or PGP? Do you use different passwords for different services? Do you use 2FA/ MFA to secury importan services? Do you never work with admin privileges when doing normal office tasks? No? Why? Because it’s uncomfortable to do it right, isn’t it?

My focus is on infrastructure, and I’m trying to educate my customers that hey have to take care about security. It’s not the missing dedicated management network, or the usage of self signed certificates that makes an infrastructre unsecure. Mostly it’s the missing user management, the same password for different admin users, doing office work with admin privileges, or missing security patches because of “never touch a running system”, or “don’t ruin my uptime”. I don’t khow how often I heard the story of ransomware attacks, that were caused by admins opening email attachments with admin privileges…

My theory

Security must approach infinitely near the point, where it becomes unusable.

Security is nothing you can take care about later. It has to be part of the design. It has to be part of the processes. Most security incidents doesn’t happen because of 0-day exploits. It’s because of default passwords for admin accounts, missing security patches, and because of lazy admins or developers.

Don’t be lazy. Do it right. Even if it’s uncomfortable.

Why “Patch Tuesday” is only every four weeks – or never

Today, this tweet caught my attention.

Patch management is currently a hot topic, primarily because of the latest ransomware attacks.

After appearance of WannaCry, one of my older blog posts got unfamiliar attention: WSUS on Windows 2012 (R2) and KB3159706 – WSUS console fails to connect. Why? My guess: Many admins started updating their Windows servers after appearance of WannaCry. Nearly a year after Microsoft has published KB3159706, their WSUS servers ran into this issue.

The truth about patch management

I know many enterprises, that patch their Windows clients and servers only every four or eight weeks, mostly during a maintenance window. Some of them do this, because their change processes require the deployment and test of updates in a test environment. But some of them are simply to lazy to install updates more frequent. So they simply approve all needed updates every four or eight weeks, push them to their servers, and reboot them.

Trond mentioned golden images and templates in his blog posts. I strongly agree to what he wrote, because this is something I see quite often: You deploy a server from a template, and the newly deployed server has to install 172 updates. This is because the template was never updated since creation. But I also know companies that don’t use templates, or goldes master images. They simply create a new VM, mount an ISO file, and install the server from scratch. And because it’s urgent, the server is not patched when it goes into production.

Sorry, but that’s the truth about patch management: Either it is made irregular, made in too long intervals, or not made at all.

Change Management from hell

Frameworks, such as ITIL, play also their part in this tragedy. Applying change management processes to something like patch managent prevents companies to respond quickly to threats. If your change management process prevents you from deploying critical security patches ASAP, you have a problem –  a problem with your change management process.

If your change management process requires the deployment from patches in a test environment, you should change your change mangement process. What is the bigger risk? Deploying a faulty patch, or being the victim of an upcoming ransomware attack?

Microsoft Windows Server Update Service (WSUS) offers a way to automatically approve patches. This is something you want! You want to automatically approve critical security patches. And you also want that your servers automatically install these updates, and restart if necessary. If you can’t restart servers automatically when required, you need short maintenance windows every week to reboot these servers. If this is not possible at all, you have a problem with your infrastructure design. And this does not only apply to Microsoft updates. This applies to ALL systems in your environment. VMware ESXi hosts with uptimes > 100 days are not a sign of stability. It’s a sign of missing patches.

Validated environments are ransomwares best friends

This is another topic I meet regularly: Validated environments. An environmentsthat was installed with specific applications, in a specifig setup. This setup was tested according to a checklist, and it’s function was documented. At the end of this process, you have a validated environments and most vendors doesn’t support changes to this environments without a new validation process. Sorry, but this is pain in the rear! If you can’t update such an environment, place it behind a firewall, disconnect it from your network, and prohibit the use of removable media such as USB sticks. Do not allow this environment to be Ground Zero for a ransomware attack.

I know many environments with Windows 2000, XP, 2003, or even older stuff, that is used to run production facilities, test stands, or machinery. Partially, the software/ hardware vendor is no longer existing, thus making the application, that is needed to keep the machinery running, another security risk.

Patch quick, patch often

IT departments should install patches more often, and short after the release. The risk of deploying a faulty patch is lower than the risk of being hit by a security attack. Especially when we are talking about critical security patches.

IT departments should focus on the value that they deliver to the business. IT services that are down due to a security attack can’t deliver any value. Security breaches in general, are bad for reputation and revenue. If your customers and users complain about frequent maintenance windows due to critical security patches, you should improve your communication about why this is important.

Famous last words

I don’t install Microsoft patches instantly. Some years ago, Microsoft has published a patch that causes problems. Imagine, that a patch would cause our users can’t print?! That would be bad!

We don’t have time to install updates more often. We have to work off tickets.

We don’t have to automate our server deployment. We deploy only x servers a week/ month/ year.

We have a firewall from $VENDOR.