Tag Archives: processes

Why “Patch Tuesday” is only every four weeks – or never

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today, this tweet caught my attention.

Patch management is currently a hot topic, primarily because of the latest ransomware attacks.

After appearance of WannaCry, one of my older blog posts got unfamiliar attention: WSUS on Windows 2012 (R2) and KB3159706 – WSUS console fails to connect. Why? My guess: Many admins started updating their Windows servers after appearance of WannaCry. Nearly a year after Microsoft has published KB3159706, their WSUS servers ran into this issue.

The truth about patch management

I know many enterprises, that patch their Windows clients and servers only every four or eight weeks, mostly during a maintenance window. Some of them do this, because their change processes require the deployment and test of updates in a test environment. But some of them are simply to lazy to install updates more frequent. So they simply approve all needed updates every four or eight weeks, push them to their servers, and reboot them.

Trond mentioned golden images and templates in his blog posts. I strongly agree to what he wrote, because this is something I see quite often: You deploy a server from a template, and the newly deployed server has to install 172 updates. This is because the template was never updated since creation. But I also know companies that don’t use templates, or goldes master images. They simply create a new VM, mount an ISO file, and install the server from scratch. And because it’s urgent, the server is not patched when it goes into production.

Sorry, but that’s the truth about patch management: Either it is made irregular, made in too long intervals, or not made at all.

Change Management from hell

Frameworks, such as ITIL, play also their part in this tragedy. Applying change management processes to something like patch managent prevents companies to respond quickly to threats. If your change management process prevents you from deploying critical security patches ASAP, you have a problem –  a problem with your change management process.

If your change management process requires the deployment from patches in a test environment, you should change your change mangement process. What is the bigger risk? Deploying a faulty patch, or being the victim of an upcoming ransomware attack?

Microsoft Windows Server Update Service (WSUS) offers a way to automatically approve patches. This is something you want! You want to automatically approve critical security patches. And you also want that your servers automatically install these updates, and restart if necessary. If you can’t restart servers automatically when required, you need short maintenance windows every week to reboot these servers. If this is not possible at all, you have a problem with your infrastructure design. And this does not only apply to Microsoft updates. This applies to ALL systems in your environment. VMware ESXi hosts with uptimes > 100 days are not a sign of stability. It’s a sign of missing patches.

Validated environments are ransomwares best friends

This is another topic I meet regularly: Validated environments. An environmentsthat was installed with specific applications, in a specifig setup. This setup was tested according to a checklist, and it’s function was documented. At the end of this process, you have a validated environments and most vendors doesn’t support changes to this environments without a new validation process. Sorry, but this is pain in the rear! If you can’t update such an environment, place it behind a firewall, disconnect it from your network, and prohibit the use of removable media such as USB sticks. Do not allow this environment to be Ground Zero for a ransomware attack.

I know many environments with Windows 2000, XP, 2003, or even older stuff, that is used to run production facilities, test stands, or machinery. Partially, the software/ hardware vendor is no longer existing, thus making the application, that is needed to keep the machinery running, another security risk.

Patch quick, patch often

IT departments should install patches more often, and short after the release. The risk of deploying a faulty patch is lower than the risk of being hit by a security attack. Especially when we are talking about critical security patches.

IT departments should focus on the value that they deliver to the business. IT services that are down due to a security attack can’t deliver any value. Security breaches in general, are bad for reputation and revenue. If your customers and users complain about frequent maintenance windows due to critical security patches, you should improve your communication about why this is important.

Famous last words

I don’t install Microsoft patches instantly. Some years ago, Microsoft has published a patch that causes problems. Imagine, that a patch would cause our users can’t print?! That would be bad!

We don’t have time to install updates more often. We have to work off tickets.

We don’t have to automate our server deployment. We deploy only x servers a week/ month/ year.

We have a firewall from $VENDOR.

Industrialize your IT – after you have done your homework

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today a tweet from Keith Townsend (@CTOAdvisor) has caught my attention:

Keith wrote a nice blog post and I really recommend to read it. His point is, that automation enables business agility.

The point of automation is to enable business agility. Business agility isn’t achieved by automating inefficient processes. The start of an IT automation project begins by examining existing processes and eliminating inefficiency.

Agility is the ability of an organization to act flexible, proactive, adaptable and with initiative in times of change and uncertainty. Can IT automation enable business agility? No, I don’t think that IT automation can enable business agility.

It’s all about processes

Business processes are the key to business agility. Keith wrote, that

Business processes are very organic systems that grow and evolve as a company’s business and culture matures.

and I’m totally fine with this statement. Processes tend to optimize themselves, if you let them. In cases where consultants are hired to improve faulty processes, the processes are often not the problem. The people who run them are often the problem.

Before you should think about IT automation, you should optimize your processes. Keith has a similar view:

Automation comes after you’ve established a repeatable process that has few if any points of contention.

Lean processes are the key

A business process has to deliver value. That’s quite clear. But a process has to deliver value and it has to be lean. Otherwise the business would lose its agility.

What are the characteristics of a lean process?

  • It delivers value
  • It respects and involves the people who run the process
  • It’s steamlined and free of waste

Streamlining processes and avoiding waste are the anchors for IT automation! But you should only automate what you have fully understood. Keith mentioned a nice example:

Money and time are better spent implementing an IP management system and continuing the manual processes. The problem wasn’t self-provisioning of VM’s but waiting for IP address assignment.

This example clearly shows that it’s mandatory to check each and every process for potential waste and pain points. Only after this step, you will be able to design a new, lean process.

IT automation can’t enable business agility. But lean business processes can do. IT automation can help you to streamline your business processes.

Time is changing

At the end I would like to refer to an important statement in Keiths blog post:

All of this goes back to my drumbeat that IT infrastructure practitioners acquire business alongside their technical skill set.

100% agree! The time of the people who have been sitting in dark basements, with their “There are 10 types of people” shirts and the opinion, that IT is the navel of the world, comes to an end.