Category Archives: Automation

CloudFlare API v4 and Fail2ban: Fixing the unban action

In January 2017, I wrote an article about how to protect your WordPress blog using the WP Fail2Ban plugin, fail2ban on your Linux/ FreeBSD host, and CloudFlare. Back then, the fail2ban was using the CloudFlare API V1, which was already deprecated since November 2016.

Free-Photos/ pixabay.com/ Creative Commons CC0

Although the actions were updated later to use CloudFlare API V4, I still had problems with the unbaning of IP addresses. IP addresses were banned, but the unban action failed. 

This is the unban action, which is included in fail2ban (taken from fail2ban-0.10.3.1 which is shipped with FreeBSD 11.1-RELEASE-p10):

And this is the unban action, which finally solved this issue:

I found the solution at serverfault.com. The only difference is an additional tr -d '\n'  in the last line of the statement. Kudos to Jake for fixing this!

To prevent the action file to being overwritten, you should copy the original cloudflare.conf  located in the  action.d  directory, e.g. to mycloudflare.conf , and use the copied action file in your fail definition.

Azure PowerShell vs. Azure RM PowerShell

In 2014, Microsoft announced the Azure Preview Portal, which was going GA in December 2015. Since January 8, 2018, the classic Azure Portal is turned off. The “Preview Portal” was more than a facelift. The classic Azure Portal was based on the Service Management mode, often called the “classic deployment model”, whereas the new Azure Portal uses the Resource Manager model. Azure Service Management (ASM) and Azure Resource Management are both deployment models. The Resource Manager model eases the deployment of complex setups by using templates to deploy, update and manage resources within a resource group as a single operation.

Azure PowerShell vs. Azure RM PowerShell

Different deployment models require different tools. Because of this, Microsoft offers two PowerShell modules for Azure. Depending on your deployment type, you have to use the Azure or AzureRM module. Both can be installed directly from the PowerShell Gallery using Install-Module -Name Azure or Install-Module -Name AzureRM .

Connect to Azure

Depending on the used module, the ways to connect to Azure differ.

Module AzureRM

You will notice, that AzureRM sessions does not persist between PowerShell sessions. This behaviour differs from Add-AzureAccount . But you can save and load your AzureRM session once you are connected.

Module Azure

Using WP fail2ban with the CloudFlare API to protect your website

The downside of using WordPress is that many people use it. That makes WordPress a perfect target for attacks. I have some trouble with attacks, and one of the consequences is, that my web server crashes under load. The easiest way to solve this issue would be to ban those IP addresses. I use Fail2ban to protect some other services. So the idea of using Fail2ban to ban IP addresses, that are used for attacks, was obvious.

From the Fail2ban wiki:

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs — too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc).

That works for services, like IMAP, very good. Unfortunately, this does not work out of the box for WordPress. But adding the WordPress plugin WP fail2ban brings us closer to the solution. For performance and security reasons, vcloudnine.de can only be accessed through a content delivery network (CDN), in this case CloudFlare. Because CloudFlare acts as a reverse proxy, I can not see “the real” IP address. Furthermore, I can not log the IP addresses because of the German data protection law. This makes the Fail2ban and the WordPress Fail2ban plugin nearly useless, because all I would ban with iptables, would be the CloudFlare CND IP ranges. But CloudFlare offers a firewall service. CloudFlare would be the right place to block IP addresses.

So, how can I stick Fail2ban, the WP Fail2ban plugin and CloudFlares firewall service together?

APIs FTW!

APIs are the solution for nearly every problem. Like others, CloudFlare offers an API that can be used to automate tasks. In this case, I use the API to add entries to the CloudFlare firewall. Or honestly: Someone wrote a Fail2ban action that do this for me.

First of all, you have to install the WP Fail2ban plugin. That is easy. Simply install the plugin. Then copy the wordpress-hard.conf from the plugin directory to the filters.d directory of Fail2ban.

Then edit the /etc/fail2ban/jail.conf and add the necessary entries for WordPress.

Please note, that in my case, the plugin logs to /var/log/messages. The action is “cloudflare”. To allow Fail2ban to work with the CloudFlare API, you need the CloudFlare API Key. This key is uniqe for every CloudFlare account. You can get this key from you CloudFlare user profile. Go to the user settings and scroll down.

Cloudflare Global API Key

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Open the /etc/fail2ban/action.d/cloudflare.conf and scroll to the end of the file. Add the token and your CloudFlare login name (e-mail address) to the file.

Last step is to tell the WP Fail2ban plugin which IPs should be trusted. We have to add subnets of the CloudFlare CDN. Edit you wp-config.php and add this line at the end:

The reason for this can be found in the FAQ of the WP Fail2ban plugin. The IP ranges used by CloudFlare can be found at CloudFlare.

Does it work?

Seems so… This is an example from /var/log/messages.

And this is a screenshot from the CloudFlare firewall section.

Cloudflare Firewall Blocked Websites

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Another short test with curl has also worked. I will monitor the firewall section of CloudFlare. Let’s see who’s added next…

Important note for those, who use SELinux: Make sure that you install the policycoreutils-python package, and create a custom policy for Fail2Ban!

A strong indicator are errors like this in /var/log/messages:

You will find corresponding audit messages in the /var/log/audit.log:

Make sure that you create a custom policy for Fail2Ban, and that you load the policy.

HPE ProLiant PowerShell SDK

Some days ago, my colleague Claudia and I started to work on a new project: A greenfield deployment consisting of some well known building blocks: HPE ProLiant, HPE MSA, HPE Networking (now Aruba) and VMware vSphere. Nothing new for us, because we did this a couple times together. But this led us to the idea, to automate some tasks. Especially the configuration of the HPE ProLiants: Changing BIOS settings and configuring the iLO.

Do not automate what you have not fully understood

Some of the wisest words I have ever said to a customer. Modifying the BIOS and iLO settings is a well understood task. But if you have to deploy a bunch of ProLiants, this is a monotonous, and therefore error prone process. Perfect for automation!

Scripting Tools for Windows PowerShell

To support the automation of HPE ProLiant deployments, HPE offers the Scripting Tools for Windows PowerShell. HPE offers the PowerShell modules free for charge. There are three different downloads:

  • iLO cmdlets
  • BIOS cmdlets
  • Onboard Administrator (OA) cmdlets

The iLO cmdlets include PowerShell cmdlets to configure and manage iLO on HPE ProLiant G7, Gen8 or Gen9 servers. The BIOS cmdlets does not support G7 servers, so you can only configure and manage legacy and UEFI BIOS for Gen8 (except DL580) and all Gen9 models. The OA cmdlets support the configuration and management of the HPE Onboard Administrator, which is used with HPEs well known ProLiant BL blade servers. The OA cmdlets need at least  OA v3.11, whereby v4.60 is the latest version available.  All you need to get started are

  • Microsoft .NET Framework 4.5, and
  • Windows Management Framework 3.0 or later

If you are using Windows 8 or 10, you already have PowerShell 4 respectively PowerShell 5.

Support for HPE ProLiant Gen9 iLO RESTful API

If you have ever seen a HPE ProLiant Gen9 booting up, you might have noticed the iLO RESTful API icon down right. Depending on the server model, the BIOS cmdlets utilize the ILO4 RESTful API. But the iLO RESTful API ecosystem is it worth to be presented in an own blog post. Stay tuned.

Documentation and examples

HPE offers a simple documentation for the BIOS, iLO and OA cmdlets. You can find the documentation in HPEs Information Library. Documentation is important, but sometimes example code is necessary to quickly ramp up code. Check HPEs PowerShell SDK GitHub repository for examples.

Time to code

I’m keen on it and curious to automate some of my regular deployment tasks with these PowerShell modules. Some of these tasks are always the same:

  • change the power management and other BIOS settings
  • change the network settings of the iLO
  • change the initial password of the iLO administrator account and create additional iLO user accounts

Further automation tasks are not necessarily related to the HPE ProLiant PowerShell SDK, but to PowerShell, respectively VMware PowerCLI. PowerShell is great to automate the different aspects and modules of an infrastructure deployment. You can use it to build your own tool box.

PowerCLI: Get-LunPathState

Careful preparation is a key element to success. If you restart a storage controller, or even the whole storage, you should be very sure that all ESXi hosts have enough paths to every datstore. Sure, you can use the VMware vSphere C# client or the Web Client to check every host and every datastore. But if you have a large cluster with a dozen datastores and some Raw Device Mappings (RDMs), this can take a looooong time. Checking the path state of each LUN is a task, which can be perfectly automated. Get a list of all hosts, loop through every host and every LUN, output a list of all hosts with all LUNs and all paths for each LUN. Sounds easy, right?

For a long time, I used this PowerCLI script for checking the LUN path state. But now I decided to give something back and I tweaked it a bit for my needs.

Feel free to use and/ or modify it.

Certificate-based authentication of Azure Automation accounts

Before you can manage Azure services with Azure Automation, you need to authenticate the Automation account against a subscription. This authentication process is part of each runbook. There are two different ways to authenticate against an Azure subscription:

  • Active Directory user
  • Certificate

If you want to use an Active Directory account, you have to create a credential asset in the Automation account and provide username and password for that Active Directory account. You can retrieve the credentials using the Get-AzureAutomationCredential cmdlet. This cmdlet returns a System.Management.Automation.PSCredential object, which can be used with Add-AzureAccount to connect to a subscription. If you want to use a certificate, you need four assets in the Automation account: A certificate and variables with the certificate name, the subscription ID and the subscription name. The values of these assets can be retrieved with Get-AutomationVariable and Get-AutomationCertificate.

Prerequisites

Before you start, you need a certificate. This certificate can be a self- or a CA-signed certificate. Check this blog post from Alice Waddicor if you want to start with a self-signed certificate. I used a certificate, that was signed by my lab CA.

At a Glance:

  • self- or CA-signed certificate
  • Base64 encoded DER format (file name extension .cer) to upload it as a management certificate
  • PKCS #12 format with private key (file name extension .pfx or .cer) to use it as an asset inside the Automation account

Upload the management certificate

First, you must upload the certificate to the management certificates. Login to Azure and click “Settings”.

creat_automation_account_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click on “Management Certificates”

creat_automation_account_02

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

and select “Upload” at the bottom of the website.

creat_automation_account_03

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Make sure that the certificate has the correct format and file name extension (.cer).

creat_automation_account_04

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Finish the upload dialog. After a few seconds, the certificate should appear in the listing.

creat_automation_account_06

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Create a new Automation account

Now it’s time to create the Automation account. Select “Automation” from the left panel.

creat_automation_account_07

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click on “Create an Automation account”.

creat_automation_account_08

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Give your Automation account a descriptive name and select a region. Please note that an Automation account can manage Azure services from all regions!

creat_automation_account_09

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click on the newly created account and click on “Assets”.

creat_automation_account_10

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Select “Add setting” from the bottom of the website.

creat_automation_account_11

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Add a credential asset by choosing “Add credential” and select “Certificate” as “Credential type”.

creat_automation_account_12

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Enter a descriptive name for the certificate. You should remember this name. You will need it later. Now you have to upload the certificate. The certificate must have the file name extension .pfx or .cer and it must include the private key!

creat_automation_account_13

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Finish the upload of the certificate. Now add three additional assets (variables).

creat_automation_account_14

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Select the name, the value and the type from the  table below. The name of the certificate is the descriptive name, you’ve previously entered when uploading the certificate.

NameValueType
AutomationCertificateNameName of your certificateString
AzureSubscriptionNameName of your subscriptionString
AzureSubscriptionID36 digit ID of the subscriptionString

Done. You’ve uploaded and created all the required certificates and variables.

How to use it

To use the certificate and the variables to connect to an Azure subscription, you have to use the two cmdlets Get-AutomationCertificate and Get-AutomationVariable. I use this code block in my runbooks:

Works like a charm.

Summary

Certificate-based authentication is an easy way to authenticate an Automation account against an Azure subscription. It’s easy to implement and you don’t have to maintain users and passwords. You can use different certificates for different Automation accounts. I really recommend this, especially if you have separate accounts for dev, test and production.

All you need is to upload a certificate as a management certificates, and as a credential asset in the Automation account.  You can use a self- or CA-signed certificate. The subscription ID, the subscription name and the name of the certificate are stored in variables.

At the beginning of each runbook, you have to insert a code block. This code block takes care of authentication.

A brief introduction into Azure Automation

Automation is essential to reduce friction and to streamline operational processes. It’s indispensable when it comes to the automation of manual, error-prone and frequently repeated tasks in a cloud or enterprise environment. Automation is the key to IT industrialization. Azure Automation is used to automate operational processes withing Microsoft Azure.

Automation account

The very first thing you have to create is an Automation account. You can have multiple Automation accounts per subscription. An Automation account allows you so separate automation resources from other Automation accounts. Automation resources are runbooks and assets (credentials, certificates, connection strings, variables, scheudles etc.). So each Automation account has its own set of runbooks and assets. This is perfect to separate production from development. An Automation account is associated with an Azure region, but the Automation account can manage Azure services in all regions.

Runbooks

A runbook is a collection of PowerShell script or PowerShell workflows. You can automate nearly everything with it. If something provides an API, you can use a runbook and PowerShell to automate it. A runbook can run other runbooks, so you can build really complex automation processes. A runbook can access any services that can be accessed by Microsoft Azure, regardless if it’s an internal or external service.

There are three types of runbooks:

  • Graphical runbooks
  • PowerShell Workflow runbooks
  • PowerShell runbooks

Graphical runbooks can be created and maintained with a graphical editor within the Azure portal. Graphical runbooks use PowerShell workflow code, but you can’t directly view oder modify this code. Graphical runbooks are great for customers, that don’t have much automation and/ or PowerShell knowledge. Once you created a graphical runbook with an automation account, you can export and import this runbook into another automation accounts, but you can modify the runbook only with the account which was used during the creation of the runbook.

PowerShell Workflow runbooks doesn’t have a graphical presentation of the workflow. You can use a text editor to create and modify PowerShell Workflow runbooks. But you need to know how to deal with the logic of PowerShell Workflow code.

PowerShell runbooks are plain PowerShell code. Unlike PowerShell Workflows, a PowerShell runbook is faster, because it doesn’t have to be compiled before the run. But you have to be familiar with PowerShell. There is no parallel processing and you can’t use checkpoints (if a snapshot fails, it will be suspended. With a checkpoint, the workflow can started at the last sucessful checkpoint).

Schedule

Schedules are used to run runbooks to a specific point in time. Runbooks and schedules have a M:N relationship. A schedule can be associated with one or more runbooks, and a runbook can be linked to one or more schedules.

Summary

This is only a brief introduction into Azure Automation. Azure Automation uses Automation accounts to execute runbooks. A runbook consists of PowerShell Workflow or plain PowerShell code. You can use runbooks to automate nearly all operations of Azure services. To execute runbooks to a specific point in time, you can use schedules Runbooks, schedules and automation assets, like credentials, certificates etc., are associated with a specific Automation account. This helps you to separate between different Automation accounts, e.g. accounts for development and for production.

Starting and stopping Azure VMs with Azure PowerShell

To be honest: I’m lazy and I have a wife and two kids. Therefore I have to minimize the costs of my lab. I have a physical lab at the office and some VMs running on Microsoft Azure. Azure is nice, because I only have to pay what I really use. And because I’m only paying the actual use, I start the VMs only when I need them. Inspired by this very handy Azure VM wakeup & shutdown script, I decided to write my own script (yes, I invented a wheel again…). Very simple, nothing fancy. Feel free to use and modify the script according to your needs.

How to shrink thin-provisioned disks

Disk space is rare. I only have about 1 TB of SSD storage in my lab and I don’t like to waste too much of it. My hosts use NFS to connect to my Synology NAS, and even if I use the VAAI-NAS plugin, I use thin-provisioned disks only. Thin-provisioned disks tend to grow over time. If you copy a 1 GB file into a VM and you delete this file immediately, you will find that the VMDK is increased by 1 GB. This is caused by the guest filesystem. It marks the blocks of deleted files as free, even if it only deletes metadata and not the data itself. Later, the data is overwritten with new data, since the blocks are marked as free and the new data is written in there. VMware ESXi doesn’t know that the guest has marked blocks as free. So ESXi can’t shrink the thin-provisioned VMDK.

You can observe a similar behavior in case of VMFS and underlying thin-provisioned LUNs: If a VMDK is removed from a VMFS datastore, the underlying  thin-provisioned LUN doesn’t show more free space. In this case, the VAAI UNMAP primitive can be used to tell the storage system which blocks are free and can be reclaimed. Some storage system that doesn’t support VAAI UNMAP use contiguous regions filled with zeros to identify reclaimable storage space. Before free space can be reclaimed, the VMFS has to be filled with zeros. A similar technique can be used to shrink thin-provisioned guest hard disks. Please note that I don’t want to focus on reclaiming space from underlaying LUNs. I’m only talking about shrinking thin-provisioned disks!

To shrink a thin-provisioned VMDK the guest filesystem has to be zeroed out. If you use Windows, you can use SDelete. In case of a unixoide OS (Linux, FreeBSD, Solaris…), use dd. After you have zeroed out the guest file system, you have to move the VM with Storage vMotion to another datastore. Now it’s getting complicated: You have to make sure that the legacy datamover (fsdm) is used for the Storage vMotion. There are three different datamovers:

  • fsdm
  • fs3dm, and
  • fs3dm – hardware offload

The fsdm is the oldest and slowest datamover. The fs3dm and fs3dm with HW offload are newer. In case of the latter, the process is offloaded to the hardware using VAAI (Full Copy primitive). At this point, I’d like to refer to a blog post of Duncan Epping (Blocksize impact?) , who has highlighted the differences between the datamovers more detailed. The point is, that the fsdm doesn’t copy blocks that are filled with zeros. But how can I make sure, that the fsdm is used?

  • Move the VM to a datastore with another blocksize

This can be difficult, because VMFS5 datastores have a block size of 1 MB, except they were upgraded from VMFS3. Simply create a new VMFS3 datastore and use it as destination.

  • Move the VM from VMFS to NFS, from NFS to VMFS or from NFS to NFS

In this case fsdm will be used. Please note that fsdm will not be used if you move a VM from a VMFS5 to a VMFS5 datastore! In this case the fs3dm is used. This wouldn’t shrink the thin-provisioned VMDK. On the downside the fsdm is slow. Really slow. If you have a monster VM, a vMotion can take a looooong time (worth reading: “VMware Storage vMotion, Data Movers, Thin Provisioning, Barriers to Monster VM’s” by Michael Webster).

I wrote a PowerShell script that uses PowerShell remoting and VMwares PowerCLI cmdlets to do the following tasks:

  • get a list of all local disks using Get-WmiObject
  • zero-out filesystem on those disks
  • move the VM to a destination datastore
  • move the VM back to its source host and source datastore

For the moment, the script only works with Windows VMs. SDelete must be available in the VM. Make sure that you use the latest release of SDelete (currently 1.61). PowerShell remoting has to be enabled on the VMs. Feel free to use and/ or edit my script. To get this script working, please change the content of the variables for

  • $PathToSDelete
  • $VIServer
  • $CredFile
  • $Username
  • $DstDS
  • $DstDSHost and
  • $ClusterName

according to your environment. The script skips VMs with active snapshots and VMs that have one or more ZeroedThick or EagerZeroedThick disks attached. Because the script use all local disks, it will also zero-out disks that were attached using in-guest iSCSI. So please be test the script in your lab until you try it in production.

This is an example for the output of the script:

VMDK-thin-reclaim_script_output

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

In this picture you can see, that the script processes one disk after another:

VMDK-thin-reclaim_performance_zero_out

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

This script is provided “AS IS” with no warranty expressed or implied. Run at your own risk. Please test the script in your lab.

Automating updates during MDT 2013 Lite-Touch deployments

I use Microsofts Deployment Toolkit (MDT) in my lab to deploy Windows VMs with Windows Server 2008 and Windows Server 2012. I described the installation and configuration of MDT in a small blog post series. Take a look into the intro post, if you’re a new to MDT. But the OS installation isn’t the time consuming part of a deployment: It’s the installation of patches. Because of this, I decided to automate the patch installation and make it part of the OS installation.

The requirements

To automate the installation of patches, we need

To save resources, I’ve installed WSUS on the server I also use for MDT. In Windows Server 2008 R2 and Server 2012 (R2) WSUS is a installable role. Because I use a Windows 2008 R2 host for MDT, I could simply add the role to the server. I will not describe the installation of the WSUS role, because this is really easy.

Configuration of MDT 2013

In principle, there are two changes:

  • Enableing Windows update in the task sequence
  • Adding WSUS server to the CustomSettings.ini file

First of all you need to enable the Windows update part in the task sequence. Start the Deployment Workbench and navigate to the task sequences. Go into the properties, switch to the “Task Sequence” tab and enable the “Windows Update (Post-Application Installation)” task by unchecking the “Disable this step” box on the “Options” tab.

configure_mdt_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click “OK” and switch to the deployment share. Go into the “Control” directory and open the CustomSettings.ini. Add this line to the end of the [Default] section:

Make sure that you change the FQDN to your WSUS host and save the file.

If everything went fine, you should see this during the deployment process:

configure_mdt_3

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The host, that is currently deployed, should also appear in the WSUS console.

configure_mdt_4

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

How it works

During the deployment process the script ZTIWindowsUpdate.wsf is called. This script connects to the WSUS server and installs all appropriate updates, servicepacks etc. This includes the latest version of the Windows Update API and the Microsoft Update binaries. Because the script install ALL appropriate updates, service packs etc., there is no way to exclude updates from being installed. Really no way? However, there is a way. You can use the WUMU_ExcludeKB switch in the CustomSettings.ini to exclude updates. Simply add one line for each KB that you want to suppress.