Category Archives: Automation

Use app-only authentication with the Microsoft Graph PowerShell SDK

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

In the previous blog post I have showed you how to interactively log in into the Microsoft Graph API. You had to enter a username, a password, and you had to enter a second factor. This is typically not want you want if you want to automate things. But there is another way to get access to the Microsoft Graph API.

Create an app registration

To get access, you have to register an app in your AzureAD. Go to your Azure portal and select “App registration” from the “Manage” section. Add a new registration by clicking to “New registration”.

Give your registration a meaningful name. Usually, only accounts in your AzureAD should be able to use this app.

The next step is to add permissions. This is equivalent to defining permission scopes during an interactive login. Make sure that you only follow the least-privilege method. In contrast to delegate access, this login type is truly limited to the permissions you grant in this step.

Select “Microsoft Graph” from the list.

Choose “Application permissions”.

Then select the necessary permissions.

Grant the permissions and select “Grant admin consent”. This step is pretty important. You, the admin, consent to the selected permissions. There is no further question to consent to the enduser.

Login with a client secret

But before we can use, we have to add something to use in the authentication process. There are two different methods to authenticate:

  • Certificate, or
  • client secret

A client secret is okay for test or dev environments. But I would not recommend the usage in a prod environment. You have to add this secret to a script or something, which is hard to protect. Create a new client secret. Please note, that a client secret has a lifetime. And make sure that you copy it. Tge client secret will be hidden later!

Make sure that you give your client secret a descriptive name.

The usage of a client secret is a two-step process. We need to get an access token, using the client secret, and use the token to connect to the Graph API. To get an access token, you need to install the Microsoft Authentication Libraries (MSAL) PowerShell module.

Install-Module MSAL.PS -Scope CurrentUser

Then we can aquire the token.

$AppId = '525b0e65-xxxx-xxxx-xxxx-7f8c32536247'
$TenantId = 'ffbc872a-xxxx-xxxx-xxxx-d81b43c67ffe'
$ClientSecret = 'NmO8Q~PPzVqZnxxxxxxxxi0vfRBhj8_xxxxxxx'
 
$Token = Get-MsalToken -TenantId $TenantId -ClientId $AppId -ClientSecret ($ClientSecret | ConvertTo-SecureString -AsPlainText -Force)
 
Connect-Graph -AccessToken $MsalToken.AccessToken

As you can see, this service principal login was made by the client secret.

Login with a certificate

Something more appropriate for a prod environment, is to use a certificate for the login. You can create a self-signed certificate, or use any other kind of X.509 certificate to authenticate. I used a S/MIME certificate in this case.

Make sure that you only upload the public key!! The certificate with the private key must be stored in the computer or user certificate store on the machine from which you want to access.

Upload the public key of a certificate.

Next step is to use the certificate hash during the login process:

Connect-MgGraph -ClientId 525b0e65-xxxx-xxxx-xxxx-7f8c32536247 -TenantId ffbc872a-xxxx-xxxx-xxxx-d81b43c67ffe -CertificateThumbprint DC427652498895A6F453671275BC69B352F3510A

Same result, a successful login, but different authentication method.

As already mentioned: I would prefer certificate over client secret. :)

CloudFlare API v4 and Fail2ban: Fixing the unban action

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

In January 2017, I wrote an article about how to protect your WordPress blog using the WP Fail2Ban plugin, fail2ban on your Linux/ FreeBSD host, and CloudFlare. Back then, the fail2ban was using the CloudFlare API V1, which was already deprecated since November 2016.

Free-Photos/ pixabay.com/ Creative Commons CC0

Although the actions were updated later to use CloudFlare API V4, I still had problems with the unbaning of IP addresses. IP addresses were banned, but the unban action failed. 

This is the unban action, which is included in fail2ban (taken from fail2ban-0.10.3.1 which is shipped with FreeBSD 11.1-RELEASE-p10):

actionunban = curl -s -o /dev/null -X DELETE -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' \
            https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules/$(curl -s -X GET -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' \
            'https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules?mode=block&configuration_target=ip&configuration_value=<ip>&page=1&per_page=1' | cut -d'"' -f6)

And this is the unban action, which finally solved this issue:

actionunban = curl -s -o /dev/null -X DELETE -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' \
            https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules/$(curl -s -X GET -H 'X-Auth-Email: <cfuser>' -H 'X-Auth-Key: <cftoken>' \
            'https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules?mode=block&configuration_target=ip&configuration_value=<ip>&page=1&per_page=1' | tr -d '\n' | cut -d'"' -f6)

I found the solution at serverfault.com. The only difference is an additional tr -d ‘\n’  in the last line of the statement. Kudos to Jake for fixing this!

To prevent the action file to being overwritten, you should copy the original cloudflare.conf  located in the  action.d  directory, e.g. to mycloudflare.conf , and use the copied action file in your fail definition.

[wordpress]
enabled  = true
filter   = wordpress
logpath  = /var/log/messages
action   = mycloudflare

Azure PowerShell vs. Azure RM PowerShell

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

In 2014, Microsoft announced the Azure Preview Portal, which was going GA in December 2015. Since January 8, 2018, the classic Azure Portal is turned off. The “Preview Portal” was more than a facelift. The classic Azure Portal was based on the Service Management mode, often called the “classic deployment model”, whereas the new Azure Portal uses the Resource Manager model. Azure Service Management (ASM) and Azure Resource Management are both deployment models. The Resource Manager model eases the deployment of complex setups by using templates to deploy, update and manage resources within a resource group as a single operation.

Azure PowerShell vs. Azure RM PowerShell

Different deployment models require different tools. Because of this, Microsoft offers two PowerShell modules for Azure. Depending on your deployment type, you have to use the Azure or AzureRM module. Both can be installed directly from the PowerShell Gallery using Install-Module -Name Azure or Install-Module -Name AzureRM .

Connect to Azure

Depending on the used module, the ways to connect to Azure differ.

Module AzureRM

PS C:\Users\p.terlisten> Connect-AzureRmAccount -Subscription 37cbb19b-cc7f-402a-xxxx-yxcvbnmasdfg


Account          : patrick@blazilla.de
SubscriptionName : vcloudnine Azure Lab
SubscriptionId   : 37cbb19b-cc7f-402a-xxxx-yxcvbnmasdfg
TenantId         : 2795c72a-aad7-404e-xxxx-yxcvbnmasdfg
Environment      : AzureCloud


PS C:\Users\p.terlisten>

You will notice, that AzureRM sessions does not persist between PowerShell sessions. This behaviour differs from Add-AzureAccount . But you can save and load your AzureRM session once you are connected.

PS C:\Users\p.terlisten> Save-AzureRmContext -Path X:\Secure\azurerm.json
PS C:\Users\p.terlisten> Import-AzureRmContext -Path X:\Secure\azurerm.json

Module Azure

PS C:\Users\p.terlisten> Add-AzureAccount

Id                  Type Subscriptions                           Tenants
--                  ---- -------------                           -------
patrick@blazilla.de User 37cbb19b-cc7f-402a-xxxx-yxcvbnmasdfg... {2795c72a-aad7-404e-xxxx-yxcvbnmasdfg}

PS C:\Users\p.terlisten>

Using WP fail2ban with the CloudFlare API to protect your website

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

The downside of using WordPress is that many people use it. That makes WordPress a perfect target for attacks. I have some trouble with attacks, and one of the consequences is, that my web server crashes under load. The easiest way to solve this issue would be to ban those IP addresses. I use Fail2ban to protect some other services. So the idea of using Fail2ban to ban IP addresses, that are used for attacks, was obvious.

From the Fail2ban wiki:

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs — too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc).

That works for services, like IMAP, very good. Unfortunately, this does not work out of the box for WordPress. But adding the WordPress plugin WP fail2ban brings us closer to the solution. For performance and security reasons, vcloudnine.de can only be accessed through a content delivery network (CDN), in this case CloudFlare. Because CloudFlare acts as a reverse proxy, I can not see “the real” IP address. Furthermore, I can not log the IP addresses because of the German data protection law. This makes the Fail2ban and the WordPress Fail2ban plugin nearly useless, because all I would ban with iptables, would be the CloudFlare CND IP ranges. But CloudFlare offers a firewall service. CloudFlare would be the right place to block IP addresses.

So, how can I stick Fail2ban, the WP Fail2ban plugin and CloudFlares firewall service together?

APIs FTW!

APIs are the solution for nearly every problem. Like others, CloudFlare offers an API that can be used to automate tasks. In this case, I use the API to add entries to the CloudFlare firewall. Or honestly: Someone wrote a Fail2ban action that do this for me.

First of all, you have to install the WP Fail2ban plugin. That is easy. Simply install the plugin. Then copy the wordpress-hard.conf from the plugin directory to the filters.d directory of Fail2ban.

[root@webserver filters.d]# cp wordpress-hard.conf /etc/fail2ban/filter.d/

Then edit the /etc/fail2ban/jail.conf and add the necessary entries for WordPress.

[wordpress-hard]

enabled  = true
filter   = wordpress-hard
logpath  = /var/log/messages
action   = cloudflare
maxretry = 3
bantime  = 604800

Please note, that in my case, the plugin logs to /var/log/messages. The action is “cloudflare”. To allow Fail2ban to work with the CloudFlare API, you need the CloudFlare API Key. This key is uniqe for every CloudFlare account. You can get this key from you CloudFlare user profile. Go to the user settings and scroll down.

Cloudflare Global API Key

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Open the /etc/fail2ban/action.d/cloudflare.conf and scroll to the end of the file. Add the token and your CloudFlare login name (e-mail address) to the file.

# Default Cloudflare API token
cftoken = 1234567890abcdefghijklmopqrstuvwxyz99

cfuser = user@domain.tld

Last step is to tell the WP Fail2ban plugin which IPs should be trusted. We have to add subnets of the CloudFlare CDN. Edit you wp-config.php and add this line at the end:

/** CloudFlare IP Ranges */
define('WP_FAIL2BAN_PROXIES','103.21.244.0/22,103.22.200.0/22,103.31.4.0/22,104.16.0.0/12,108.162.192.0/18,131.0.72.0/22,141.101.64.0/18,162.158.0.0/15,172.64.0.0/13,173.245.48.0/20,188.114.96.0/20,190.93.240.0/20,197.234.240.0/22,198.41.128.0/17,199.27.128.0/21,2400:cb00::/32,2405:8100::/32,2405:b500::/32,2606:4700::/32,2803:f800::/32,2c0f:f248::/32,2a06:98c0::/29');

The reason for this can be found in the FAQ of the WP Fail2ban plugin. The IP ranges used by CloudFlare can be found at CloudFlare.

Does it work?

Seems so… This is an example from /var/log/messages.

Jan 15 20:01:46 webserver wordpress(www.vcloudnine.de)[4312]: Authentication attempt for unknown user vcloudnine from 195.154.183.xxx
Jan 15 20:01:46 webserver fail2ban.filter[4393]: INFO [wordpress-hard] Found 195.154.183.xxx

And this is a screenshot from the CloudFlare firewall section.

Cloudflare Firewall Blocked Websites

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Another short test with curl has also worked. I will monitor the firewall section of CloudFlare. Let’s see who’s added next…

Important note for those, who use SELinux: Make sure that you install the policycoreutils-python package, and create a custom policy for Fail2Ban!

[root@webserver ~]# grep fail2ban /var/log/audit/audit.log | audit2allow -M myfail2banpolicy
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i myfail2banpolicy.pp

A strong indicator are errors like this in /var/log/messages:

Jan 22 12:06:03 webserver fail2ban.actions[16399]: NOTICE [wordpress-hard] Ban xx.xx.xx.xx
Jan 22 12:06:03 webserver fail2ban.action[16399]: ERROR curl -s -o /dev/null https://www.cloudflare.com/api_json.html -d 'a=ban' -d 'tkn=7c8e62809d4183931347772b366e621003c63' -d 'email=patrick@blazilla.de' -d 'key=xx.xx.xx.xx' -- stdout: ''
Jan 22 12:06:03 webserver fail2ban.action[16399]: ERROR curl -s -o /dev/null https://www.cloudflare.com/api_json.html -d 'a=ban' -d 'tkn=7c8e62809d4183931347772b366e621003c63' -d 'email=patrick@blazilla.de' -d 'key=xx.xx.xx.xx' -- stderr: ''
Jan 22 12:06:03 webserver fail2ban.action[16399]: ERROR curl -s -o /dev/null https://www.cloudflare.com/api_json.html -d 'a=ban' -d 'tkn=7c8e62809d4183931347772b366e621003c63' -d 'email=patrick@blazilla.de' -d 'key=xx.xx.xx.xx' -- returned 7
Jan 22 12:06:03 webserver fail2ban.actions[16399]: ERROR Failed to execute ban jail 'wordpress-hard' action 'cloudflare' info 'CallingMap({'ipjailmatches': <function <lambda> at 0x7f49967edc80>, 'matches': '', 'ip': 'xx.xx.xx.xx', 'ipmatches': <function <lambda> at 0x7f49967edde8>, 'ipfailures': <function <lambda> at 0x7f49967edc08>, 'time': 1485083163.0328701, 'failures': 2, 'ipjailfailures': <function <lambda> at 0x7f49967eded8>})': Error banning xx.xx.xx.xx

You will find corresponding audit messages in the /var/log/audit.log:

type=AVC msg=audit(1485083254.298:17688): avc:  denied  { name_connect } for  pid=16575 comm="curl" dest=443 scontext=unconfined_u:system_r:fail2ban_t:s0 tcontext=system_u:object_r:http_port_t:s0 tclass=tcp_socket

Make sure that you create a custom policy for Fail2Ban, and that you load the policy.

HPE ProLiant PowerShell SDK

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Some days ago, my colleague Claudia and I started to work on a new project: A greenfield deployment consisting of some well known building blocks: HPE ProLiant, HPE MSA, HPE Networking (now Aruba) and VMware vSphere. Nothing new for us, because we did this a couple times together. But this led us to the idea, to automate some tasks. Especially the configuration of the HPE ProLiants: Changing BIOS settings and configuring the iLO.

Do not automate what you have not fully understood

Some of the wisest words I have ever said to a customer. Modifying the BIOS and iLO settings is a well understood task. But if you have to deploy a bunch of ProLiants, this is a monotonous, and therefore error prone process. Perfect for automation!

Scripting Tools for Windows PowerShell

To support the automation of HPE ProLiant deployments, HPE offers the Scripting Tools for Windows PowerShell. HPE offers the PowerShell modules free for charge. There are three different downloads:

  • iLO cmdlets
  • BIOS cmdlets
  • Onboard Administrator (OA) cmdlets

The iLO cmdlets include PowerShell cmdlets to configure and manage iLO on HPE ProLiant G7, Gen8 or Gen9 servers. The BIOS cmdlets does not support G7 servers, so you can only configure and manage legacy and UEFI BIOS for Gen8 (except DL580) and all Gen9 models. The OA cmdlets support the configuration and management of the HPE Onboard Administrator, which is used with HPEs well known ProLiant BL blade servers. The OA cmdlets need at least  OA v3.11, whereby v4.60 is the latest version available.  All you need to get started are

  • Microsoft .NET Framework 4.5, and
  • Windows Management Framework 3.0 or later

If you are using Windows 8 or 10, you already have PowerShell 4 respectively PowerShell 5.

Support for HPE ProLiant Gen9 iLO RESTful API

If you have ever seen a HPE ProLiant Gen9 booting up, you might have noticed the iLO RESTful API icon down right. Depending on the server model, the BIOS cmdlets utilize the ILO4 RESTful API. But the iLO RESTful API ecosystem is it worth to be presented in an own blog post. Stay tuned.

Documentation and examples

HPE offers a simple documentation for the BIOS, iLO and OA cmdlets. You can find the documentation in HPEs Information Library. Documentation is important, but sometimes example code is necessary to quickly ramp up code. Check HPEs PowerShell SDK GitHub repository for examples.

Time to code

I’m keen on it and curious to automate some of my regular deployment tasks with these PowerShell modules. Some of these tasks are always the same:

  • change the power management and other BIOS settings
  • change the network settings of the iLO
  • change the initial password of the iLO administrator account and create additional iLO user accounts

Further automation tasks are not necessarily related to the HPE ProLiant PowerShell SDK, but to PowerShell, respectively VMware PowerCLI. PowerShell is great to automate the different aspects and modules of an infrastructure deployment. You can use it to build your own tool box.

PowerCLI: Get-LunPathState

This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Careful preparation is a key element to success. If you restart a storage controller, or even the whole storage, you should be very sure that all ESXi hosts have enough paths to every datstore. Sure, you can use the VMware vSphere C# client or the Web Client to check every host and every datastore. But if you have a large cluster with a dozen datastores and some Raw Device Mappings (RDMs), this can take a looooong time. Checking the path state of each LUN is a task, which can be perfectly automated. Get a list of all hosts, loop through every host and every LUN, output a list of all hosts with all LUNs and all paths for each LUN. Sounds easy, right?

For a long time, I used this PowerCLI script for checking the LUN path state. But now I decided to give something back and I tweaked it a bit for my needs.

Feel free to use and/ or modify it.

Certificate-based authentication of Azure Automation accounts

This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Before you can manage Azure services with Azure Automation, you need to authenticate the Automation account against a subscription. This authentication process is part of each runbook. There are two different ways to authenticate against an Azure subscription:

  • Active Directory user
  • Certificate

If you want to use an Active Directory account, you have to create a credential asset in the Automation account and provide username and password for that Active Directory account. You can retrieve the credentials using the Get-AzureAutomationCredential cmdlet. This cmdlet returns a System.Management.Automation.PSCredential object, which can be used with Add-AzureAccount to connect to a subscription. If you want to use a certificate, you need four assets in the Automation account: A certificate and variables with the certificate name, the subscription ID and the subscription name. The values of these assets can be retrieved with Get-AutomationVariable and Get-AutomationCertificate.

Prerequisites

Before you start, you need a certificate. This certificate can be a self- or a CA-signed certificate. Check this blog post from Alice Waddicor if you want to start with a self-signed certificate. I used a certificate, that was signed by my lab CA.

At a Glance:

  • self- or CA-signed certificate
  • Base64 encoded DER format (file name extension .cer) to upload it as a management certificate
  • PKCS #12 format with private key (file name extension .pfx or .cer) to use it as an asset inside the Automation account

Upload the management certificate

First, you must upload the certificate to the management certificates. Login to Azure and click “Settings”.

creat_automation_account_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click on “Management Certificates”

creat_automation_account_02

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

and select “Upload” at the bottom of the website.

creat_automation_account_03

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Make sure that the certificate has the correct format and file name extension (.cer).

creat_automation_account_04

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Finish the upload dialog. After a few seconds, the certificate should appear in the listing.

creat_automation_account_06

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Create a new Automation account

Now it’s time to create the Automation account. Select “Automation” from the left panel.

creat_automation_account_07

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click on “Create an Automation account”.

creat_automation_account_08

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Give your Automation account a descriptive name and select a region. Please note that an Automation account can manage Azure services from all regions!

creat_automation_account_09

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click on the newly created account and click on “Assets”.

creat_automation_account_10

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Select “Add setting” from the bottom of the website.

creat_automation_account_11

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Add a credential asset by choosing “Add credential” and select “Certificate” as “Credential type”.

creat_automation_account_12

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Enter a descriptive name for the certificate. You should remember this name. You will need it later. Now you have to upload the certificate. The certificate must have the file name extension .pfx or .cer and it must include the private key!

creat_automation_account_13

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Finish the upload of the certificate. Now add three additional assets (variables).

creat_automation_account_14

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Select the name, the value and the type from the  table below. The name of the certificate is the descriptive name, you’ve previously entered when uploading the certificate.

NameValueType
AutomationCertificateNameName of your certificateString
AzureSubscriptionNameName of your subscriptionString
AzureSubscriptionID36 digit ID of the subscriptionString

Done. You’ve uploaded and created all the required certificates and variables.

How to use it

To use the certificate and the variables to connect to an Azure subscription, you have to use the two cmdlets Get-AutomationCertificate and Get-AutomationVariable. I use this code block in my runbooks:

$AzureSubscriptionName = Get-AutomationVariable -Name "AzureSubscriptionName" 
$AzureSubscriptionID = Get-AutomationVariable -Name "AzureSubscriptionID" 
$AutomationCertificateName = Get-AutomationVariable -Name "AutomationCertificateName"
$CertificateName = Get-AutomationCertificate -Name $AutomationCertificateName

Set-AzureSubscription -SubscriptionName $AzureSubscriptionName -SubscriptionId $AzureSubscriptionID -Certificate $CertificateName
Select-AzureSubscription $AzureSubscriptionName

Works like a charm.

Summary

Certificate-based authentication is an easy way to authenticate an Automation account against an Azure subscription. It’s easy to implement and you don’t have to maintain users and passwords. You can use different certificates for different Automation accounts. I really recommend this, especially if you have separate accounts for dev, test and production.

All you need is to upload a certificate as a management certificates, and as a credential asset in the Automation account.  You can use a self- or CA-signed certificate. The subscription ID, the subscription name and the name of the certificate are stored in variables.

At the beginning of each runbook, you have to insert a code block. This code block takes care of authentication.

A brief introduction into Azure Automation

This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Automation is essential to reduce friction and to streamline operational processes. It’s indispensable when it comes to the automation of manual, error-prone and frequently repeated tasks in a cloud or enterprise environment. Automation is the key to IT industrialization. Azure Automation is used to automate operational processes withing Microsoft Azure.

Automation account

The very first thing you have to create is an Automation account. You can have multiple Automation accounts per subscription. An Automation account allows you so separate automation resources from other Automation accounts. Automation resources are runbooks and assets (credentials, certificates, connection strings, variables, scheudles etc.). So each Automation account has its own set of runbooks and assets. This is perfect to separate production from development. An Automation account is associated with an Azure region, but the Automation account can manage Azure services in all regions.

Runbooks

A runbook is a collection of PowerShell script or PowerShell workflows. You can automate nearly everything with it. If something provides an API, you can use a runbook and PowerShell to automate it. A runbook can run other runbooks, so you can build really complex automation processes. A runbook can access any services that can be accessed by Microsoft Azure, regardless if it’s an internal or external service.

There are three types of runbooks:

  • Graphical runbooks
  • PowerShell Workflow runbooks
  • PowerShell runbooks

Graphical runbooks can be created and maintained with a graphical editor within the Azure portal. Graphical runbooks use PowerShell workflow code, but you can’t directly view oder modify this code. Graphical runbooks are great for customers, that don’t have much automation and/ or PowerShell knowledge. Once you created a graphical runbook with an automation account, you can export and import this runbook into another automation accounts, but you can modify the runbook only with the account which was used during the creation of the runbook.

PowerShell Workflow runbooks doesn’t have a graphical presentation of the workflow. You can use a text editor to create and modify PowerShell Workflow runbooks. But you need to know how to deal with the logic of PowerShell Workflow code.

PowerShell runbooks are plain PowerShell code. Unlike PowerShell Workflows, a PowerShell runbook is faster, because it doesn’t have to be compiled before the run. But you have to be familiar with PowerShell. There is no parallel processing and you can’t use checkpoints (if a snapshot fails, it will be suspended. With a checkpoint, the workflow can started at the last sucessful checkpoint).

Schedule

Schedules are used to run runbooks to a specific point in time. Runbooks and schedules have a M:N relationship. A schedule can be associated with one or more runbooks, and a runbook can be linked to one or more schedules.

Summary

This is only a brief introduction into Azure Automation. Azure Automation uses Automation accounts to execute runbooks. A runbook consists of PowerShell Workflow or plain PowerShell code. You can use runbooks to automate nearly all operations of Azure services. To execute runbooks to a specific point in time, you can use schedules Runbooks, schedules and automation assets, like credentials, certificates etc., are associated with a specific Automation account. This helps you to separate between different Automation accounts, e.g. accounts for development and for production.

Starting and stopping Azure VMs with Azure PowerShell

This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

To be honest: I’m lazy and I have a wife and two kids. Therefore I have to minimize the costs of my lab. I have a physical lab at the office and some VMs running on Microsoft Azure. Azure is nice, because I only have to pay what I really use. And because I’m only paying the actual use, I start the VMs only when I need them. Inspired by this very handy Azure VM wakeup & shutdown script, I decided to write my own script (yes, I invented a wheel again…). Very simple, nothing fancy. Feel free to use and modify the script according to your needs.

How to shrink thin-provisioned disks

This posting is ~9 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Disk space is rare. I only have about 1 TB of SSD storage in my lab and I don’t like to waste too much of it. My hosts use NFS to connect to my Synology NAS, and even if I use the VAAI-NAS plugin, I use thin-provisioned disks only. Thin-provisioned disks tend to grow over time. If you copy a 1 GB file into a VM and you delete this file immediately, you will find that the VMDK is increased by 1 GB. This is caused by the guest filesystem. It marks the blocks of deleted files as free, even if it only deletes metadata and not the data itself. Later, the data is overwritten with new data, since the blocks are marked as free and the new data is written in there. VMware ESXi doesn’t know that the guest has marked blocks as free. So ESXi can’t shrink the thin-provisioned VMDK.

You can observe a similar behavior in case of VMFS and underlying thin-provisioned LUNs: If a VMDK is removed from a VMFS datastore, the underlying  thin-provisioned LUN doesn’t show more free space. In this case, the VAAI UNMAP primitive can be used to tell the storage system which blocks are free and can be reclaimed. Some storage system that doesn’t support VAAI UNMAP use contiguous regions filled with zeros to identify reclaimable storage space. Before free space can be reclaimed, the VMFS has to be filled with zeros. A similar technique can be used to shrink thin-provisioned guest hard disks. Please note that I don’t want to focus on reclaiming space from underlaying LUNs. I’m only talking about shrinking thin-provisioned disks!

To shrink a thin-provisioned VMDK the guest filesystem has to be zeroed out. If you use Windows, you can use SDelete. In case of a unixoide OS (Linux, FreeBSD, Solaris…), use dd. After you have zeroed out the guest file system, you have to move the VM with Storage vMotion to another datastore. Now it’s getting complicated: You have to make sure that the legacy datamover (fsdm) is used for the Storage vMotion. There are three different datamovers:

  • fsdm
  • fs3dm, and
  • fs3dm – hardware offload

The fsdm is the oldest and slowest datamover. The fs3dm and fs3dm with HW offload are newer. In case of the latter, the process is offloaded to the hardware using VAAI (Full Copy primitive). At this point, I’d like to refer to a blog post of Duncan Epping (Blocksize impact?) , who has highlighted the differences between the datamovers more detailed. The point is, that the fsdm doesn’t copy blocks that are filled with zeros. But how can I make sure, that the fsdm is used?

  • Move the VM to a datastore with another blocksize

This can be difficult, because VMFS5 datastores have a block size of 1 MB, except they were upgraded from VMFS3. Simply create a new VMFS3 datastore and use it as destination.

  • Move the VM from VMFS to NFS, from NFS to VMFS or from NFS to NFS

In this case fsdm will be used. Please note that fsdm will not be used if you move a VM from a VMFS5 to a VMFS5 datastore! In this case the fs3dm is used. This wouldn’t shrink the thin-provisioned VMDK. On the downside the fsdm is slow. Really slow. If you have a monster VM, a vMotion can take a looooong time (worth reading: “VMware Storage vMotion, Data Movers, Thin Provisioning, Barriers to Monster VM’s” by Michael Webster).

I wrote a PowerShell script that uses PowerShell remoting and VMwares PowerCLI cmdlets to do the following tasks:

  • get a list of all local disks using Get-WmiObject
  • zero-out filesystem on those disks
  • move the VM to a destination datastore
  • move the VM back to its source host and source datastore

For the moment, the script only works with Windows VMs. SDelete must be available in the VM. Make sure that you use the latest release of SDelete (currently 1.61). PowerShell remoting has to be enabled on the VMs. Feel free to use and/ or edit my script. To get this script working, please change the content of the variables for

  • $PathToSDelete
  • $VIServer
  • $CredFile
  • $Username
  • $DstDS
  • $DstDSHost and
  • $ClusterName

according to your environment. The script skips VMs with active snapshots and VMs that have one or more ZeroedThick or EagerZeroedThick disks attached. Because the script use all local disks, it will also zero-out disks that were attached using in-guest iSCSI. So please be test the script in your lab until you try it in production.

This is an example for the output of the script:

VMDK-thin-reclaim_script_output

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

In this picture you can see, that the script processes one disk after another:

VMDK-thin-reclaim_performance_zero_out

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

This script is provided “AS IS” with no warranty expressed or implied. Run at your own risk. Please test the script in your lab.