Tag Archives: automation

Wrong iovDisableIR setting on ProLiant Gen8 might cause a PSOD

TL;DR: There’s a script at the bottom of the page that fixes the issue.

Some days ago, this HPE customer advisory caught my attention:

Advisory: (Revision) VMware – HPE ProLiant Gen8 Servers running VMware ESXi 5.5 Patch 10, VMware ESXi 6.0 Patch 4, Or VMware ESXi 6.5 May Experience Purple Screen Of Death (PSOD): LINT1 Motherboard Interrupt

And there is also a corrosponding VMware KB article:

ESXi host fails with intermittent NMI PSOD on HP ProLiant Gen8 servers

It isn’t clear WHY this setting was changed, but in VMware ESXi 5.5 patch 10, 6.0  patch 4, 6.0 U3 and, 6.5 the Intel IOMMU’s interrupt remapper functionality was disabled. So if you are running these ESXi versions on a HPE ProLiant Gen8, you might want to check if you are affected.

To make it clear again, only HPE ProLiant Gen8 models are affected. No newer (Gen9) or older (G6, G7) models.

Currently there is no resolution, only a workaround. The iovDisableIR setting must set to FALSE. If it’s set to TRUE, the Intel IOMMU’s interrupt remapper functionality is disabled.

To check this setting, you have to SSH to each host, and use esxcli  to check the current setting:

I have written a small PowerCLI script that uses the Get-EsxCli cmdlet to check all hosts in a cluster. The script only checks the setting, it doesn’t change the iovDisableIR setting.

Here’s another script, that analyzes and fixes the issue.

Tiny PowerShell/ Azure project: Deploy-AzureLab.ps1

One of my personal predictions for 2017 is, that Microsoft Azure will gain more market share. Especially here in Germany. Because of this, I have started to refresh my knowledge about Azure. A nice side effect is that I can also improve my PowerShell skills.

Currently, the script creates a couple of VMs and resource groups. Nothing more, nothing less. The next features I want to add are:

  • add additional disks to the DCs (for SYSVOL and NTDS)
  • promote both two servers to domain controllers
  • change the DNS settings for the Azure vNetwork
  • deploy a Windows 10 client VM

I created a new repository on GitHub and shared a first v0.1 as public Gist. Please note, that this is REALLY a v0.1.

Using WP fail2ban with the CloudFlare API to protect your website

The downside of using WordPress is that many people use it. That makes WordPress a perfect target for attacks. I have some trouble with attacks, and one of the consequences is, that my web server crashes under load. The easiest way to solve this issue would be to ban those IP addresses. I use Fail2ban to protect some other services. So the idea of using Fail2ban to ban IP addresses, that are used for attacks, was obvious.

From the Fail2ban wiki:

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs — too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc).

That works for services, like IMAP, very good. Unfortunately, this does not work out of the box for WordPress. But adding the WordPress plugin WP fail2ban brings us closer to the solution. For performance and security reasons, vcloudnine.de can only be accessed through a content delivery network (CDN), in this case CloudFlare. Because CloudFlare acts as a reverse proxy, I can not see “the real” IP address. Furthermore, I can not log the IP addresses because of the German data protection law. This makes the Fail2ban and the WordPress Fail2ban plugin nearly useless, because all I would ban with iptables, would be the CloudFlare CND IP ranges. But CloudFlare offers a firewall service. CloudFlare would be the right place to block IP addresses.

So, how can I stick Fail2ban, the WP Fail2ban plugin and CloudFlares firewall service together?

APIs FTW!

APIs are the solution for nearly every problem. Like others, CloudFlare offers an API that can be used to automate tasks. In this case, I use the API to add entries to the CloudFlare firewall. Or honestly: Someone wrote a Fail2ban action that do this for me.

First of all, you have to install the WP Fail2ban plugin. That is easy. Simply install the plugin. Then copy the wordpress-hard.conf from the plugin directory to the filters.d directory of Fail2ban.

Then edit the /etc/fail2ban/jail.conf and add the necessary entries for WordPress.

Please note, that in my case, the plugin logs to /var/log/messages. The action is “cloudflare”. To allow Fail2ban to work with the CloudFlare API, you need the CloudFlare API Key. This key is uniqe for every CloudFlare account. You can get this key from you CloudFlare user profile. Go to the user settings and scroll down.

Open the /etc/fail2ban/action.d/cloudflare.conf and scroll to the end of the file. Add the token and your CloudFlare login name (e-mail address) to the file.

Last step is to tell the WP Fail2ban plugin which IPs should be trusted. We have to add subnets of the CloudFlare CDN. Edit you wp-config.php and add this line at the end:

The reason for this can be found in the FAQ of the WP Fail2ban plugin. The IP ranges used by CloudFlare can be found at CloudFlare.

Does it work?

Seems so… This is an example from /var/log/messages.

And this is a screenshot from the CloudFlare firewall section.

Another short test with curl has also worked. I will monitor the firewall section of CloudFlare. Let’s see who’s added next…

Important note for those, who use SELinux: Make sure that you install the policycoreutils-python package, and create a custom policy for Fail2Ban!

A strong indicator are errors like this in /var/log/messages:

You will find corresponding audit messages in the /var/log/audit.log:

Make sure that you create a custom policy for Fail2Ban, and that you load the policy.

HPE ProLiant PowerShell SDK

Some days ago, my colleague Claudia and I started to work on a new project: A greenfield deployment consisting of some well known building blocks: HPE ProLiant, HPE MSA, HPE Networking (now Aruba) and VMware vSphere. Nothing new for us, because we did this a couple times together. But this led us to the idea, to automate some tasks. Especially the configuration of the HPE ProLiants: Changing BIOS settings and configuring the iLO.

Do not automate what you have not fully understood

Some of the wisest words I have ever said to a customer. Modifying the BIOS and iLO settings is a well understood task. But if you have to deploy a bunch of ProLiants, this is a monotonous, and therefore error prone process. Perfect for automation!

Scripting Tools for Windows PowerShell

To support the automation of HPE ProLiant deployments, HPE offers the Scripting Tools for Windows PowerShell. HPE offers the PowerShell modules free for charge. There are three different downloads:

  • iLO cmdlets
  • BIOS cmdlets
  • Onboard Administrator (OA) cmdlets

The iLO cmdlets include PowerShell cmdlets to configure and manage iLO on HPE ProLiant G7, Gen8 or Gen9 servers. The BIOS cmdlets does not support G7 servers, so you can only configure and manage legacy and UEFI BIOS for Gen8 (except DL580) and all Gen9 models. The OA cmdlets support the configuration and management of the HPE Onboard Administrator, which is used with HPEs well known ProLiant BL blade servers. The OA cmdlets need at least  OA v3.11, whereby v4.60 is the latest version available.  All you need to get started are

  • Microsoft .NET Framework 4.5, and
  • Windows Management Framework 3.0 or later

If you are using Windows 8 or 10, you already have PowerShell 4 respectively PowerShell 5.

Support for HPE ProLiant Gen9 iLO RESTful API

If you have ever seen a HPE ProLiant Gen9 booting up, you might have noticed the iLO RESTful API icon down right. Depending on the server model, the BIOS cmdlets utilize the ILO4 RESTful API. But the iLO RESTful API ecosystem is it worth to be presented in an own blog post. Stay tuned.

Documentation and examples

HPE offers a simple documentation for the BIOS, iLO and OA cmdlets. You can find the documentation in HPEs Information Library. Documentation is important, but sometimes example code is necessary to quickly ramp up code. Check HPEs PowerShell SDK GitHub repository for examples.

Time to code

I’m keen on it and curious to automate some of my regular deployment tasks with these PowerShell modules. Some of these tasks are always the same:

  • change the power management and other BIOS settings
  • change the network settings of the iLO
  • change the initial password of the iLO administrator account and create additional iLO user accounts

Further automation tasks are not necessarily related to the HPE ProLiant PowerShell SDK, but to PowerShell, respectively VMware PowerCLI. PowerShell is great to automate the different aspects and modules of an infrastructure deployment. You can use it to build your own tool box.

HPE StoreVirtual REST API

Representational State Transfer (REST) APIs are all the rage. REST was defined by Roy Thomas Fielding in his PhD dissertation “Architectural Styles and the Design of Network-based Software Architectures“. The architectural style of REST describes six constraints:

  • Uniform interface
  • Stateless
  • Cacheable
  • Client – Server communication
  • Layered system
  • Code on demand

RESTful APIs typically use HTTP and HTTP verbs (GET, POST, PUT, DELETE, etc.) to send data to, or retrieve data from remote systems. To do so, REST APIs use Uniform Resource Identifiers (URIs) to interact with remote systems. Thus, a client can interact with a remote system over a REST API using standard HTTP URIs and HTTP verbs. For the data transfer, common internet media types, like JSON or XML are used. It’s important to understand that REST is not a standard per se. But most implementations make use of standards such as HTTP, URI, JSON or XML.

Because of the uniform interface, you have different choices in view of a client. I will use PowerShell and the Invoke-RestMethod cmdlet in my examples.

HPE StoreVirtual REST API

With the release of LeftHand OS 11.5 (the latest release is 12.6), HPE added a REST API for management and storage provisioning. Due to a re-engineered management stack, the REST API is significantly faster than the same task processed on the CLI or using the  Centralized Management Console (CMC). It’s perfect for automation and scripting. It allows customers to achieve a higher level of automation and operational simplicity. The StoreVirtual REST API is using JavaScript Object Notation (JSON) for data transfer between client and the StoreVirtual management group. With the REST API, you can

  • Read, create, and modify volumes
  • Create and delete snapshots
  • Create, modify, and delete servers
  • Grant and revoke access of servers to volumes

I use two StoreVirtal VSA (LeftHand OS 12.6) in my lab. Everything I show in this blog post is based on LeftHand OS 12.6.

The REST API in LeftHand OS 12.6 uses:

  • HTTPS 1.1
  • media types application/JSON
  • Internet media types application/schema+JSON
  • UTF-8 character encoding

RESTful APIs typically use HTTP and HTTP verbs (GET, POST, PUT, DELETE, etc.). I case of the StoreVirtual REST API:

  • GET is used to retrieve an object. No body is necessary.
  • PUT is used to update an object. The information to update the object is sent within the body.
  • POST is used to create of an object, or to invoke an action or event. The necessary information are sent within the body.
  • DELETE is used to delete an object.

Entry point for all REST API calls is /lhos, starting from a node, eg.

Subsequent resources are relative to this base URI. Resources are:

Resource pathDescription
/lhos/managementGroupManagement group entity
/lhos/clustersCluster collection
/lhos/cluster/<id>Cluster entity
/lhos/credentialsCredentials collection
/lhos/credentials/<session token>Credentials entity
/lhos/serversServer collection
/lhos/servers/<id>Server entity
/lhos/snapshotsSnapshot collection
/lhos/snapshots/<id>Snapshot entity
/lhos/volumesVolume collection
/lhos/volumes/<id> Volume entity

The object model of the StoreVirtual REST API uses

  • Collections, and
  • Entities

to address resources. An entity is used to address individual resources, whereas a collection is a group of individual resources. Resources can be addressed by using a URI.

Exploring the API

First of all, we need to authenticate us. Without a valid authentication token, no REST API queries can be made. To create a credential entity, we have to use the POST method.

$cred is a hash table which includes the username and the password. This hash table is converted to the JSON format with the ConvertTo-Json cmdlet. The JSON data will be used as body for our query. The result is an authentication token.

This authentication token must be used for all subsequent API queries. This query retrieves a collection of all valid sessions.

The GET method is used, and the authentication token is sent with the header of the request.

To retrieve an individual credential entity, the URI of the entity must be used.

The result of this query is the individual credential entity

It’s important to know, that if a session has not been used for 15 minutes, it is automatically removed. The same applies to constantly active sessions after 24 hours. After 24 hours, the credential entity will be automatically removed.

Let’s try to create a volume. The information about this new volume has to be sent within the body of our request. We use again the ConvertTo-Json cmdlet to convert a hash table with the necessary information to the JSON format.

The size must be specified in bytes. As a result, Invoke-RestMethod will output this:

Using the CMC, we can confirm that the volume was successfully created.

storevirtual_rest_api_vol_1

Since we have a volume, we can create a snapshot. To create a snapshot, we need to invoke an action on the volume entity. We have to use the POST method and the URI of our newly created volume.

In case of a successful query, Invoke-RestMethod will give us this output.

Again, we can use the CMC to confirm the success of our operation.

storevirtual_rest_api_vol_2

To delete the snapshot, the DELETE method and the URI of the snapshot entity must be used.

To confirm the successful deletion of the snapshot, the GET method can be used. The GET method will retrieve a collection of all snapshot entities.

The result will show no members inside of the snapshot collection.

At the end of the day, we remove our credential entity, because it’s not longer used. To delete the credential entity, we use the DELETE method with the URI of our credential entity.

The next query should fail, because the credential entity is no longer valid.

HTTPS workaround

The StoreVirtual API is only accessable over HTTPS. By default, the StoreVirtual nodes use an untrusted HTTPS certifificate. This will cause Invoke-RestMethod to fail.

After a little research, I found a workaround. This workaround uses the System.Security.Cryptography.X509Certificates namespace. You can use this snippet to build a function or add it to a try-catch block.

Final words

The StoreVirtual REST API is really handy. It can be used to perform all important tasks. It’s perfect for automation and it’s faster than the CLI. I’ve used PowerShell in my examples, but I’ve successfully tested it with Python. Make sure to take a look in to the HPE StoreVirtual REST API Reference Guide.

Dynamic VLAN assignment with AOS 6

Manually assigning ports to VLANs can be a time consuming and error prone process. Depending on the size of the network, there is a point where it doesn’t make sense to do this manually. Especially in SMB networks, VLANs are assigned manually, because the effort of automating the VLAN assignment exceeds the effort for manually assigning VLANs. Those environments are often very static. I know many SMB networks where VLAN have not been addressed for a long time. With declining costs for Layer 3 switches, the separation of workloads in VLANs for SMB customers became affordable. Server virtualization was another mainspring for VLANs and inter-VLAN routing. To be honest: I’m talking about SMB customers, not enterprise customers or enterprise-grade SMB customers (latter is my special term for SMB customers with enormous IT budgets…). But the main driver for VLANs was Voice over IP (VoIP). With the increasing proliferation of VoIP, even the smallest SMB customer were forced to use VLANs. But this led to situations, where customers had to change the switch config every time a new client or IP phone was added to the network. Common workarounds:

  • pre-configuring switches, eg. port 1 to 12 for clients and 13 to 24 for IP phones
  • connecting clients behind IP phones and pre-configuring all switch ports (untagged client and tagged VoIP VLAN)

Suitable for small environments, but difficult to handle if environments grow over time. And I’m not a friend of connecting clients behind IP phones… Enterprise, or enterprise-grade SMB customers tend to implement 802.1x to manage access to their network. With 802.1x it’s possible to assign ports to VLANs depending on the user identity. But 802.1x is complex. If you have the the knowhow, the time and the budget, please do 802.1x! But you should take the complexity into account. Today I want to show options, offisde of 802.1x, to dynamically assign ports to VLANs with Alcatel-Lucent Enterprise OmniSwitches.

First of all: We have to differ between AOS 6, AOS 7 and AOS 8. Alcatel-Lucent Enterprise (ALE) currently uses three different software releases, depending on the switch platform.

Switch modelAOS release
OmniSwitch 6250AOS 6
OmniSwitch 6350AOS 6
OmniSwitch 6450AOS 6
OmniSwitch 6850EAOS 6
OmniSwitch 6855AOS 6
OmniSwitch 6860(E)AOS 8
OmniSwitch 6900AOS 7
OmniSwitch 9000(E)AOS 6
OmniSwitch 9900AOS 6
OmniSwitch 10KAOS 7

Depending on the specific AOS release, there are various ways to enable dynamic VLAN assignment. The main reason for the different AOS releases is, that ALE shifts its networking core platform from Windriver VxWorks (AOS 6) to Linux (AOS 7 and AOS 8) (source #1, source #2).

This blog post will focus on

  • OmniSwitch 6250/ 6350/ 6450 running AOS 6.7.1

I plan to publish similar blog posts for

  • OmniSwitch 6900/ 10k running AOS 7.3.4
  • OmniSwitch 6860/ 6860E running AOS 8.2.1

Dynamic VLAN assignment with AOS 6

In general, there are three different ways to dynamically assign ports to VLANs with AOS 6:

  • VLAN mobility
  • User Network Profiles (UNP)
  • LLDP Media Endpoint Detection (LLDP-MED)

Let’s take a look at VLAN mobility. VLAN mobility is used to dynamically assign one or more VLANs to a port, based on traffic characteristics that were received on that port. The following information can be used to classify traffic:

  • 802.1Q VLAN ID tag
  • DHCP MAC address
  • DHCP MAC range
  • DHCP port
  • DHCP generic
  • MAC address
  • MAC address range
  • Network address
  • Protocol
  • Port

You can’t use VLAN mobility on ports that

  • is an 802.1Q tagged port
  • belongs to a Link Aggregation Group (LAG)
  • has Spanning Tree enabled and the BPDU ignore status is disabled
  • is used to mirror traffic

To allow the switch to dynamically assign ports to VLANs, VLAN mobility has to be enabled. By default, all ports are non-mobile ports. A non-mobile port is statically assigned to a specific VLAN.

To enable VLAN mobility for a port:

You can also use a port range.

To disable VLAN mobility use the “no” form of the command.

If a device sends ethernet frames with a 802.1Q VLAN ID tag, you can use the VLAN ID tag to dynamically assign a port to a VLAN. With VLAN mobility enabled, you only have to enable the “mobile-tag” option for the desired VLAN.

As soon as the switch receives a frame with a 802.1Q VLAN ID tag for VLAN 199, the port that received this frame is dynamically assigned to VLAN 199. That’s VLAN mobility based on 802.1Q VLAN ID tags. But you can also use VLAN rules. VLAN rules are created per VLAN. You can have one or more rules per VLAN. You can use the

  • Source MAC address
  • Source MAC address ranges
  • Switch ports, or
  • the DHCP request itself

to dynamically assign a port to a VLAN. This rule matches to DHCP requests from a single MAC address.

If a DHCP request with the specified MAC address is received, the port is dynamically assigned to VLAN 199. Because managing MAC addresses is not very handy, you can use MAC address ranges:

To use all DHCP requests on a specific port, use the DHCP port rule:

To use all received DHCP requests, use the DHCP generic rule:

To remove a rule, use the “no” form of the command.

Once the device has received an IP address from the DHCP server, the VLAN port assignment is dropped! Because of this, you can combine DHCP and network address rules. A network address rule dynamically assigns the VLAN depending on the IP subnet.

This rule assigns VLAN 199 to a port, that receives traffic from a client in the subnet 192.168.20.0/24. If the DHCP server in VLAN 199 assigns IP addresses from this subnet, you can easily combine the DHCP and network address rule.

A MAC address rule assigns the VLAN depending on a single MAC address

 or on a range of MAC addresses.

Less frequently used are port and protocol rules. A port rule doesn’t require incoming traffic to trigger dynamic VLAN assignment. The specified mobile port is immediately assigned to the specified VLAN. Port rules only apply to outgoing broadcast traffic. You still need rules for the incoming traffic. To create a VLAN port rule:

A protocol rule uses the protocol type in an ethernet frame to assign VLANs to ports. Valid values for the port type are:

  • IP Ethernet-II
  • IP SNAP
  • Ethernet II
  • DECNet
  • AppleTalk
  • Ethertype
  • DSAP/SSAP
  • SNAP

A protocol rule is created by issuing

As always, the “no” form of the command removes the rule.

or

User Network Profiles (UNP) is a feature of Access Guardian. Access Guardian refers to security functions, like

  • Authentication and Classification
  • Host Integrity Check (HIC)
  • User Network Profiles (UNP), and
  • Virtual Network Profile (VPN)

UNP are available in AOS 6, AOS 7 and AOS 8. In AOS 6 we need a

  • policy condition
  • policy action
  • policy rule, and a
  • policy list

These four characteristics belong to the QoS feature of AOS. But a UNP needs a policy list, more specific the policy rules that are part of the policy list, to classify traffic and devices. The policy condition is necessary to identifiy a devices on which this policy should match.

Beside the MAC address, you can use source and destionation IP addresses, switch ports, source and destination TCP/ UDP ports, VLANs and many more. To use one or more IP addresses, simple use a network group.

The group “sales” consists of two subnets. To remove a subnet, use the “no” form of the command.

The policy action is used to determine, what should happen with the traffic. In this case: Priorize the traffic.

The rule binds condition and action.

A policy list is used to group one or more policy rules.

A UNP binds a name, a VLAN and a policy list together.

A third way to dynamically assign ports to VLANs is LLDP Media Endpoint Detection (LLDP-MED). LLDP Media Endpoint Detection was developed to increase the interoperability of VoIP devices with other devices on the network (eg. PC, switches etc). AOS uses LLDP-MED network policies to advertise information to devices. A network policy contains information about VLAN ID and L2/ L3 priorities. First, we have to enable network policy support, either for a port or for the chassis.

To enable network policies for the chassis use the keyword “chassis” instead of a port.

To create a network policy enter:

The created policy (ID 1) will advertise the VLAN 100, L2 priority 5 and DSCP 46 to voice devices. The next step is to bind the policy to a port or the chassis.

Or for a specific port.

Furthermore, you need to enable VLAN mobility on the ports. If the IP phones sends tagged VLAN frames, you also have to enable the “mobile tag” feature for the VLAN.

The IP phone receives the configuration information over of the network policy. VLAN mobility and “mobile tag” will make sure, that the VoIP phone is pushed to the correct VLAN.

Summary

Manually assigning VLANs can be a time consuming and error prone process. AOS 6 offers

  • VLAN mobility
  • User Network Profiles, and
  • LLDP-MED

to dynamically assign ports to VLANs. Each of the options has its pros and cons. Especially the combination of VLAN mobility and LLDP-MED is really easy to implement. I will publish more blog posts about the same topic, but with AOS 7 and AOS 8.

First steps with Python and pyVmomi (vSphere SDK for Python)

In December 2013, VMware made an christmas gift to the community by releasing pyVmomi. pyVmomi is a SDK that allows you to manage VMware ESXi and vCenter using Python and the VMware vSphere API. Nearly 18 months are past since then and pyVmomi has developed over time.

I’ve started to play around with Python, and I’ve written about the reasons in one of my last blog posts (Hey infrastructure guy, you should learn Python!).

How to get pyVmomi?

You can install the official release of pyVmomi using pip (pip installs packages, a recursive acronym).

The latest version is available on GitHub. To get the latest version, use

or

That you can fetch the latest version from GitHub is pretty cool and shows a big benefit: The community can contribute to pyVmomi and it’s more frequently updated. A huge benefit in regard of code quality and features.

What Python releases are support?

The latest information about supported Python releases can be found on the GitHub page of the project.

  • pyVmomi 6.0.0.2016.4 and later support 2.7, 3.3 and 3.4
  • pyVmomi 6.0.0 and later support 2.7, 3.3 and 3.4
  • pyVmomi 5.5.0-2014.1 and 5.5.0-2014.1.1 support Python 2.6, 2.7, 3.3 and 3.4
  • pyVmomi 5.5.0 and below support Python 2.6 and 2.7

Interesting fact: pyVmomi version numbers correlate with vSphere releases. pyVmomi 6.0.0 was released with the GA of VMware vSphere 6. pyVmomi supports the corresponding vSphere release and the previous four vSphere releases.

I’m using Python 3 for my examples. I wouldn’t recommend to start with Python 2 these days.

First steps

pyVmomi allows you to manage VMware ESXi and vCenter using Python and the VMware vSphere API. Because of this, the VMware vSphere API Reference Documentation will be your best friend.

First of all, you need a connection to the API. To connect to the vSphere API, we have to import and use the module pyVim, more precise, the pyVim.connect module and the SmartConnect function. pyVim.connect is used for the connection handling (creation, deletion…) to the Virtualization Management Object Management Infrastructure (VMOMI). pyVim is part of pyVmomi and it’s installed automatically.

SmartConnect accepts various parameters, but for the beginning it’s sufficient to use three of them: host, user and pwd. You can use “help(SmartConnect)” to get information about the SmartConnect function. “c” is the object (pyVmomi.VmomiSupport.vim.ServiceInstance) which we will use later.

A connection itself is useless. But how can we explore the API? Python doesn’t support typing, so it can be difficult to “explore” an API. That’s why the VMware vSphere API Reference Documentation and the Managed Object Browser (MOB) will be your best friends. The MOB is a web-based interface and represents the vSphere API. It allows you to navigate through the API. Any changes you make through the MOB, by invoking methods, take effect and change the config or will give you an output.

Important note: If you are using VMware vSphere 6 (ESXi 6.0 and vCenter 6.0), you have to enable the MOB. The MOB is disabled by default. Check VMware KB2108405 (The Managed Object Browser is disabled by default in vSphere 6.0) for more details.

Open a browser and open https://ip-or-fqdn/mob. You can use the IP address or the FQDN of an ESXi host or a vCenter Server (Appliance). I use a standalone ESXi 5.5 host in this example.

python_esxi_mob_1

Our first code

Let’s try something easy. I’ve framed a method in the screenshot above. We will use this method now.

This code will connect to the vSphere API, invoke the method “CurrentTime()” and prints the result. What happens if we execute our first lines of Python code? We will get an error…

Python checks SSL certificates in strict mode. Because of this, untrusted certificates will cause trouble. This applies to Python 3, as well as to Python >= 2.7.9 (PEP 0466). Most people use untrusted certificates. To deal with this, we have to create a context for our HTTP connection. This context can be used by the SmartConnect function. To create a context, we have to import the ssl module of Python.

“s” is the new object (ssl.SSLContext) we will use and the parameter “sslContext=s” will told SmartConnect to use this object.

Save this code into a file (I called it pyvmomitest.py in my example). Navigate to the folder, open a Python REPL and import the file you’ve saved (module) a moment ago.

Hurray! We used the vSphere API to get the current date and time (CET).

But what if we have deployed valid certificates? And what about housekeeping? We have connected, but we haven’t disconnected from the API? We can use a try-except block to handle this. And because we are nice, we import also the function “Disconnect” from pyVim.connect to disconnect from the vSphere API at the end.

With this code, we should get the following output.

Okay, the vSphere API wasn’t designed to retrieve the current date and time. Let’s look at something more useful. This script will give us the names of all VMs in the datacenter.

Let’s take this statement and look at everything after the “c”. We will use the MOB to navigate through the API. This will help you to understand, how the Python code and the structure of the vSphere API correlate.

Open the MOB. You will easily find  the property “content”.

python_esxi_mob_2

Click on “content” and search for the property “rootFolder”.

python_esxi_mob_3

Click on the value “ha-folder-root.” The property “childEntity” is an ManagedObjectReference (MOR) and references to all datacenters (the counting starts at 0) known to the ESXi or vCenter. The value “childEntity[0]” will give us the first datacenter.

python_esxi_mob_4

If we have the datacenter, the way to get the names of the VMs is the same. You can use the MOB, to verify this.

Click on the value “ha-datacenter”. At the bottom of the list, you will find the property “vmFolder”.

python_esxi_mob_5

Click on the value “ha-folder-vm”.

python_esxi_mob_6

The MOR “childEntity” references to two VMs. Click on one of the IDs.

python_esxi_mob_7

The property “name” includes the name of the VM. Because of this, we can use a simple

to get the name for each VM.

Summary

This was only a short introduction into pyVmomi. You should be now able to install pyVmomi, make a connection to the vSphere API and retrieve some basic stuff.

Every day I discover something new. It’s important to understand how the vSphere API works. Play with pyVmomi and with the vSphere API. It looks harder as it is.

btw: There is a Hands-on-Lab available “HOL-SDC-1622 VMware Development Tools and SDKs“. Check it out!

Hey infrastructure guy, you should learn Python!

I’m not a developer. I’m an infrastructure guy. All I ever needed was to write some scripts. Therefore, I never needed more than DOS batches, BASH/ CSH/ KSH, Visual Basic Script and nowadays PowerShell. So why should I learn another programming language?

One to rule them all?

I don’t think that there is a single programming language that is perfect for all use cases. The spread and acceptance of a language shows a positive correlation with the number of available frameworks, tools and libraries. That’s why I love the Microsoft PowerShell. Nearly all vendors offer a PowerShell module for their products (think about VMware PowerCLI, Rubrik, Veeam, DataCore and much more). The downside: The PowerShell code has to run on a Windows box. I think the time of writing DOS batches is over. UNIX shell scripts are still awesome, but focused on UNIX.

Different problems require different tools. I think it’s better to know a few, general-purpose tools well, as every conceivable special tool. Don’t get me wrong: PowerShell is awesome powerful! It’s quite easy to learn and you will have quick success.

Why Python?

Python is easy to learn (I can confirm this, at least for what I’ve seen). Python was developed from scratch by Guido van Rossum in the early 1990s. Python is an interpreted and dynamic programming language, which supports multiple paradigms, like the object-oriented or the functional programming. Python features a dynamic type system and automatic memory management. It uses only 35 keywords, what makes it easy to lern. It’s underlying philosophy is The Zen of Python.

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.

These rules lead to code with a high legibility, and it is possible to solve problems with fewer lines of code. Python is highly extensible. It comes with a large standard library and you can choose from 72.000 packages, that are available using the official 3rd party repository.

For me, as an infrastructure guy, the VMware vSphere API Python Bindings, the 3PAR Python client or the module for the Alcatel-Lucent Enterprise OmniSwitch RESTful API are reasons enough to start with Python. It’s the extensibility and platform independence of Python, what makes it so interesting. Like PowerShell, Python is an awesome language to automate things.

First steps with Python

Currently, the stabled releases are 2.7 and 3.5. I recommend to start with the 3.5 release. You can get the latest release from python.org. They offer packages for Windows, MacOS X and Linux/ UNIX. Python comes with an IDE called IDLE (Integrated Development and Learning Environment). Make sure that you take a look into the official documentation! If you want something more comfortable, try JetBrain PyCharm. JetBrains offer a free community edition for Windows, MacOS X and Linux. But it’s not the worst idea to start with IDLE. I use both IDEs, IDLE and PyCharm.

Where can you get help? YouTube is full of videos about Python. If you have a Pluralsight subscription, checkout the courses on Pluralsight. There are many good books out there, as well as some good howtos. Just use Google. It depends on what type of learner you are.

Learn the basics and try to strengthen them during a small project. Buy a Raspberry Pi. Raspberry Pi and Python are the biggest friends. If you are focused on VMware vSphere, take a closer look at the VMware vSphere API Python Bindings. Create yourself a project to learn.

I just started to learn Python, but I think that this wasn’t the worst idea in my life.

PowerCLI: Get-LunPathState

Careful preparation is a key element to success. If you restart a storage controller, or even the whole storage, you should be very sure that all ESXi hosts have enough paths to every datstore. Sure, you can use the VMware vSphere C# client or the Web Client to check every host and every datastore. But if you have a large cluster with a dozen datastores and some Raw Device Mappings (RDMs), this can take a looooong time. Checking the path state of each LUN is a task, which can be perfectly automated. Get a list of all hosts, loop through every host and every LUN, output a list of all hosts with all LUNs and all paths for each LUN. Sounds easy, right?

For a long time, I used this PowerCLI script for checking the LUN path state. But now I decided to give something back and I tweaked it a bit for my needs.

Feel free to use and/ or modify it.

Certificate-based authentication of Azure Automation accounts

Before you can manage Azure services with Azure Automation, you need to authenticate the Automation account against a subscription. This authentication process is part of each runbook. There are two different ways to authenticate against an Azure subscription:

  • Active Directory user
  • Certificate

If you want to use an Active Directory account, you have to create a credential asset in the Automation account and provide username and password for that Active Directory account. You can retrieve the credentials using the Get-AzureAutomationCredential cmdlet. This cmdlet returns a System.Management.Automation.PSCredential object, which can be used with Add-AzureAccount to connect to a subscription. If you want to use a certificate, you need four assets in the Automation account: A certificate and variables with the certificate name, the subscription ID and the subscription name. The values of these assets can be retrieved with Get-AutomationVariable and Get-AutomationCertificate.

Prerequisites

Before you start, you need a certificate. This certificate can be a self- or a CA-signed certificate. Check this blog post from Alice Waddicor if you want to start with a self-signed certificate. I used a certificate, that was signed by my lab CA.

At a Glance:

  • self- or CA-signed certificate
  • Base64 encoded DER format (file name extension .cer) to upload it as a management certificate
  • PKCS #12 format with private key (file name extension .pfx or .cer) to use it as an asset inside the Automation account

Upload the management certificate

First, you must upload the certificate to the management certificates. Login to Azure and click “Settings”.

creat_automation_account_01

Click on “Management Certificates”

creat_automation_account_02

and select “Upload” at the bottom of the website.

creat_automation_account_03

Make sure that the certificate has the correct format and file name extension (.cer).

creat_automation_account_04

Finish the upload dialog. After a few seconds, the certificate should appear in the listing.

creat_automation_account_06

Create a new Automation account

Now it’s time to create the Automation account. Select “Automation” from the left panel.

creat_automation_account_07

Click on “Create an Automation account”.

creat_automation_account_08

Give your Automation account a descriptive name and select a region. Please note that an Automation account can manage Azure services from all regions!

creat_automation_account_09

Click on the newly created account and click on “Assets”.

creat_automation_account_10

Select “Add setting” from the bottom of the website.

creat_automation_account_11

Add a credential asset by choosing “Add credential” and select “Certificate” as “Credential type”.

creat_automation_account_12

Enter a descriptive name for the certificate. You should remember this name. You will need it later. Now you have to upload the certificate. The certificate must have the file name extension .pfx or .cer and it must include the private key!

creat_automation_account_13

Finish the upload of the certificate. Now add three additional assets (variables).

creat_automation_account_14

Select the name, the value and the type from the  table below. The name of the certificate is the descriptive name, you’ve previously entered when uploading the certificate.

NameValueType
AutomationCertificateNameName of your certificateString
AzureSubscriptionNameName of your subscriptionString
AzureSubscriptionID36 digit ID of the subscriptionString

Done. You’ve uploaded and created all the required certificates and variables.

How to use it

To use the certificate and the variables to connect to an Azure subscription, you have to use the two cmdlets Get-AutomationCertificate and Get-AutomationVariable. I use this code block in my runbooks:

Works like a charm.

Summary

Certificate-based authentication is an easy way to authenticate an Automation account against an Azure subscription. It’s easy to implement and you don’t have to maintain users and passwords. You can use different certificates for different Automation accounts. I really recommend this, especially if you have separate accounts for dev, test and production.

All you need is to upload a certificate as a management certificates, and as a credential asset in the Automation account.  You can use a self- or CA-signed certificate. The subscription ID, the subscription name and the name of the certificate are stored in variables.

At the beginning of each runbook, you have to insert a code block. This code block takes care of authentication.