Tag Archives: vsphere

Missing hardware status tab in the vSphere Client

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

I thought, everyone knows it, but I’m always being asked “Where’s the hardware status tab?” after an update from vSphere 5.x to 6. Many customers still use the vSphere Client (C # client), and steer clear of the vSphere Web Client. To be honest: Me too. I often use the C# client, especially if I do mass operations, or for a quick look at something.

This is really nothing new, the answer is clear. But I think it’s a good idea to write it down. At least for myself. As a reminder to use the vSphere Web Client.

The hardware status tab

Many customers used the hardware status tab to get a quick overview about the health of the ESXi host hardware.

vsphere_client_hw_status_tab

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

But after an update to vSphere 6 the hardware status tab is missing in the vSphere Client. This is an expected behaviour! VMware has published an knowledge base article about this (The Hardware Status Tab is no longer available in the vSphere Client after upgrading to vCenter Server 6.0). The only solution is to use the vSphere Web Client.

Use the vSphere Web Client

Meanwhile, the old vSphere Client has many downsides. All features introduced in vSphere 5.5 and later are only available through the vSphere Web Client. This also applies to the hardware status tab.

vsphere_web_client_hw_status_tab

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

You find the hardware status on the “Monitor” tab of a host. It offers the same information as the legacy hardware status tab from the vSphere Client.

Do yourself a favor and use the vSphere Web Client. Always, any time.

First steps with Python and pyVmomi (vSphere SDK for Python)

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

In December 2013, VMware made an christmas gift to the community by releasing pyVmomi. pyVmomi is a SDK that allows you to manage VMware ESXi and vCenter using Python and the VMware vSphere API. Nearly 18 months are past since then and pyVmomi has developed over time.

I’ve started to play around with Python, and I’ve written about the reasons in one of my last blog posts (Hey infrastructure guy, you should learn Python!).

How to get pyVmomi?

You can install the official release of pyVmomi using pip (pip installs packages, a recursive acronym).

The latest version is available on GitHub. To get the latest version, use

or

That you can fetch the latest version from GitHub is pretty cool and shows a big benefit: The community can contribute to pyVmomi and it’s more frequently updated. A huge benefit in regard of code quality and features.

What Python releases are support?

The latest information about supported Python releases can be found on the GitHub page of the project.

  • pyVmomi 6.0.0.2016.4 and later support 2.7, 3.3 and 3.4
  • pyVmomi 6.0.0 and later support 2.7, 3.3 and 3.4
  • pyVmomi 5.5.0-2014.1 and 5.5.0-2014.1.1 support Python 2.6, 2.7, 3.3 and 3.4
  • pyVmomi 5.5.0 and below support Python 2.6 and 2.7

Interesting fact: pyVmomi version numbers correlate with vSphere releases. pyVmomi 6.0.0 was released with the GA of VMware vSphere 6. pyVmomi supports the corresponding vSphere release and the previous four vSphere releases.

I’m using Python 3 for my examples. I wouldn’t recommend to start with Python 2 these days.

First steps

pyVmomi allows you to manage VMware ESXi and vCenter using Python and the VMware vSphere API. Because of this, the VMware vSphere API Reference Documentation will be your best friend.

First of all, you need a connection to the API. To connect to the vSphere API, we have to import and use the module pyVim, more precise, the pyVim.connect module and the SmartConnect function. pyVim.connect is used for the connection handling (creation, deletion…) to the Virtualization Management Object Management Infrastructure (VMOMI). pyVim is part of pyVmomi and it’s installed automatically.

SmartConnect accepts various parameters, but for the beginning it’s sufficient to use three of them: host, user and pwd. You can use “help(SmartConnect)” to get information about the SmartConnect function. “c” is the object (pyVmomi.VmomiSupport.vim.ServiceInstance) which we will use later.

A connection itself is useless. But how can we explore the API? Python doesn’t support typing, so it can be difficult to “explore” an API. That’s why the VMware vSphere API Reference Documentation and the Managed Object Browser (MOB) will be your best friends. The MOB is a web-based interface and represents the vSphere API. It allows you to navigate through the API. Any changes you make through the MOB, by invoking methods, take effect and change the config or will give you an output.

Important note: If you are using VMware vSphere 6 (ESXi 6.0 and vCenter 6.0), you have to enable the MOB. The MOB is disabled by default. Check VMware KB2108405 (The Managed Object Browser is disabled by default in vSphere 6.0) for more details.

Open a browser and open https://ip-or-fqdn/mob. You can use the IP address or the FQDN of an ESXi host or a vCenter Server (Appliance). I use a standalone ESXi 5.5 host in this example.

python_esxi_mob_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Our first code

Let’s try something easy. I’ve framed a method in the screenshot above. We will use this method now.

This code will connect to the vSphere API, invoke the method “CurrentTime()” and prints the result. What happens if we execute our first lines of Python code? We will get an error…

Python checks SSL certificates in strict mode. Because of this, untrusted certificates will cause trouble. This applies to Python 3, as well as to Python >= 2.7.9 (PEP 0466). Most people use untrusted certificates. To deal with this, we have to create a context for our HTTP connection. This context can be used by the SmartConnect function. To create a context, we have to import the ssl module of Python.

“s” is the new object (ssl.SSLContext) we will use and the parameter “sslContext=s” will told SmartConnect to use this object.

Save this code into a file (I called it pyvmomitest.py in my example). Navigate to the folder, open a Python REPL and import the file you’ve saved (module) a moment ago.

Hurray! We used the vSphere API to get the current date and time (CET).

But what if we have deployed valid certificates? And what about housekeeping? We have connected, but we haven’t disconnected from the API? We can use a try-except block to handle this. And because we are nice, we import also the function “Disconnect” from pyVim.connect to disconnect from the vSphere API at the end.

With this code, we should get the following output.

Okay, the vSphere API wasn’t designed to retrieve the current date and time. Let’s look at something more useful. This script will give us the names of all VMs in the datacenter.

Let’s take this statement and look at everything after the “c”. We will use the MOB to navigate through the API. This will help you to understand, how the Python code and the structure of the vSphere API correlate.

Open the MOB. You will easily find  the property “content”.

python_esxi_mob_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click on “content” and search for the property “rootFolder”.

python_esxi_mob_3

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click on the value “ha-folder-root.” The property “childEntity” is an ManagedObjectReference (MOR) and references to all datacenters (the counting starts at 0) known to the ESXi or vCenter. The value “childEntity[0]” will give us the first datacenter.

python_esxi_mob_4

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

If we have the datacenter, the way to get the names of the VMs is the same. You can use the MOB, to verify this.

Click on the value “ha-datacenter”. At the bottom of the list, you will find the property “vmFolder”.

python_esxi_mob_5

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click on the value “ha-folder-vm”.

python_esxi_mob_6

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The MOR “childEntity” references to two VMs. Click on one of the IDs.

python_esxi_mob_7

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The property “name” includes the name of the VM. Because of this, we can use a simple

to get the name for each VM.

Summary

This was only a short introduction into pyVmomi. You should be now able to install pyVmomi, make a connection to the vSphere API and retrieve some basic stuff.

Every day I discover something new. It’s important to understand how the vSphere API works. Play with pyVmomi and with the vSphere API. It looks harder as it is.

btw: There is a Hands-on-Lab available “HOL-SDC-1622 VMware Development Tools and SDKs“. Check it out!

HPE Hyper Converged 380 – A look under the hood

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

In March 2016, HPE CEO Meg Whitman announced a ProLiant-based HCI solution, that should be easier to use and cheaper than Nutanix.

This isn’t HPEs first dance on this floor. In August 2015, HP launched the Hyper Converged 250 System (HC250), which is based on the Apollo server platform. The HW design of the HC250 comes close to a Nutanix Block, because the Apollo platform supports up to four nodes in 2U. Let me say this clear: The Hyper Converged 380 (HC380) is not a replacement for the HC250! And before the HC250, HPE offered the Converged System 200-HC StoreVirtual and 200-HC EVO:RAIL (different models).

The HC380 is based on the ProLiant DL380 Gen9 platform. The DL380 Gen9 is one of the, if not the best selling x86 server on the market. Instead of developing everything from scratch, HPE build their new HC380 from different already available HPE products. With one exception: HPE OneView User Experience (UX). IT was developed from scratch and consolidates all management and monitoring tasks into a single console. The use of already available components was the reason for the low time-to-market (TTM) of the HC380.

Currently, the HC380 can only run VMware vSphere (HPE CloudSystem uses VMware vSphere). Support for Microsoft Hyper-V and Citrix XenServer will be added later. If you wish to run Microsoft Hyper-V, check the HC250 or wait until it’s supported with the HC380.

What flavor would you like?

The HC380 is available in three editions (use cases):

  • HC380 (Virtualization)
  • HC380 (HPE CloudSystem)
  • HC380 (VDI)

All three use cases are orderable using a single SKU and include two DL380 Gen9 nodes (2U). You can add up to 14 expansion nodes, so that you can have up to 16 dual-socket DL380 Gen9.

Each node comes with two Intel Xeon E5 CPUs. The exact CPU model has to be selected before ordering. The same applies to the memory (128 GB or 256 GB per node, up to 1,5 TB) and disk groups (up to three disk groups, each with 4,5 to 8 TB usable capacity per block, 8 drives either SSD/ HDD or all HDD with a maximum of 25 TB usable per node). The memory and disk group configuration depends on the specific use case (virtualization, CloudSystem, VDI). The same applies to the number of network ports (something between 8x 1 GbE and 6x 10 GbE plus 4x 1 GbE). For VDI, customers can add NVIDIA GRID K1, GRID K2 or Telsa M60 cards.

VMware vSphere 6 Enterprise or Enterprise Plus are pre-installed and licences can be bought from HPE. Interesting note from the QuickSpecs:

NOTE: HPE Hyper Converged 380 for VMware vSphere requires valid VMware vSphere Enterprise or higher, and vCenter licenses. VMware licenses can only be removed from the order if it is confirmed that the end-customer has a valid licenses in place (Enterprise License Agreement (ELA), vCloud Air Partner or unused Enterprise Purchasing Program tokens).

Hewlett Packard Enterprise supports VMware vSphere Enterprise, vSphere Enterprise Plus and Horizon on the HPE Hyper Converged 380.

No support for vSphere Standard or Essentials (Plus)! Let’s see how HPE will react on the fact, that VMware will phase out vSphere Enterprise licenses.

The server includes 3y/ 3y/ 3y onsite support with next business day response. Nevertheless, at least 3-year HPE Hyper Converged 380 solution support is requires according to the latest QuickSpecs.

What’s under the hood?

As I already mentioned, the HC380 was built from well known HPE products. Only HPE OneView User Experience (UX) was developed from scratch. OneView User Experience (UX) consolidates the following tasks into a single console (source QuickSpecs):

  • Virtual machine (VM) vending (create, edit, delete)
  • Hardware/driver and appliance UI frictionless updates
  • Advanced capacity and performance analytics (optional)
  • Backup and restore of appliance configuration details
  • Role-based access
  • Integration with existing LDAP or Active Directory
  • Physical and virtual hardware monitoring

Pretty cool fact: HPE OneView User Experience (UX) will be available for the HC250 later this year. Part of a 2-node cluster are not only the two DL380 Gen9 servers, but also three VMs:

  • HC380 Management VM
  • HC380 OneView VM
  • HC380 Management UI VM

The Management VM is used for VMware vCenter (local install) and HPE OneView for vCenter. You can use a remote vCenter (or a vCenter Server Appliance), but you have to make sure that the remote vCenter has HPE Oneview for vCenter integrated. The OneView VM running HPE OneView for for HW/ SW management. The Management UI VM is running HPE OneView User Experience.

The shared storage is provided by HPE StoreVirtual VSA. A VSA is running on each node. As you might know, StoreVirtual VSA comes with an all-inclusive license. No need to buy additional licenses. You can have it all: Snapshots, Remote Copy, Clustering, Thin Provisioning, Tiering etc. The StoreVirtual VSA delivers sustainable performance, a good VMware vSphere integration and added value, for example support for Veeam Storage Snapshots.

When dealing with a 2-node cluster, the 25 TB usable capacity per node means in fact 25 TB usable for the whole 2-node cluster. This is because of the Network RAID 1 between the two StoreVirtual VSA. The data is mirrored between the VSAs. When adding more nodes, the data is striped accross the nodes in the cluster (Network RAID 10+2).

Also important in case of the 2-node cluster: The quorum. At least two StoreVirtual VSA build a cluster. As in every cluster, you need some kind of quorum. StoreVirtual 12.5 added support for a NFSv3 based quorum witness. This is in fact a NFS file share, which has to be available for both nodes. This is only supported in 2-node clusters and I highly recommend to use this. I have a customer that uses a Raspberry Pi for this…

Start the engine

You have to meet some requirements before you can start.

  • 1 GbE connections for each nodes iLO and 1 GbE ports
  • 1 GbE or 10 GbE connections for each node FlexLOM ports
  • Windows-based computer directly connected to a node (MacOS X or Linux should also work)
  • VMware vSphere Enterprise or Enterprise Plus licenses
  • enough IP addresses and VLANs (depending on the use case)

For general purpose server virtualization, you need at least three subnets and three VLANs:

  • Management
  • vMotion
  • Storage (iSCSI)

Although you have the choice between a flat (untagged) and a VLAN-tagged network design, I would always recomment a VLAN-tagged approach. It’s highly recommended to use multiple VLANs to get the traffic seperated. The installation guide includes worksheets and examples to help you planning the deployment. For a 2-node cluster you need at least:

  • 5 IP addresses for the management network
  • 2 IP addresses for the vMotion network
  • 8 IP addresses for the iSCSI storage network

You should leave space for expansion nodes. A proper planning saves you later trouble.

HP OneView InstantOn is used for the automated deployment. It guides you through the necessary configuration steps. HPE says that the deployment requires less than 60 minutes and all you need to enter are

  • IP addresses
  • credentials
  • VMware licenses

After the deployment, you have to install the StoreVirtual VSA licenses. Then you can create datastores and, finally, VMs.

hpehc380_ux

HPE/ hpw.com

Summary

Hyper-Converged has nothing to do with the form factor. Despite the fact that a 2-node cluster comes in 4U, the HC380 has everything you would expect from a HCIA. The customers will decide if HPE held promise. The argument for the HC380 shouldn’t be the lower price compared to Nutanix or other HCI players. Especially, HPE should not repeat the mistake of the HC200 EVO:RAIL: To buggy and to expensive. The HC380 combines known and mature products (ProLiant DL380 Gen9, StoreVirtual VSA, OneView). It’s now up to HPE.

I have several small and mid-sized customers that are running two to six nodes VMware vSphere environments. Also the HC380 for VDI can be very interesting.

VMware Update Manager reports “error code 99” during scan operation

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

After updating my lab to VMware vSphere 6.0 U2, one of my hosts continuously thrown an error during an update scan.

The first thing I’ve checked was the esxupdate.log on the affected ESXi host. This is the output, that was logged during a scan operation.

You might notice the “Unrecognized file vendor-index.xml in Metadata file” error. I also found this error message on the other hosts, so I excluded it from further research. It was unlikely, that this error was related to the observed problem. I started searching differences between the hosts and found out, that the output of “esxcli software vib list” was different on the faulty host.

This is the output on the faulty host:

This is the output on a working host. You see the difference?

Doesn’t look right… I investigated further, still searching for differences. And then I found two empty directories under /var/db/esximg.

The same directory was populated on other, working hosts.

One possible solution was therefore to copy the missing files to the faulty host. I used SCP for this. To get SCP working, you have to enable the SSH Client in the ESXi firewall.

After that, I’ve copied the files from a working host to the faulty host. Please make sure that the hosts have the same build! In my case, both hosts had the same build. Don’t try to copy files from an older or newer build to the host!!

And because we are pros, we disable the SSH Client after using it.

As expected, “esxcli software vib list” was working again.

A rescan operation in the vSphere Client was also successful. It seems that the root cause for the problem were missing files under /var/db/esximg.

Please don’t ask why this has happened. I really have no idea. But VMware KB2043170 (Initializing the VMware vCenter Update Manager database without reinstalling it) isn’t always the solution for “error code 99”, as sometimes written somewhere in the internet. Always try to analyze the problem and try to filter out unlikely and likely solutions.

VMware Horizon View space reclamation fails due to activated CBT

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Nearly two weeks ago, I wrote a blog post (VMware Horizon View space reclamation fails) about failing space reclamation on automated desktop pools with linked clones. Today I write about the same error, but caused by another problem. In both cases, the error is logged in the View Administrator and the vSphere (Web) Client. On the View Administrator, the following error is shown:

“Failed to perform space reclamation on machine COMPUTER NAME in Pool POOL NAME”

view_admin_reclaim_error

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The vSphere Web Client shows a different error message:

“A general system error occurred: Wipe Disk failed: Failed to complete wipe operation.”

web_client_wipe_disk_error

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

If a issue with the permissions of the initiating user can be excluded, you should check if Change Block Tracking (CBT) is enabled for the parent VM and the linked clones. The easiest way to check this, is the vSphere Web Client.

web_client_adv_settings

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Highlight the VM > Manage > VM Options > Advanced settings > Configuration parameters.

Check the configuration parameters for:

To solve this issue, power off the parent VM, remove all snapshots and change the following advanced parameters:

Then take a new snapshot and recompose the desktop pool.

VMware has documented this in VMware KB2032214 (Deploying or recomposing View desktops fails when the parent virtual machine has CBT enabled). Even if the name of the article doesn’t mention failing space reclamation, it’s mentioned as one possible symptom.

VMware Horizon View space reclamation fails

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

A customer notified me, that he observed an issue with the space reclamation on two automated desktop pools with linked clones. His environment is based on Horizon View 6.2.1 and vSphere 5.5U3. The error was logged in the View Administrator and the vSphere (Web) Client. In the View Administrator, the following error was visible:

“Failed to perform space reclamation on machine COMPUTER NAME in Pool POOL NAME”

view_admin_reclaim_error

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

In the vSphere Web Client, the error messages was different.

“A general system error occurred: Wipe Disk failed: Failed to complete wipe operation.”

web_client_wipe_disk_error

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

As you can see, a service account was created for view management tasks. A custom user role was also created and this role was assigned to the service account. The necessary privileges were taken from the Horizon View documentation (Privileges Required for the vCenter Server User). There is a nice table, that lists all necessary privileges. All necessary privileges? No, one important privilege was missing.

I’ve tried to reproduce this issue in my lab. Unfortunately, the space reclamation was working in my lab. To be honest: I’m not using a specific service account and a custom user role in my lab. I’m using a domain admin account, that has the “Administrator” role assigned in the vCenter. I searched a bit in the VMware Knowledge Base, but I was unable to find a suitable KB entry for my problem. I re-created the view manager user role in my lab and changed the role for the “Administrator” account. After that, I was able to reproduce the problem. So it was clear, that the role was the problem. I checked the privileges again and found an interesting privilege, that was not listed in the VMware documentation.

web_client_composer_missing_right

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

You can find this privilege under: All Privileges > Virtual Machine > Interaction > Perform wipe or shrink operations.

Setting this privilege solved the observed issue. With this knowledge in mind, I found a blog post from Andy Barnes, that describes this issue and the solution.

Guest customizations fails after upgrade to VMware vSphere 6

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

VMware vSphere 6 is now an year old and it was time to update my lab to vSphere 6. The update went smooth, and everything has worked as expected. Some days later, I updated the master VM of a small automated desktop pool. I’m using VMware Horizon 6.2.1 in my lab to deploy a small number of Windows 8.1 VMs for tests, administration etc. The recompose of the pool failed during the guest customization.

view_error_decrypt_password

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I checked the customization specification immediately and got an error in the vSphere C# client.

vcsa_error_decrypt_password

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Interestingly, I got no error in the vSphere Web Client:

vcsa_error_decrypt_password_web_client

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

After re-entering the Administrator password, the  customization specification was usable again. No errors so far.

A quick search in the VMware KB lead me to the article “Virtual machines with customizations fail to deploy when using Custom SSL Certificates (1019893)“. But this article doesn’t apply to vCenter 6.0. For the notes: I’m using CA-signed certificates in my environment. It seems to be a good idea to re-enter the passwords in customization specifications after a vCenter migration/ upgrade (5.x to 6.x or from VCSA 5.x to 6.x).

VMware vCenter Storage Monitoring Service & Auto Deploy plug-in failed after upgrade to vSphere 6.0

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Yesterday I did an upgrade of my vCenter Server Appliance 5.5 U3 to 6.0 U1. This was the first step to update my lab infrastructure to vSphere 6.0. A bit late, but better late than never. The update of the VCSA itself went smooth. No problems with certificates, hosts, VMs or PernixData FVP. But then, I discovered two errors on the old vSphere C# client (I know that I should use the Web Client…)

vsphere_client_plug-in_error

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The first message indicates a problem with the VMware vCenter Storage Monitoring Service.

The other error message indicates a problem with Auto Deploy.

Both error messages have harmless reasons.

VMware vCenter Storage Monitoring Service

The error message regarding the VMware vCenter Storage Monitoring Service is an expected behaviour. The answer to this is written in the VMware vCenter Server 6.0 Release Notes.

  • vSphere Web Client. The Storage Reports selection from an object’s Monitor tab is no longer available in the vSphere 6.0 Web Client.
  • vSphere Client. The Storage Views tab is no longer available in the vSphere 6.0 Client.

To get rid of this message, simply ignore it or remove the old vSphere C# client. Newer releases of the C# client don’t have this plugin. If you need older C# client releases, you can ignore this error message.

Auto Deploy

The error message regarding Auto Deploy is caused by a stopped Auto Deploy service. The Auto Deploy service is stopped by default.

vcenter_6_auto_deploy

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

You can find the Auto Deploy service in the vSphere Web Client under Administration > System Configuration > Services > Auto Deploy. The error message will be gone, if you start the service. If you don’t need Auto Deploy, you can ignore this error message.

Marco van Baggum wrote a blog post about the same messages some months ago. He highlighted an alternative way to get rid of these messages.

HP ProLiant BL460c Gen9: MicroSD card missing during ESXi 5.5 setup

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today, I was at a customer to prepare a two node vSphere cluster for some MS SQL server tests. Nothing fancy, just two HP ProLiant BL460c Gen9 blades and two virtual volumes from a HP 3PAR. Each blade had two 400 GB SSDs, two 64 GB M.2 SSDs and a 1 GB MicroSD card. Usually, I install ESXi to a SD card. In this case, a MicroSD card. The SSDs were dedicated for PernixData FVP. Although I saw the MicroSD card in the boot menu, ESXi doesn’t showed it as a installation target.

bl460_no_sdcard

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I’ve read a lot about similar observations of other users, but no solution seemed to be plausible. It’s not a solution to switch from UEFI to legacy BIOS or play with power management settings. BIOS and ILO firmware were up to date. But disabling USB3 support seemed to be a possible solution.

You can disable USB3 support in the RBSU.

bl460_rbsu

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

After a reboot, the MicroSD card appeared as installation target during the ESXi setup.

bl460_with_sdcard

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I never noticed this behaviour with ProLiant DL360 Gen9, 380 Gen9, 560 Gen9 or 580 Gen9. I only saw this with BL460c Gen9. The affected blade servers had System ROM I36 05/06/2015 and I36 09/24/2015, as well as ILO4 2.30 installed.

Storage vMotion stuck at 100% – cleaning up migration state

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Moving VMs from an old cluster with old ESXi hosts to a new cluster with new hosts can be so easy, even if the clusters doesn’t share any storage. A PowerCLI one-liner or the Web Client allow you to migrate VMs between hosts and datastores, while the VMs are running. This enhancement was added with vSphere 5.1. I’m often suprised how many customers doesn’t know this feature, just because they are still using the old vSphere C# client.

Some days ago, I had to move some VMs to a new cluster and I used this well known feature to move the running VMs to a new hosts. Everything was fine, until I realized, that the vMotion process stopped at 100%. Usually, the cleanup is finishes very quickly. But this time, the vMotion stopped.

Poking around in the fog

After waiting 15 minutes, I started investigating what is going on. One of the first things I discovered were large vmware.log files in the working directories of all running VMs. Each VM had a vmware.log with up to 8 GB. With lsof, I found a vpxa-worker process that was accessing the vmware.log of the currently moved VM. This corresponded with a slow file transfer (~ 5 MB/s), that was running for more than 15 minutes between the ESXi hosts. The vmware.log itself was full of VMware Tools debug messages. I checked the guest OS (Windows Server 2008 R2) and found a tools.conf (C:\ProgramData\VMware\VMware Tools\tools.conf) with this content:

Looks like someone has enabled the debugging mode for the VMware Tools… more then 18 months ago. I’ve changed the log level to “error”, copied the tools.conf to all VMs and did a restart of the VMware Tools service on each VM. This stopped the growth of the vmware.log files immediately.

But how to deal with the vmware.log files? You can instantly “shrink” the vmware.log file. All you need is SSH access to the ESXi host, that is running the VM.

Use the cp command to overwrite the vmware.log file. As you can see, the log file has then a size of 0 bytes. You don’t have to shutdown the VM for this. But you lose all the data stored in the vmware.log file!

Last words

You should ALWAYS disable debug logging, after you have the data you want. Never enable debug logging permanently!