Tag Archives: esxi

HP ProLiant BL460c Gen9: MicroSD card missing during ESXi 5.5 setup

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today, I was at a customer to prepare a two node vSphere cluster for some MS SQL server tests. Nothing fancy, just two HP ProLiant BL460c Gen9 blades and two virtual volumes from a HP 3PAR. Each blade had two 400 GB SSDs, two 64 GB M.2 SSDs and a 1 GB MicroSD card. Usually, I install ESXi to a SD card. In this case, a MicroSD card. The SSDs were dedicated for PernixData FVP. Although I saw the MicroSD card in the boot menu, ESXi doesn’t showed it as a installation target.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I’ve read a lot about similar observations of other users, but no solution seemed to be plausible. It’s not a solution to switch from UEFI to legacy BIOS or play with power management settings. BIOS and ILO firmware were up to date. But disabling USB3 support seemed to be a possible solution.

You can disable USB3 support in the RBSU.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

After a reboot, the MicroSD card appeared as installation target during the ESXi setup.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I never noticed this behaviour with ProLiant DL360 Gen9, 380 Gen9, 560 Gen9 or 580 Gen9. I only saw this with BL460c Gen9. The affected blade servers had System ROM I36 05/06/2015 and I36 09/24/2015, as well as ILO4 2.30 installed.

Storage vMotion stuck at 100% – cleaning up migration state

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Moving VMs from an old cluster with old ESXi hosts to a new cluster with new hosts can be so easy, even if the clusters doesn’t share any storage. A PowerCLI one-liner or the Web Client allow you to migrate VMs between hosts and datastores, while the VMs are running. This enhancement was added with vSphere 5.1. I’m often suprised how many customers doesn’t know this feature, just because they are still using the old vSphere C# client.

Some days ago, I had to move some VMs to a new cluster and I used this well known feature to move the running VMs to a new hosts. Everything was fine, until I realized, that the vMotion process stopped at 100%. Usually, the cleanup is finishes very quickly. But this time, the vMotion stopped.

Poking around in the fog

After waiting 15 minutes, I started investigating what is going on. One of the first things I discovered were large vmware.log files in the working directories of all running VMs. Each VM had a vmware.log with up to 8 GB. With lsof, I found a vpxa-worker process that was accessing the vmware.log of the currently moved VM. This corresponded with a slow file transfer (~ 5 MB/s), that was running for more than 15 minutes between the ESXi hosts. The vmware.log itself was full of VMware Tools debug messages. I checked the guest OS (Windows Server 2008 R2) and found a tools.conf (C:\ProgramData\VMware\VMware Tools\tools.conf) with this content:

log = true

# Enable tools service logging to vmware.log
vmsvc.level = debug
vmsvc.handler = vmx

# Enable new “vmusr” service logging to vmware.log
vmusr.level = error
vmusr.handler = vmx

# Enable “Volume Shadow Copy” service logging to vmware.log
vmvss.level = debug
vmvss.handler = vmx

Looks like someone has enabled the debugging mode for the VMware Tools… more then 18 months ago. I’ve changed the log level to “error”, copied the tools.conf to all VMs and did a restart of the VMware Tools service on each VM. This stopped the growth of the vmware.log files immediately.

But how to deal with the vmware.log files? You can instantly “shrink” the vmware.log file. All you need is SSH access to the ESXi host, that is running the VM.

/vmfs/volumes/f222c18e-31aad6e9/veeam # ls -l vmware.log
-rwxrwxrwx    1 1024     users       118531 Jan 14 08:07 vmware.log
/vmfs/volumes/f222c18e-31aad6e9/veeam # ls -lh vmware.log
-rwxrwxrwx    1 1024     users     115.8K Jan 14 08:07 vmware.log
/vmfs/volumes/f222c18e-31aad6e9/veeam # cp /dev/null vmware.log
/vmfs/volumes/f222c18e-31aad6e9/veeam # ls -lh vmware.log
-rwxrwxrwx    1 1024     users          0 Jan 14  2016 vmware.log
/vmfs/volumes/f222c18e-31aad6e9/veeam #

Use the cp command to overwrite the vmware.log file. As you can see, the log file has then a size of 0 bytes. You don’t have to shutdown the VM for this. But you lose all the data stored in the vmware.log file!

Last words

You should ALWAYS disable debug logging, after you have the data you want. Never enable debug logging permanently!

ESXi 5.5 U3b and later are no longer manageable without vCenter 5.5 U3b

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

On December 8, 2015, VMware released VMware ESXi 5.5 patch ESXi550-201512001 (2135410). This patch is known as U3b and it contains general and security fixes, nothing special. Usually, you would install this update without notice. But this time, you should better take a look into the release notes of ESXi 5.5 U3b, before you install this update. This is taken from the release notes:

Note: In your vSphere environment, you need to update vCenter Server to vCenter Server 5.5 Update 3b before updating ESXi to ESXi 5.5 Update 3b. vCenter Server will not be able to manage ESXi 5.5 Update 3b, if you update ESXi before updating vCenter Server to version 5.5 Update 3b. For more information about the sequence in which vSphere environments need to be updated, refer, KB 2057795.

To avoid problems, you have to update your vCenter first, then update your hosts. This sequence is recommended by VMware and I recommend to take a look into KB 2057795 (Update sequence for VMware vSphere 5.5 and its compatible VMware products)

Reset the HP iLO Administrator password with hponcfg on ESXi

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Sometimes you need to reset the ILO Administrator password. Sure, you can reboot the server, press F8 and then reset the Administrator password. If you have installed a HP customized ESXi image, then there is a much better way to reset the password: HPONCFG.

Check the /opt/hp/tools directory. You will find a binary called hponcfg.

~ # ls -l /opt/hp/tools/
total 5432
-r-xr-xr-x 1 root root 5129574 Oct 28 2014 conrep
-r--r--r-- 1 root root 108802 Oct 28 2014 conrep.xml
-r-xr-xr-x 1 root root 59849 Jan 16 2015 hpbootcfg
-r-xr-xr-x 1 root root 251 Jan 16 2015 hpbootcfg_esxcli
-r-xr-xr-x 1 root root 232418 Jul 14 2014 hponcfg
-r-xr-xr-x 1 root root 12529 Oct 31 2013 hptestevent
-r-xr-xr-x 1 root root 250 Oct 31 2013 hptestevent_esxcli

All you need is a simple XML file. You can use the VI editor or you can copy the necessary file with WinSCP to the root home directory on your ESXi host. I prefer VI. Change the directory to /opt/hp/tools. Then open the pwreset.xml.

~ # vi pwreset.xml

Press i to switch to the insert mode. Then paste this content into the file. You don’t have to know the current password!

<LOGIN USER_LOGIN="Administrator" PASSWORD="unknown">
<USER_INFO MODE="write">
<MOD_USER USER_LOGIN="Administrator">
<PASSWORD value="password"/>

Press ESC and then :wq<ENTER> to save the file and leave the VI. Now use HPONCFG together with the XML file to reset the password.

~ # /opt/hp/tools/hponcfg -f pwreset.xml
HP Lights-Out Online Configuration utility

Version 4.4-0 (c) Hewlett-Packard Company, 2014
Firmware Revision = 1.85 Device type = iLO 3 Driver name = hpilo
iLO IP Address:
Script succeeded

That’s it! You can now login with “Administrator” and “password”.

VMware publishes patch for ESXi 6.0 CBT bug (KB2114076)

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today, VMware has published the long-awaited patch for the ESXi 6.0 CBT bug. This patch is the result of a problem, which is described in KB2114076 (Backing up a virtual machine with Change Block Tracking (CBT) enabled fails after upgrading to or installing VMware ESXi 6.0). All customers that upgraded to ESXi 6.0 or installed ESXi 6.0 were affected.

Symptoms of this bug were:

  • Powering on virtual machines fails
  • Expanding the size of a virtual disk fails
  • Taking virtual machine quiesced snapshots fails
  • Error messages like “An error occurred while taking a snapshot: msg.snapshot.error-QUIESCINGERROR” (vSphere Client), “WARNING: CBT: 191: No memory available! Called from 0x4180219af50e” (vmkernel.log) or “Creating cbt node 92b78c-cbt failed with error Cannot allocate memory (0xbad0014, Out of memory)” in the vmware.log of the affected virtual machine

A workaround was to disable CBT, which resulted in longer running backups.

You can download the patch (ESXi600-201505401-BG) from VMware to manually update your ESXi hosts. Otherwise you can use VMware Update Manager to patch your hosts. A reboot is necessary!

If you decide to manually update your hosts with this patch, you have to use the “esxcli software vib” command to install the patch.

  • Download the ZIP file and upload it to a datastore that is reachable for the host you want to patch (e.g. a local datastore)
  • Bring the host into the maintenance mode
  • Connect with SSH to your ESXi host and run:
esxcli software vib install -d "/vmfs/volumes/datastore1/patch-directory/ESXi600-201505001.zip"
  • Reboot the host and leave the maintenance mode

If you have disabled CBT for all or some of your VMs, make sure that you re-enable CBT.

What to consider when implementing HP 3PAR with iSCSI in VMware environments

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Some days ago a colleague and I implemented a small 3-node VMware vSphere Essentials Plus cluster with a HP 3PAR StoreServ 7200c. Costs are always a sore point in SMB environments, so it should not surprise that we used iSCSI in this design. I had some doubt about using iSCSI with a HP 3PAR StoreServ, mostly because of the performance and complexity. IMHO iSCSI is more complex to implement then Fibre Channel (FC). But in this case I had to deal with it.

iSCSI options for HP 3PAR StoreServ 7000

If you decide to use iSCSI with a HP 3PAR StoreServ, you have only one option: Adding a 2-port 10GbE iSCSI/ FCoE adapter to each node. There is no other iSCSI option. The available 2-port 10GbE ethernet adapter and 4-port 1GbE ethernet adapter can’t be used for iSCSI connectivity. These adapters can only be used with the HP 3PAR File Persona Software Suite.

The 2-port 10GbE iSCSI/ FCoE adapter is a converged network adapter (CNA) and supports iSCSI or Fibre Channel over Ethernet (FCoE). The adapter can only be used for host connectivity and you have to select iSCSI or FCoE. You can’t use the CNA for remote copy. You have to add a CNA to each nodes in a node pair. You can have up to four 10 GbE ports in a 3PAR 7200 series, or up to eight 10 GbE ports in a 3PAR 7400 series.

Network connectivity

10 GbE means 10 GbE, there is no way to connect the CNA to 1 GbE transceivers. The 2-port 10GbE iSCSI/ FCoE includes two 10 GbE SR SFP+ transceivers. With 3PAR OS 3.1.3 and later, you can use Direct Attach Copper (DAC) cables for network connectivity, but not for FCoE. Make sure that you use the correct cables for your switch! HP currently offers the following cables in different length:

  • X242 for HP ProVision switches
  • X240 for HP Comware switches
  • HP B-series SFP+ to SFP+ Active Copper for Brocade switches, or
  • HP C-series SFP+ to SFP+ Active Copper for Cisco switches

If you use any other switch vendor, I strongly recommend to use the included 10 GbE SR SFP+ transceivers and 10 GbE SR SFP+ transceivers on the switch-side. In this case you have to use fiber cable to connect the 3PAR to the network. In any other case I recommend to use DAC for network connectivity.

It’s a common practice to run iSCSI traffic in its own VLAN. Theoretically a single iSCSI VLAN is sufficient. I recommend to use two iSCSI VLANs in case of a 3PAR, one for each iSCSI subnet. Why two subnets? The answer is easy: Persistent Ports. Persistent Ports allows an host port to assume the identity (port WWN for Fibre Channel or IP address for iSCSI ports) of a failed port while retaining its own identity. This minimizes I/O disruption during failures or upgrade. Persistent Ports uses the NPIV feature for Fibre Channel/ FCoE and IP address failover for iSCSI. With the release of 3PAR OS 3.1.3, Persistent Ports was also available for iSCSI. A hard requirement of Persistent Ports is, that the same host ports of nodes of a node pair must be connected to the same IP network on the fabric. An example clarifies this:

Host port (N:S:P)VLAN IDIP subnet

The use of jumbo frames with iSCSI is a often discussed topic. It’s often argued that complexity and performance gain would be disproportionate. I’m a bit biased. I think that the use of jumbo frames is a must when using iSCSI. I always configure jumbo frames for vMotion, so the costs for configuring Jumbo frames is low for me in an iSCSI environment. Don’t forget to configure jumbo frames on all devices in the path: VMkernel ports, vSwitches, physical switches and 3PAR CNAs.

Always use at least two physical switches for iSCSI connectivity. This concept is compareable to a Fibre Channel dual-fabric SAN. I like the concept of switch aggegration (the wording may vary between vendors). I often work with HP Networking and I like the HP 2920 or 5820 Switch Series. These switches can form stacks in which multiple physical switches act a as a single virtual device. These stacks provide redundancy and operational simplicity. In combination with two VLANs you can build a powerful, redundant and resilient iSCSI SAN.

Host port configuration

The CNA ports can only be used for host connectivity, therefore there is no way to use them for disk or remote copy connectivity. Before you can use the port for host connectivity, you have to select iSCSI or FCoE as storage protocol.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Host and Virtual Volume sets

You can organize hosts and volumes in host sets and volume sets. I recommend to create a host set for all ESXi hosts in a vSphere cluster. I also recommend to create a volume set to group all volumes that should be presented to a host or host set. When exporting Virtual Volumes (VV), you can export a volume set to a host set. If you add a host to the host set, the host will see all volumes in the volume set. If you add a volume to a volume set, the hosts in the host set will all see the newly added volume. This simplifies host and volume management and it reduced the possibilty of human errors.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Custom SATP rule for ESXi 5.x and ESXi 6.0

3PAR OS 3.1.2 introduced the new Host Persona 11 for VMware which enables asymmetric logical unit access (ALUA). Beside Host Persona 11, Host Persona 6 for VMware is also available, but it doesn’t support ALUA. 3PAR OS 3.1.3 is the last release that included support Host Persona 6 AND 11 for VMware. All later releases only include Host Persona 11. I strongly recommend to use Host Persona 11 for VMware. You should also add a custom SATP rule. This rule can be added by using ESXCLI.

# esxcli storage nmp satp rule add -s "VMW_SATP_ALUA" -P "VMW_PSP_RR" -O iops=1 -c "tpgs_on" -V "3PARdata" -M "VV" -e "HP 3PAR Custom iSCSI/FC/FCoE ALUA Rule"

This custom rule sets VMW_PSP_RR as the default PSP and it evenly distribute the IOs over all active paths by switching to the next active path after each IO.

iSCSI discovery

Before you can use an exported volume, the host needs to discover the volume from the target. You have to configure the iSCSI discovery in the settings of the software iSCSI initiator. Typically you will use the dynamic discovery process. In this case, the initiator uses SendTargets request to get a list of available targets. After adding the IP addresses of the 3PAR CNAs to the dynamic discovery list, the static discovery list is filled automatically. In case of multiple subnets, the dynamic discovery process can carry some caveats. Chris Wahl has highlighted this problem in his blog post “Exercise Caution Using Dynamic Discovery for Multi-Homed iSCSI Targets“. My colleague Claudia and I stumbled over this behaviour in our last 3PAR project. Removing the IP addresses from the dynamic discovery will result in the loss of the static discovery entries. After a reboot, the entries in the static discovery list will be gone and therefore no volumes will be discovered. I added a comment to Chris blog post and he was able to confirm this behaviour. The solution is to use the dynamic discovery to get a list of targets, and then add the targets manually to the static discovery list.

Final words

HP 3PAR and iSCSI is a equivalent solution to HP 3PAR and Fibre Channel/ FCoE. Especially in SMB environments, iSCSI is a good choice to bring to the customer 3PAR goodness to a reasonable price.

Add a new version of HP Agentless Management Service to a customized ESXi 5.5.0 ISO

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

While preparing for a VMware vSphere 5.5 update at a customer of mine, I stumbled over VMware KB2085618 (ESXi host cannot initiate vMotion or enable services and reports the error: Heap globalCartel-1 already at its maximum size.Cannot expand.). I checked the HP AMS version in the latest HP custom ESXi image and found out, that version hp-ams-esx-550.10.0.0-18.1198610 is included (source). Unfortunately the bug is not fixed in 10.0.0, but it’s fixed in 10.0.1 (source).


HPE/ hpe.com

According to the VMware KB article only the HP AMS versions hp-ams 500.9.6.0-12.434156 and hp-ams-550.9.6.0-12.1198610 should be affected. But since I do not like surprises, I decided to update the HP AMS version in the latest HP custom ESXi image from 10.0.0 to 10.0.1.


Before you can start building a new customized ESXi image, you have to fulfill some prerequisites.

  1. Latest version of the HP customized ESXi Image. Make sure that you download the ZIP and not the ISO file! Download
  2. Latest version of the HP Agentless Management Service Offline Bundle for VMware vSphere 5.5. Download
  3. VMware Power CLI installed on your computer. Download

Updating HP AMS

Copy both downloaded files into a temporary folder. Then import both depot files. You can proof the success with Get-EsxImageProfile which should show you the just imported ESXi image file version.

Add-EsxSoftwareDepot .\VMware-ESXi-5.5.0-Update2-2068190-HP-5.76.36-Sep2014-depot.zip
Add-EsxSoftwareDepot .\hp-ams-esxi5.5-bundle-10.0.1-2.zip

The next step is to clone the image profile. This cloned image profile will be the target for our software package update. You can check the success again with Get-EsxImageProfile. At this point you should get two image profiles listed.

New-EsxImageProfile -CloneProfile HP-ESXi-5.5.0-Update2-5.76.36 -Name HP-ESXi-5.5.0-Update2-5.76.36-AMS-updated -AcceptanceLevel PartnerSupported -Vendor "Hewlett-Packard"

Now you can update the HP AMS package. The update is done using the Add-EsxSoftwarePackage commandlet.

Add-EsxSoftwarePackage -ImageProfile HP-ESXi-5.5.0-Update2-5.76.36-AMS-updated -SoftwarePackage hp-ams

When you compare the original and the clones profile, you should get this result. Note the UpgradeFromRef entry.

Compare-EsxImageProfile -ReferenceProfile HP-ESXi-5.5.0-Update2-5.76.36 -ComparisonProfile HP-ESXi-5.5.0-Update2-5.76.36-AMS-updated

The last step is to export the clones and updates image profile to a ZIP or, this is our case, to a ISO file. This ISO file can be used to upgrade hosts using VMware Update Manager.

Export-EsxImageProfile -Imageprofile HP-ESXi-5.5.0-Update2-5.76.36-AMS-updated -ExportToIso -Filepath HP-ESXi-5.5.0-Update2-5.76.36-AMS-updated.iso

That’s it. Now you can update you hosts with this ISO and you have automatically updates the HP Agentless Management Services.


Ivo Beerens wrote a nice script to check the installed version of the HP AMS version. Checkout his blog post about this topic.


I discovered today that HP has published a new version of their customized HP ESXi release (vSphere 5.5 U2 Nov 2014). This release includes the latest version of the HP Agentless Management Service Offline Bundle for VMware vSphere 5.5.

Memory management: VMware ESXi vs. Microsoft Hyper-V

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Virtualization is an awesome technology. Last weeks I visited a customer and we took a walk through their data centers. While standing in one of their data centers I thought: Imagine that all server, that they are currently run as VMs, would be physical?. I’m still impressed about the influence of virtualization. The idea is so simple You share the resources of the physical hardware between multiple virtual instances. I/O, network bandwidth, CPU cycles and memory. After nearly 10 years of experience with server virtualization I can tell that especially the memory resources is one of the weak points. When a customer experiences performance problems, they were mostly caused by a  lack of storage I/O or memory.

The reason for this post

Today I like to write a bit about memory management of hypervisors, in this case the memory management of VMware ESXi (the trombone in the flutes orchestra) and Microsoft Hyper-V. They are the leading hypervisors on the market (source: Magic Quadrant for x86 Server Virtualization Infrastructure). But there is another cause, why I took a closer look at the memory management of Hyper-V: Microsofts support policies and recommendations for exchange servers in hardware virtualization environments. In the run up to a Exchange migration project I took a quick look into Microsofts TechNet, just to verify some questions. And then I stumbled over this statement, valid for Exchange 2013:

Exchange memory requirements and recommendations

Some hypervisors have the ability to oversubscribe or dynamically adjust the amount of memory available to a specific guest machine based on the perceived usage of memory in the guest machine as compared to the needs of other guest machines managed by the same hypervisor. This technology makes sense for workloads in which memory is needed for brief periods of time and then can be surrendered for other uses. However, it doesn’t make sense for workloads that are designed to use memory on an ongoing basis. Exchange, like many server applications with optimizations for performance that involve caching of data in memory, is susceptible to poor system performance and an unacceptable client experience if it doesn’t have full control over the memory allocated to the physical or virtual machine on which it’s running. As a result, using dynamic memory features for Exchange isn’t supported.

There are similar statement for Exchange 2007 and 2010. At the first moment I thought “Okay, looks like the Exchange-on-NFS thing”. Check Josh Odgers blog post if you want to know more about this Exchange-on-NFS thing. If you’re running your Exchange on NFS, don’t read it. There is reason to believe that you will go out and shoot a Microsoft engineer after reading it. After a couple of seconds I thought “What does dynamic memory feature mean?” This was the beginning of a journey into the deep of hypervisor memory management.

The derivation

Memory is the only component in a server that can’t be oversubscribed. That’s plausible, because you can schedule multiple VMs to a single CPU core using a  time-slice mechanism. But you can’t share a memory cell, if a VM has stored data in it. Now you have a number of options. You can configue a static memory size for each VM. If you have 32 GB memory in your virtualization host, you can run e.g. two VMs with 8 GB and four VMs with 4 GB memory. But what if a VM needs more memory? Either you reduce the amount of memory for the other VMs or you have to add memory to your host. Not very flexible. Now suppose that the VMs take full advantage of its configured memory only very rarely. In this case we can use the unused memory for the running or new VMs. We can oversubscribe the memory of the physical host. But this only works as long as the number of actively used memory is less or equal the memory size of the host. We have to react only in the case that a VM wants to take full advantage of its configured memory.

How does VMware ESXi manages its memory?

VMware ESXi uses four technologies to manage its memory:

  • Transparent Page Sharing (TPS)
  • Ballooning
  • Memory Compression
  • Swapping

Since the introduction of large pages (2 MB memory pages), TPS is only used under memory contention (thanks to Manfred for this hint). With TPS the memory is divided into pages and the hypervisor checks, if some of the pages are identical. If this is the case, the hypervisor stores only one copy of page and sets pointers to the identical ones. If you’re running a lot of similar VMs, then TPS can reduce the amount of used memory. Ballooning uses a special driver inside the VM. The hypervisor can use this driver to allocate memory inside of a VM. The OS inside the VM then frees up memory that isn’t used. The hypervisor then can reclaim that memory. Memory compression is used shortly before the hypervisor has to swap to disk. If a memory page can be compressed by at least 50% it’s held in the memory compression cache (10% of the memory is reserved for this). Otherwise it’s swapped to disk. Swapping is the last technology. If there is no more memory left and the other technologies are used up to their maximum, memory pages are swapped out to disk. Please note, that this is a very rough summary. For more information, please check VMware vSphere Resource Management Guide. With this techniques you can easily create four VMs with each 16 GB memory on a host with 32 GB memory. Important is, that the VMs can only allocate less then 32 GB memory, because the hypervisor also needs some memory for itself and virtualization overhead. A VM needs at least the amount of overhead memory to start on a VMware ESXi.

How does Microsoft Hyper-V manages its memory?

Until Windows Server 2008 R2 SP1 Microsoft Hyper-V was unable to do dynamic memory management. Only static memory allocation was possible. To stay with the example, it wasn’t possible to start four 16 GB VMs on a 32 GB host with Hyper-V. During the power-on, Hyper-V reserves the configured memory of the VM, which makes unused memory unavailable for other VMs. With Windows Server 2008 R2 SP1 Microsoft added dynamic memory management to Hyper-V. Since then you can enable dynamic memory on VM-level. After anabling it for a VM you can set a so called “Startup RAM”. This is the amount of memory which is assigned to the VM during startup. This is because Windows needs more memory during startup than the steady state (Source). It should be set to the amount of memory which is needed to run the server and application with the desired performance. You can also configure a “Minimum RAM”. This is the amount of memory up to which the hypervisor can reclaim memory using a ballooning technique. Because you can configure a “Minimum RAM”, you can also configure a “Maximum RAM”. This is the amount of memory up to which the hypervisor can add memory to the VM. And now comes the interesting part. Any idea how the hypervisor adds memory to the VM? No? He’s using memory hot-add! If the VMs needs more memory, it’s simply hot-added to the VM. This explains why the OS inside the VM has to support memory hot-add in the case you want to use dynamic memory. And it explains also why some applications are not supported with Hyper-V dynamic memory.


IMHO it’s not the dynamic memory managemet itself, which leads to Microsofts support statement. It’s the way how Microsoft Hyper-V manages the dynamic memory. Exchange checks the configured memory during the start of its services. If the memory size is increased after the start of the services, Exchange simply doesn’t recognizes it. On the other hand Microsoft SQL Server can profit from hot-added memory. Because of this, dynamic memory is supported with Microsoft SQL Server (check question 7 and answer 7 in the linked KB article). VMware ESXi doesn’t hot-add memory to a VM. Therefore you have to configure a suitable memory size. If you hot-add memory, then the same restrictions apply as for Hyper-V. Instead of relying on memory hot-add you can configure the suitable memory size when using VMware ESXi. But always remember: Memory oversubscription can lead to performance problems, if the VMs try to allocate the configured memory! Best practice is not to oversubscribe memory.


Memory management in ESXi and Hyper-V strongly differs. There’s no better or worse. They are too different to compare and they are developed for different use cases.

Conflicting information: Setting iops option for VMW_PSP_RR for HP 3PAR StoreServ on ESXi

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Yesterday I received the following tweet:

Later Craig Kilborn joined the conversation and I decided to clarify this 100 or 1 IOPS myth the next morning.

In order to give you some context: I wrote a blog post about adding a custom SATP claimrule for HP 3PAR StoreServ storage on ESXi. In this blog post I pointed out, that the claim rule is usually used to change the default behaviour for switching the path for active IO. For the VMW_PSP_RR this is 1000 IOPS, which means, that after 1000 IOPS for a specific device, the path for the active IO to this device ist changed to the next active and optimized IO path. I recommend to read this blog post from Duncan Epping for more information.

I checked two documents which are important and recommened to read, if you want to implement a HP 3PAR StoreServ with VMware ESXi:

The implementation guide provides you the necessary information for the implementation. The best practice whitepaper will cater to its name: it is a best practice whitepaper. So the first one shows you how to do the right things, the second document shows you how to do things right.

Lets have a look into the implementation guide on page 46. There are several commands to add a custom claimrule, depending on the selected host persona.


HPE/ www.hpe.com

On page 43 HP writes:

As part of PSP Round-Robin configuration, the value of IOPS can be specified. IOPS is the number of IO operations scheduled for each path during path changes within the Round-Robin path selection scheme. The default IOPS value is 1000. HP recommends IOPS=100 as an initial value and starting point for further optimization of IO throughput with PSP Round-Robin. With the exception of ESX/ESXi 4.0 versions, it is preferable to set the IOPS value within a SATP custom rule.

Okay. Lets take a look into the best practice guide. On page 8 HP describes how to add a custom claimrule. This time with IOPS=1.


HPE/ www.hpe.com

On the bottom of page 7 HP writes:

Managing a Round Robin I/O path policy scheme through the vSphere Web Client on a per datastore basis does not allow setting important Round Robin policy details that can be specified when using the command line on the ESXi host. To achieve better load balancing across paths, the –iops option may be issued on the command line to specify that the path should be switched after performing the specified number of I/Os on the current path. By default, the –iops option is set to 1000. The recommended setting for HP 3PAR Storage is 1, and this setting may be changed as needed to suit the demands of various workloads.

In a version of March 2013 of the HP 3PAR StoreServ Storage and VMware vSphere 5 best practices guide, this statement isn’t included! I assume that this happened due to the release of 3PAR OS 3.1.2, which was announced in December 2012. With 3PAR OS 3.1.2 the host persona 11 was released and it was recommended to use this with VMware ESXi. With host persona 11 the hosts use VMW_SATP_ALUA instead of VMW_SATP_DEFAULT_AA.

Depending on the document and the age of the document, different IOPS values are stated. IOPS=100 is stated in the implementation guide and should be used as a start value for further optimizations. The best practice guide clearly recommended IOPS=1. I recommend to do some tests with your customers workload and then choose whether 100 or 1 IOPS.

As a reference:

HP 3PAR VMware ESX Implementation Guide
HP 3PAR StoreServ Storage and VMware vSphere 5 best practices (April 2014)
HP 3PAR StoreServ Storage and VMware vSphere 5 best practices (March 2013)