I was onsite at one of my customers to update a small VMware vSphere 6.0 U3 environment to 6.5 U2c. The environment consists of three hosts. Two hosts in a cluster, and a third host is only used to run a HPE StoreVirtual Failover Manager.
The update of the first host, using the Update Manager and a HPE custom ESX 6.5 image, was pretty flawless. But the update of the second host failed with “Cannot execute upgrade script on host”
typographyimages/ pixabay.com/ Creative Commons CC0
I checked the host and found it with ESXi 6.5 installed. But I was missing one of the five iSCSI datastores. Then I tried to patch the host with the latest patches and hit “Remidiate”. The task failed with “Cannot execute upgrade script on host”. So I did a rollback to ESXi 6.0 and tried the update again, but this time using ILO and the HPE custom ISO. But the result was the same: The host was running ESXi 6.5 after the update, but the upgrade failed with the “Upgrade Script” error. After this attempt, the host was unable to mount any of the iSCSI datastores. This was because the datastores were mounted ATS-only on the other host, and the failed host was unable to mount the datastores in this mode. Very strange…
I checked the vua.log and found this error message:
2018-11-05T16:35:56.614Z info vua[A3CAB70] [[email protected] sub=VUA] Command '/tmp/vuaScript-xMVUfb/precheck.py --ip=172.19.0.14' finished with exit status 1 --> stderr: -------- --> INFO:root:Running esxcfg-info --> Traceback (most recent call last): --> File "/build/mts/release/bora-9298722/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 385, in run --> File "/build/mts/release/bora-9298722/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 788, in communicate --> File "/build/mts/release/bora-9298722/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/encodings/ascii.py", line 26, in decode --> UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 1272423: ordinal not in range(128)
Focus on this part of the error message:
--> UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 1272423: ordinal not in range(128)
The upgrade script failed due to an illegal character in the output of esxcfg-info. First of all, I had to find out what this 0x80 character is. I checked UTF-8 and the windows1252 encoding, and found out, that 0x80 is the € (Euro) symbol in the windows-1252 encoding. I searched the output of esxcfg-info for the € symbol – and found it.
\==+Heap :
|----Name............................................€A
|----Growable........................................true
|----Max Size........................................41848 bytes
|----Max Available...................................40816 bytes
|----Current Size....................................29560 bytes
|----Current Size....................................29560 bytes
|----Current Allocation..............................1032 bytes
|----Current Available...............................1032 bytes
|----Current Releasable..............................20400 bytes
|----Percent Free of Current.........................96
|----Percent Free of Max.............................97
|----Percent Releasable..............................69But how to get rid of it? Where does it hide in the ESXi config? I scrolled a bit up and down around the € symbol. A bit above, I found a reference to HPE_SATP_LH . This took immidiately my attention, because the customer is using StoreVirtual VSA and StoreVirtual HW appliances.
Now, my second educated guess of the day came into play. I checked the installed VIBs, and found the StoreVirtual Multipathing Extension installed on the failed host – but not on the host, where the ESXi 6.5 update was successful.
I removed the VIB from the buggy host, did a reboot, tried to update the host with the latest patches – with success! The cross-checking showed, that the € symbol was missing in the esxcfg-info output of the host that was upgraded first. I don’t have a clue why the StoreVirtual Multipathing Extension caused this error. The customer and I decided to not install the StoreVirtual Multipathing Extension again.
Feel free to follow him on Twitter and/ or leave a comment.
- Why you should change your KRBTGT password prior disabling RC4 - July 28, 2022
- Use app-only authentication with the Microsoft Graph PowerShell SDK - July 22, 2022
- Getting started with the Microsoft Graph PowerShell SDK - July 21, 2022
Had the same error but very different problem.
There was a symbolic link to the vFlash device that was missing, that was causing the script to fail.
To fix, I simply re-added the vFlash to be available, rebooted the host, rescanned successfully and then run in the VUM patches.
Love to hate VUM.. Works one day, not always the next… But is far better than the alternative!
Thanks for this feedback, Paul! :)
I ran into the same “Cannot execute upgrade script on host” while trying to do an upgrade from 6.5 to 6.7 and this post helped me solve the problem.
Steps taken:
1. Log into to the host via SSH and confirm the “ordinal not in range(128)” error in vua.log
In my case it was the symbol 0xc3, “Ô.
2. Log in with root to https:///cgi-bin/esxcfg-info.cgi.
3. Copy the output to Notepad++, since Chrome wouldn’t let me search for “Ô without also finding “A” and “a”.
4. Searched and found a Virtual Switch Port Group with an “å” in its name, which in turn was written out as “Ã¥” by esxcfg-info.
5. Renamed the Port Group to only contain ASCII characters.
6. Success! I could now run the upgrade process.