VMware Update Manager reports “error code 99” during scan operation

After updating my lab to VMware vSphere 6.0 U2, one of my hosts continuously thrown an error during an update scan.

The first thing I’ve checked was the esxupdate.log on the affected ESXi host. This is the output, that was logged during a scan operation.

You might notice the “Unrecognized file vendor-index.xml in Metadata file” error. I also found this error message on the other hosts, so I excluded it from further research. It was unlikely, that this error was related to the observed problem. I started searching differences between the hosts and found out, that the output of “esxcli software vib list” was different on the faulty host.

This is the output on the faulty host:

This is the output on a working host. You see the difference?

Doesn’t look right… I investigated further, still searching for differences. And then I found two empty directories under /var/db/esximg.

The same directory was populated on other, working hosts.

One possible solution was therefore to copy the missing files to the faulty host. I used SCP for this. To get SCP working, you have to enable the SSH Client in the ESXi firewall.

After that, I’ve copied the files from a working host to the faulty host. Please make sure that the hosts have the same build! In my case, both hosts had the same build. Don’t try to copy files from an older or newer build to the host!!

And because we are pros, we disable the SSH Client after using it.

As expected, “esxcli software vib list” was working again.

A rescan operation in the vSphere Client was also successful. It seems that the root cause for the problem were missing files under /var/db/esximg.

Please don’t ask why this has happened. I really have no idea. But VMware KB2043170 (Initializing the VMware vCenter Update Manager database without reinstalling it) isn’t always the solution for “error code 99”, as sometimes written somewhere in the internet. Always try to analyze the problem and try to filter out unlikely and likely solutions.

VMware Update Manager reports “error code 99” during scan operation
4.9 (98.67%) 15 votes
Patrick Terlisten
Follow me

Patrick Terlisten

vcloudnine.de is the personal blog of Patrick Terlisten. Patrick has a strong focus on virtualization & cloud solutions, but also storage, networking, and IT infrastructure in general. He is a fan of Lean Management and agile methods, and practices continuous improvement whereever it is possible.

Feel free to follow him on Twitter and/ or leave a comment.
Patrick Terlisten
Follow me

14 thoughts on “VMware Update Manager reports “error code 99” during scan operation

  1. Anders Olsson

    I have this exact error on one host after updating it with the latest HPE drivers and software. Two other identical hosts have gone through the same patching without getting the error. Will try to get VMware GSS to have a look at this.

    Are your hosts HPE servers as well? I suspect the HPE VUM metadata might be messed up, since I’m seeing a lot of error messages about them in /var/log/esxupdate.log like

    “2016-04-13T08:35:17Z esxupdate: Metadata.pyc: INFO: Unrecognized file vendor-index.xml in Metadata file
    2016-04-13T08:35:17Z esxupdate: downloader: DEBUG: Downloading http://[VUM-server]:9084/vum/repository/hostupdate/HPQ/metadata-hp-esxi5.5uX-bundle-2.3-17.zip to /tmp/tmpjOAQ52…”

    Reply
    1. Patrick Terlisten Post author

      Hi Anders. Thank you for your comment. Yes, two HPE DL360 G7. I used the latest ESXi 6 image from HPE. Please leave a comment here if you have further information.

      Reply
      1. Luciano Patrao

        Hi Patrick,

        Had the exact some issue with a DL360 G7 this week.

        Also looking at the ESXi software profile I get this

        #> esxcli software profile get
        [Exception]
        No host image profile defined
        Please refer to the log file for more details.

        Also my /var/db/esximg/ was empty and if I try to get the vib list, get the same output

        Name Version Vendor Acceptance Level Install Date
        ———– —————— —— —————- ————
        tools-light 6.0.0-2.34.3620759 VMware VMwareCertified 2016-04-23

        Also try to install a VIB manually and was not possible because of the Software profile.

        I am also using HPE image, and updates to latest VMware updates are not working when I try to stage the updates on this particularly HP ESXi host.

        This is a cluster with 12 exact the same HP Version hosts, and all were installed with HPE ESXi image and all have the same ESXi version and build at the moment, but this one just start with this behavior this week.

        I also don’t have an answer for this problem.

        Try your solution(copy file from one working host to the fault one) and after I was able to scan, stage and after remediate this host.

        Thanks for the solution, since I was searching in the internet a solution to fix the software profile issue without reinstall the host and your solution did the trick.

        Reply
  2. Chris

    Just updated 2 HP ProLiant DL360 G9 using VUM to update to esxi 6.0 U3. First host went alright. Second on had the problem you described.

    Thanks for the solution.

    Reply
  3. ROHIT KUMAR

    Patrick – i have a host with the same error – almost the same error logs . you said Don’t try to copy files from an older or newer build to the host!! now the only build i have matching the affected host is in another datacenter . WIll it affect or is it harmful to copy the files from a host in one datacenter to this affected host. Any help here is apppreciated.

    Reply
  4. Dale

    Life Saver. Thank you. Ran this update across 6 identical hosts and one of them had this exact issue where the other 5 did not.

    Reply
  5. RLeamey

    Anyone know how to resolve this if you don’t have another host at the exact same build level?

    Reply
  6. Pingback: ESXi 6.0 to 6.5 Upgrade – Stuff I'm Up To

Leave a Reply

Your email address will not be published. Required fields are marked *

Ich stimme zu / I accept