This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
This error gets me from time to time, regardless which server vendor, mostly on hosts that were upgraded a couple of times. In this case it was a ESXi host currently running a pretty old build of ESXi 6.7 U3 and my job was the upgrade to 7.0 Update 3c.
If you add a upgrade baseline to the cluster or host, and you try to remediate the host, the task fails with a dependency error. When taking a closer look into the taks details, you were getting told that the task has failed because of a failed dependency, but not which VIB it caused.
You can find the name if the causing VIB on the update manager tab of the host which you tried to update. The status of the baseline is “incompatible”, and not “non-compliant”.
To resolve this issue you have to remove the causing VIB. This is no big deal and can be done with esxcli. Enable SSH and open a SSH connection to the host. Then remove the VIB.
[root@esx-01:~] esxcli software vib list | grep -i ssacli
ssacli 4.17.6.0-6.7.0.7535516.hpe HPE PartnerSupported 2020-06-18
[root@esx-01:~] esxcli software vib remove -n ssacli
Removal Result
Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
Reboot Required: true
VIBs Installed:
VIBs Removed: HPE_bootbank_ssacli_4.17.6.0-6.7.0.7535516.hpe
VIBs Skipped:
[root@esx-01:~]
You need to reboot the host after the removal of the VIB. Then you can proceed with the update. The status of the upgrade baseline should be now “not-compliant”.
While debugging a vCener Lifecycle Manager, which was unable to download updates, I’ve stumbled over a weird behaviour, which is (IMHO) by design.
Some of you might use a proxy server. And some of you might use a proxy server which requires credentials. In my case, my customer uses a Sophos SG appliance as a web proxy server with authentication. The customer creaded a user with a complex password. But I was unable to get a working internet connection.
I played a bit with curl on the bash of the vCenter. The proxy settings are stored under /etc/sysconfig/proxy. These settings are used to populate the http_proxy and https_proxy environment variable. It’s important to know, that the credentials stored in the /etc/sysconfig/proxy are encoded with the percent-encoding, also known as URL encoding. So someone with root access can grab credentials from these file.
But then I noticed something weird. I set the http_proxy variable manually with
http://username:password@proxy.domain.tld:8080
and I got this error:
-bash: !": event not found
Okay… there was a ! in the password and the BASH tried to execute the part behind the !. But it was part of the password, so I had to tell the BASH that it has to take this literally.
I escaped the ! in the password with a \. And to my surprise: The vCenter was able to download updates. I decoded the percent-encoded string in the /etc/sysconfig/poxy and found the escaped ! (\!). For example. Instead of Passw0rd! I had to enter Passw0rd\! in the password field.
Long story short: Use a password without special characters, otherwise escape them, because the password is stored in BASH variables.
This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
During an vSphere 6.5 > 6.7 update a was host failing continously at the remediation with an “unknown error”. The host was updated from ESXI 6.5 to 6.7 using an upgrade baseline. Other hosts were updated to 6.7 and with the latest patches without any issues. Something strange was going on…
The esxupdate.log and the vua.log on the host itself showed nothing special. So I checked the vmware-vum-server-log4cpp.log which was much more informative!
[2020-07-19 13:03:25:217 'SingleHostScanTask.SingleHostScanTask{262}' 139762329831168 ERROR] [singleHostScanTask, 693] caught an odbc error: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vci_scanresults"; Error while executing the query" is returned when executing SQL statement "INSERT INTO VCI_SCANRESULTS(endtime, id, scan_status, scan_type, starttime, target_component, target_uid) VALUES (?, ?, ?, ?, ?, ?, ?)"
[2020-07-19 13:03:25:219 'SingleHostScanTask.SingleHostScanTask{262}' 139762329831168 ERROR] [singleHostScanTask, 404] SingleHostScan caught exception: caught an odbc error: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vci_scanresults"; Error while executing the query" is returned when executing SQL statement "INSERT INTO VCI_SCANRESULTS(endtime, id, scan_status, scan_type, starttime, target_component, target_uid) VALUES (?, ?, ?, ?, ?, ?, ?)" with code: -1
[2020-07-19 13:03:25:223 'SingleHostScanTask.SingleHostScanTask{262}' 139762329831168 ERROR] [vciTaskBase, 568] Task execution has failed: caught an odbc error: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vci_scanresults"; Error while executing the query" is returned when executing SQL statement "INSERT INTO VCI_SCANRESULTS(endtime, id, scan_status, scan_type, starttime, target_component, target_uid) VALUES (?
Well… ERROR: duplicate key value violates unique constraint “pk_vci_scanresults” is not what I expected, but it is an error, and it occured everytime I tried to remediate the host.
Google found nothing about this error, so I decided to reset the VUM database. Please don’t try this at your customer! Log a call at VMware.
To reset the VUM database:
Connect to vCenter Server Appliance via SSH
Switch to the BASH
Stop the VMware Update Manager Service with this command
service-control –stop vmware-updatemgr
To reset the VMware Update Manager Database (applies only to VCSA 6.7 and 7.0!)
Delete the contents of the VMware Update Manager Patch Store
rm -rf /storage/updatemgr/patch-store/*
Start the VMware Update Manager Service again
service-control –start vmware-updatemgr
You will lose all your baselines, so you have to configure them again. And you need to download all patches again.
For vSAN environments this procedure will also remove the vSAN default baselines, but they will recreated automatically when there is a configuration change to vSAN or an update to the HCL DB. Again: Don’t do this at home!
This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
When it comes to disaster recovery (DR), dedicated offsite infrastructure is a must. If you follow the 3-2-1 backup rule, then you should have at least three copies of your data, on two different media, and one copy should be offsite.
But an offsite copy of your data can be expensive… You have to setup storage and networking in a suitable colocation. And even if you have an offsite copy of your data, you must be able to recover the data. This could be fun in case of terabytes of data and an offsite copy on tape.
A offsite copy in a cloud is much more interesting. No need to provide hardware, software, licenses. Just provide internet-connectivity, book a suitable plan, and you are ready to go.
Replication to Cloud using Vembu CloudDR
Vembu offers a cloud-based disaster recovery plan through its own cloud services, which is hosted in Amazon Web Services (AWS). This product is designed for businesses, who can’t afford, or who are not willing, to setup a dedicated offsite infrastructure for disaster recovery.
The data, which is backuped by the Vembu BDR server, is replicated to the Vembu Cloud. In case of any disaster, the backup data can be directly restored from the cloud at anytime and anywhere. The replication is managed and monitored using the CloudDR portal.
Before you can enable the offsite replication, you have to register your Vembu BDR server with your Vembu Portal account. You can either go to onlinebackup.vembu.com, or you can go to portal.vembu.com and sign up.
Vembu Technologies/ Vembu CloudDR/ Copyright by Vembu Technologies
After configuring schedule, retention and bandwidth usage, Vembu CloudDR is ready to go.
The end is near – time for recovery
CloudDR offers two types of recovery:
Image Based Recovery
Application Based Recovery
In case of an image based recovery, you can either download a VMDK or VHD(X) image, or you can do a file level recovery. In this case you can restore single files from inside of a chosen image.
You can even download a VHD(X) image of a VMware backup, which allows you some kind of V2V or P2V restores.
In case of a application based recovery, you can recover single application items from
Microsoft Exchange
Microsoft SharePoint
Microsoft SQL Server, or
MySQL
Depending on the type of restore, you will get an encrypted and password protected ZIP file with documents, or even MDF/ LDF files. These files can than be used to restore the lost data.
Summary
Vembu CloudDR is a pretty interesting add-on for Vembu customers. It’s easy to setup, has an attractive price tag and therefore consequently addresses the SMB customers.
This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
It is pretty common that vendors offer their products in special editions for SMB customers. VMware offers VMware vSphere Essentials and Essentials Plus, Veeam offers Veeam Backup Essentials, and Vembu has Vembu BDR Essentials.
Now Vembu has extended their Vembu BDR Essentials package significantly to address the needs of mid-sized businesses.
Vembu Technologies/ Vembu BDR Essentials/ Copyright by Vembu Technologies
Affordable backup for SMB customers
Most SMB virtualization deployments consists of two or three hosts, which makes 4 or 6 used CPU sockets. Because of this, Vembu BDR Essentials supportes up to 6 sockets or 50 VMs. Yes, 6 sockets OR 50 VMs. Vembu has no rised this limit to 10 Sockets OR 100 VMs! This allows customers to use up to five 2-socket hosts or 100 VMs with less than 10 sockets.
Feature Highlights
Vembu BDR Essentials support all important features:
Agentless VMBackup to backup VMs
Continuous Data Protection with support for RPOs of less than 15 minutes
Quick VM Recovery to get failed VMs up and running in minutes
Vembu Universal Explorer to restore individual items from Microsoft applications like Exchange, SharePoint, SQL and Active Directory
Replication of VMs Vembu OffsiteDR and Vembu CloudDR
Needless to say that Vembu BDR Essentials support VMware vSphere and Microsoft Hyper-V. If necessary, customer can upgrade to the Standard or Enterprise edition.
This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
Last year in December, I updated the first customer from HPE Data Protector 9.04 to 9.05. Immediately after the first tests I noticed, that backups were made using the NBDSSL transport. I expected that the SAN transport would be used, because the prerequisites were met and it has worked until the update. I opened a case at the HPE support und I was advised to install the hotfix QCIM2A65619. With this hotfix, several files were replaced:
The hotfix solved the issue. And to be honest: I didn’t care why it has worked after applying the hotfix. I had the same issue at multiple customers and applying the hotfix solved the issue in each case.
Windows Server 2003 R2 (x64) Windows Server 2008, 2008 R2 (x64) Windows Server 2012 (x64) RHEL 5.9 (x64) RHEL 6.2, 6.3 (x64) SLES 10.4 (x64) SLES 11 (x64)
9.02, 9.03
VDDK 5.5.3
Windows Server 2003 R2 (x64) Windows Server 2008, 2008 R2 (x64) Windows Server 2012 (x64) RHEL 5.9 (x64) RHEL 6.2, 6.3, 6.4 (x64) SLES 10.4 (x64) SLES 11 (x64)
9.04
VDDK 6.0
Windows Server 2008 R2 (x64) Windows Server 2012, 2012 R2 (x64) RHEL 6.6, 7.0 (x64) SLES 11, 12 (x64)
9.05
VDDK 6.0 U1
Windows Server 2008 R2 (x64) Windows Server 2012, 2012 R2 (x64) RHEL 6.6, 7.0 (x64) SLES 11, 12 (x64)
9.06
VDDK 6.0 U2
Windows Server 2008 R2 (x64) Windows Server 2012, 2012 R2 (x64) RHEL 6.6, 7.0 (x64) SLES 11, 12 (x64)
There was a footnote for VDDK 6.0 U1.
The VM backups does not use SAN transport mode on vSphere 5.1, 5.5 (and its updates) environment and falls back to NBDSSL/NBD. This is because of VDDK 6.0 U1 issue. For more information, see VMware Knowledge Base.
Ups… that’s my issue! The footnote inclued a link to VMware KB2135621 (Virtual Disk Development Kit 6.0 U1 Backup and Restore commands fail using SAN transport mode on ESXi 5.5.x hosts on both Windows and Linux proxies). Described symptoms:
Virtual Disk Development Kit 6.0 Update 1 backup and restore commands fail using SAN transport mode on ESXi 5.5.x hosts.
This issue occurs on both Windows and Linux proxies.
Yep, that’s my issue. The customers that were observing this issue were running vSphere 5.5, not 6.0. With this knowledge, I checked the version of the vixDiskLib.dll on one of the patched Data Protector hosts. And there it was:
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
The vixDiskLib.dll had the build version 6.0.0 build-2498720, which is the build version of the Virtual Disk Development Kit 6.0. So it seems, that the Data Protector hotfix QCIM2A65619 makes a downgrade of the VDDK that is used by Data Protector.
KB2135621 describes, that this issue is resolved in in VMware vCenter Server 6.0 Update 2. This also implies, that this is fixed for VDDK 6.0 U2 and therefore Data Protector 9.06.
This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
VMware vSphere 6 is now an year old and it was time to update my lab to vSphere 6. The update went smooth, and everything has worked as expected. Some days later, I updated the master VM of a small automated desktop pool. I’m using VMware Horizon 6.2.1 in my lab to deploy a small number of Windows 8.1 VMs for tests, administration etc. The recompose of the pool failed during the guest customization.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
I checked the customization specification immediately and got an error in the vSphere C# client.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
Interestingly, I got no error in the vSphere Web Client:
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
After re-entering the Administrator password, the customization specification was usable again. No errors so far.
A quick search in the VMware KB lead me to the article “Virtual machines with customizations fail to deploy when using Custom SSL Certificates (1019893)“. But this article doesn’t apply to vCenter 6.0. For the notes: I’m using CA-signed certificates in my environment. It seems to be a good idea to re-enter the passwords in customization specifications after a vCenter migration/ upgrade (5.x to 6.x or from VCSA 5.x to 6.x).
This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
Yesterday I did an upgrade of my vCenter Server Appliance 5.5 U3 to 6.0 U1. This was the first step to update my lab infrastructure to vSphere 6.0. A bit late, but better late than never. The update of the VCSA itself went smooth. No problems with certificates, hosts, VMs or PernixData FVP. But then, I discovered two errors on the old vSphere C# client (I know that I should use the Web Client…)
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
The first message indicates a problem with the VMware vCenter Storage Monitoring Service.
The plug-in failed to load on server(s) vcsa1.lab.local due to the following error:
Could not load file or assembly 'VI, Version=5.5.0.0, Culture=neutral, PublicKeyToken=7c80a434483c7c50' or one of its dependencies. The system cannot find the file specified.
The other error message indicates a problem with Auto Deploy.
The following error occurred while downloading the script plugin from https://vcsa1.lab.local:6502/vmw/rbd/extensions.xml: The request failed because of a connection failure. (Unable to connect to the remote server)
Both error messages have harmless reasons.
VMware vCenter Storage Monitoring Service
The error message regarding the VMware vCenter Storage Monitoring Service is an expected behaviour. The answer to this is written in the VMware vCenter Server 6.0 Release Notes.
vSphere Web Client. The Storage Reports selection from an object’s Monitor tab is no longer available in the vSphere 6.0 Web Client.
vSphere Client. The Storage Views tab is no longer available in the vSphere 6.0 Client.
To get rid of this message, simply ignore it or remove the old vSphere C# client. Newer releases of the C# client don’t have this plugin. If you need older C# client releases, you can ignore this error message.
Auto Deploy
The error message regarding Auto Deploy is caused by a stopped Auto Deploy service. The Auto Deploy service is stopped by default.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
You can find the Auto Deploy service in the vSphere Web Client under Administration > System Configuration > Services > Auto Deploy. The error message will be gone, if you start the service. If you don’t need Auto Deploy, you can ignore this error message.
Marco van Baggum wrote a blog post about the same messages some months ago. He highlighted an alternative way to get rid of these messages.
This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
A load balancer is an integral component of (nearly) every VMware Horizon View design. Not only to distribute the connections among a number of connection or security servers, but also to provide high availability in case of a connection or security server failure. Without a load balancer, connection attempts will fail, if a connection or security server isn’t available. Craig Kilborn wrote an excellent article about the different possible designs of load balancing. Craig highlighted Microsoft Network Load Balancing (NLB) as one of the possible ways to implement load balancing. Jason Langer also mentioned Microsoft NLB in his worth reading article “The Good, The Bad, and The Ugly of VMware View Load Balancing“.
Why Microsoft NLB?
Why should I use Microsoft NLB to load balance connections in my VMware Horizon View environment? It’s a question of requirements. If you already have a load balancer (hopefully redundant), then there is no reason to use Microsoft NLB for load balancing. Really no reason? A single load balancer is a single point of failure and therefore you should avoid this. Instead of using a single load balancer, you could use Microsoft NLB. Microsoft NLB is for free, because it’s part of Windows Server. At least two servers can form a highly available load balancer and you can install the NLB role directly onto your Horizon View connection or security servers.
How does it work?
Microsoft Windows NLB is part of the operating system since Windows NT Server. Two or more Windows servers can form a cluster with at one or more virtual ip addresses. Microsoft NLB knows three different operating modes:
Nearly all switches support IGMP Snooping. If not, spend some money on new switches. Let me get this clear: If your switches support IGMP Snooping, enable this for the VLAN to which the cluster nodes are connected. There is no need to configure static multicast mac addresses or dedicated VLANs to avoid flooding.
If you select the multicast (IGMP) mode, each cluster node will periodically send an IGMP join message to the multicast address of the group. This address is always 239.255.x.y, where x and y corresponds to the last two octets of the virtual ip address. Upon receiving these multicast group join messages, the switch can send multicasts only to the ports of the group members. This avoids network flooding. Multicast (IGMP) simplifies the configuration of a Microsoft NLB.
Enable IGMP Snooping for the VLAN of the cluster nodes
Enable the NLB role on each server that should participate the cluster
Create a cluster with multicast (IGMP) mode on the first node
Join the other nodes
That’s it!
The installation process of a Microsoft NLB is not particularly complex, and if the NLB cluster is running, there is not much maintenance to do. As already mentioned, you can put the NLB role on each connection or connection server.
What are the caveats?
Sounds pretty good, right? But there are some caveat when using Microsoft NLB. Microsoft NLB does not support sticky connections and it does not support service awareness. Why is this a problem? Let’s assume that you have enabled “HTTP(S) Secure Tunnel”, “PCoIP Secure Gateway” and “Blast Secure Gateway”.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
In this case, all connections are proxied through the connection or security servers.
The initial connection from the Horizon View client to the connection or security server is used for authentication and selection of the desired desktop pool or application. This is a HTTPS connection. At this point, the user has no connection to a pool or an application. When the user connects to a desktop pools or application, the client will open a second HTTPS connection. This connection is used to provide a secure tunnel for RDP. Because it’s the same protocol, the connection will be directed to the same connection or security server as before. The same applies to BLAST connections. But if the user connects to a pool via PCoIP, the View client will open a new connection using PCoIP with destination port 4172. If the PCoIP External URL refers to the load balanced URL, the connection can be directed to another connection or security server. If this is the case, the PCoIP connection will fail. This is because the source ip address might be the same, but another destination port is used. VMware describes this behaviour in KB1036376 (Unable to connect to the PCoIP Secure Gateway when using Microsoft NLB Clustering).
Another big caveat is the missing service awareness. Microsoft NLB does not check, if the load balanced service is available. If the load balanced service fails, Microsoft NLB will not stop to redirect requests to the the cluster node that is running the failed service. In this case, the users connection requests will fail.
Still the ugly?
So is Microsoft NLB still the ugly option? I don’t think so. Especially for small deployments, where the customer does not have a load balancer, Microsoft NLB can be a good option. If you want to load balance connection servers, Microsoft NLB can do a great job. In case of load balancing security servers, you should take a look at KB1036376, because you might need at least 3 public IP addresses for NAT. The missing service awareness can be a problem, but you can workaround it with a responsive monitoring.
In the end, it is a question of requirements. If you plan to implement other services that might require a load balancer, like Microsoft Exchange, you should take a look at a redundant, highly available load balancer appliance.
This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
Careful preparation is a key element to success. If you restart a storage controller, or even the whole storage, you should be very sure that all ESXi hosts have enough paths to every datstore. Sure, you can use the VMware vSphere C# client or the Web Client to check every host and every datastore. But if you have a large cluster with a dozen datastores and some Raw Device Mappings (RDMs), this can take a looooong time. Checking the path state of each LUN is a task, which can be perfectly automated. Get a list of all hosts, loop through every host and every LUN, output a list of all hosts with all LUNs and all paths for each LUN. Sounds easy, right?
For a long time, I used this PowerCLI script for checking the LUN path state. But now I decided to give something back and I tweaked it a bit for my needs.
Feel free to use and/ or modify it.
To change your privacy setting, e.g. granting or withdrawing consent, click here:
Settings
