Tag Archives: bug

Microsoft Exchange 2013 shows blank ECP & OWA after changes to SSL certificates

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
EDIT
This issue is described in KB2971270 and is fixed in CU6.

I ran a couple of times in this error. After applying changes to SSL certificates (add, replace or delete a SSL certificate) and rebooting the server, the event log is flooded with events from source “HttpEvent” and event id 15021. The message says:

If you try to access the Exchange Control Panel (ECP) or Outlook Web Access (OWA), you will get a blank website. To solve this issue, open up an elevated command prompt on your Exchange 2013 server.

Check the certificate hash and appliaction ID for 0.0.0.0:443, 0.0.0.0:444 and 127.0.0.1:443. You will notice, that the application ID for this three entries is the same, but the certificate hash for 0.0.0.0:444 differs from the other two entries. And that’s the point. Remove the certificate for 0.0.0.0:444.

Now add it again with the correct certificate hash and application ID.

That’s it. Reboot the Exchange 2013 server and everything should be up and running again.

Windows guest customization fails after cloning a VM

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Last week I got a call from a customer. The customer has tried to deploy new Citrix XenApp servers, and because the VMware template was a bit outdated, he tried to clone a provisioned and running Citrix XenApp VM. During this, the customer applied a guest customization specification to customize the guest OS (IP address, hostname etc). Until this point everything was fine. But after the clone process, the guest customization started, but never finished.

vm_deployment_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Using the VMware template, deployment and customization were successful. So the main problem was, that the customer was unable to use a provisioned and running Windows guest to deploy new Windows guests. I checked to logs and found this error messages in the setupact.log (you can find this log under C:\windows\system32\sysprep\panther):

I checked the rearm count with slmgr.vbs /dlv and saw, that the remaining count was 1.

Cloning and customizing a Windows VM with a rearm count of 1 leads to the observed behaviour. After the cloning and the start of the customization, the rearm count is 0. Microsoft describes this behaviour in KB929828.

CAUSE

This error may occur if the Windows Software Licensing Rearm program has run more than three times in a single Windows image.

RESOLUTION
To resolve this issue, you must rebuild the Windows image.

vExpert Maish Saidel-Keesing wrote about this in his blog in 2011. He explained it very well, make sure that you read his three blog posts!

In my case, rebuilding the template wasn’t an option. Therefore I had to reset the rearm count. I searched a while and found a solution that has worked for me. I’m quite sure that Microsoft doesn’t allow this, therefore I will not describe this procedure in detail. You will find it easily in the web…

The main task is to remove the WPA registry key. This key is protected under normal operation, so you have to do this using WinRE (Windows ecovery Environment) or WinPE (Windows Preinstallation Environment). After the removal of the WPA registry key, reboot the VM, add a new key using slmgr.vbs /ipk and active the Windows installation. You can check the rearm counter using slmgr.vbs /dlv and you will notice that the rearm counter is resetted.

Always keep in mind that you can’t use sysprep with a Windows installations an infinite number of times.

VMware publishes patch for ESXi 6.0 CBT bug (KB2114076)

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today, VMware has published the long-awaited patch for the ESXi 6.0 CBT bug. This patch is the result of a problem, which is described in KB2114076 (Backing up a virtual machine with Change Block Tracking (CBT) enabled fails after upgrading to or installing VMware ESXi 6.0). All customers that upgraded to ESXi 6.0 or installed ESXi 6.0 were affected.

Symptoms of this bug were:

  • Powering on virtual machines fails
  • Expanding the size of a virtual disk fails
  • Taking virtual machine quiesced snapshots fails
  • Error messages like “An error occurred while taking a snapshot: msg.snapshot.error-QUIESCINGERROR” (vSphere Client), “WARNING: CBT: 191: No memory available! Called from 0x4180219af50e” (vmkernel.log) or “Creating cbt node 92b78c-cbt failed with error Cannot allocate memory (0xbad0014, Out of memory)” in the vmware.log of the affected virtual machine

A workaround was to disable CBT, which resulted in longer running backups.

You can download the patch (ESXi600-201505401-BG) from VMware to manually update your ESXi hosts. Otherwise you can use VMware Update Manager to patch your hosts. A reboot is necessary!

If you decide to manually update your hosts with this patch, you have to use the “esxcli software vib” command to install the patch.

  • Download the ZIP file and upload it to a datastore that is reachable for the host you want to patch (e.g. a local datastore)
  • Bring the host into the maintenance mode
  • Connect with SSH to your ESXi host and run:

  • Reboot the host and leave the maintenance mode

If you have disabled CBT for all or some of your VMs, make sure that you re-enable CBT.

vCenter Server Appliance: Troubleshooting full database partition

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

A customer of mine had within 6 months twice a full database partition on a VMware vCenter Server Appliance. After the first outage, the customer increased the size of the partition which is mounted to /storage/db. Some months later, some days ago, the vCSA became unresponsive again. Again because of a filled up database partition. The customer increased the size of the database partition again  (~ 200 GB!!) and today I had time to take a look at this nasty vCSA.

The situation

vcsa_overview

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Within 2 days, the storage usage of the databse increased from 75% to 77%. First, I checked the size of the database:

 As you can see, the database had only 2 GB. The pg_log directory was more interesting:

 The directory was full with log files. The log files containted only one message:

The solution

This led me to VMware KB2092127 (After upgrading to vCenter Server Appliance 5.5 Update 2, pg_log file reports this error: WARNING: there is already a transaction in progress). And yes, this appliance was upgraded to U2 with high probability. The solution is described in KB2092127, and is really easy to implement. Please note that this is only a workaround. There’s currently no solution, as mentioned in the article.

HP Data Protector: JSONizer error when restoring from StoreOnce

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

After installing the Data Protector patch bundle 8.13, you may ran into this error when trying to restore data from a HP StoreOnce appliance.

This problem is known and it is described in QCCR2A56465. A fix is available (new BMA, CMA, MMA and RMA binaries). Simply open a service request and ask for the fix. Make sure that you add a copy of the session messages or a screenshot to the service request.

HP Data Protector: Can’t delete old DCBF directories

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
This applies to upgrades from Data Protector 6.x and 7.x to 8.x and 9.x.

It seems that today is my debugging day… Yesterday I performed a Data Protector update from 7.03 to 8.13. During this update, the Data Protector IDB is migrated to another database format. Last night the backups went smoothly, but today I noticed that two old Detail Catalog Binary File (DCBF) directories were still referenced in the HP Data Protector IDB.

The two directories with the “db40” inside the path are old DCBF directories. Because the directories contained actively used DCBF files, I relocated the files and did a “omnidbutil -remap_dcdir”:

A quick check after the relocation showed no errors.

Looking good. Time to remove the old DCBF directories:

Did I mentioned that today was my debugging day? To make a long story short: HP switched the path separator character for the Data Protector IDB. They are using now a / instead a \ on both platforms (Windows & UNIX). During the update, this change is not performed correctly. Sebastian Koehler wrote a small SQL script that fixes this problem. Check his blog post (he had the same problem as me).

This is the script (you can also find it on Sebastian Koehlers blog):

This is the output of the script when I run it.

You can clearly see that the wrong path separator is used for the old DB40 directories (the upper part of the output). Compare it to the output of omnidbutil -list_dcdirs! The lower part shows that the correct path separator was set. After the run of the script I was able to delete the old DCBF directories.

Thanks to Sebastian, who described this bug.

VM deployment fails with “Authenticity of the host’s SSL certificate is not verified”

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

When you want to go fast, go slow. Otherwise you will get into trouble… Today I tried to quickly deploy a VM from a template and customize this VM with a customization specification. The codeword is “quickly”. The fun started with this error message:

vcsa_deploy_vm_error_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Fortunately I asked the VMware Knowledge Base, which lead me to VMware KB2086930 (Deploying a template with customization fails with the error: Authenticity of the host’s SSL certificate is not verified). This KB article is all you need to know to fix this error.

1. Make a snapshot of your vCenter Server appliance.

2. Stop the vCenter Server service using the appliance management website (port 5480).

vcsa_ssl_thumbprint_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

3. Connect with SSH to your vCenter Server appliance and run these commands. I tried to deploy the VM to esx2.lab.local. As you can see, there is no expected SSL thumbprint for this host (and also it’s missing for host esx1.lab.local). The solution is to set the host_ssl_thumbprint == expected_ssl_thumbprint.

This solved this issue for me. According to VMware KB2086930  only VMware vCenter Server Appliance 5.5.x is affected. If you are running VMware vCenter Server on Windows, you are not affected. If you get this error (or a similar error), it might be another problem.

VMware ESXi 5.5.0 U2 patches break Citrix NetScaler network connectivity

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

This is not a brand new issue and it’s well discussed in the VMTN. After applying the ESXi 5.5.0 U2 patches from 15. October 2014, you may notice the following symptoms:

  • Some Citrix NetScaler VMs with e1000 vNICs loses network connectivity
  • You can’t access the VM console after applying the patches

VMware has released a couple of patches in October:

  • ESXi550-201410101-SG (esx-base)
  • ESXi550-201410401-BG (esx-base)
  • ESXi550-201410402-BG (misc-drivers)
  • ESXi550-201410403-BG (sata-ahci)
  • ESXi550-201410404-BG (xhci-xhci)
  • ESXi550-201410405-BG (tools-light)
  • ESXi550-201410406-BG (net-vmxnet3)

More specifically, it’s the patch ESXi550-201410401-BG that is causing the problem. It is reported that the patch ESXi510-201410401-BG is also cause problems. VMware has published a KB article under the the KB2092809. Citrix has also published a KB article under the ID CTX200278. The VMware KB2092809 includes a workaround. You have to add the line

 in the loader.conf. Check the KB for a detailed procedure. I recommend to exclude the patches from the VMware Update Manager baselines. Open the “Admin View” and double click the “Critical Host Patches (Predefined)” baseline.

vum_exclude_patch_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click “Next” until you hit the “Patches to exclude” page. Exclude patch ESXi550-201410401-BG and click “Next” until you can finish the wizard.

vum_exclude_patch_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Now open the “Non-Critical Host Patches (Predefined)” baseline. Repeat the steps above and exclude the other 6 patches.

vum_exclude_patch_3

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

That’s it. Now the patches are excluded. New patches will be added automatically to the baselines, because both baselines are dynamic baselines. If you wish to install the patches, repeat the steps above and remove the patches from the exclude list using the up arrow button.

HP StoreOnce Enterprise Manager v1.3 installation fails on non-English OS

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Sometimes the easy jobs seems to be the hardest. Especially if you have to deal with high-quality software… As part of a project I had to install and configure a HP StoreOnce 4500 appliance in combination with HP Data Protector 8.12 and a StoreEver MSL2024 G3 tape-library. No big deal – until I hit the part, when I had to install HP StoreOnce Enterprise Manager v1.3 (SEM) on the new backup server. The installation failed with this error:

hp_sem_installation_error

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Setup could not provide access privileges to “C:\Program Files\Hewlett-Packard\HP StoreOnce Enterprise Manager\RMSDataStore\Postgres\data” directory of PostgreSQL

First I blamed UAC for this. I disabled it and re-ran the setup with “Run as administrator…”. The setup failed again. I tried the setup on my “rack ‘n stack” laptop (Windows 8.1 Enterprise) and it fails with exactly the same error. I was puzzled, because I had SEM running on Windows 8.1 – until I decided to re-install this laptop with a german-language Windows 8.1. At this point it dawned on me a sense of foreboding. I remembered a bug in a HP Command View EVA release, which couldn’t be installed on non-Englisch operating systems, because the setup relied on hardcoded, english group names. A quick cross-check with a english Windows Server 2008 R2 VM confirmed this and I was able to install SEM without problems.

I checked the temporary folder in which the setup files were extracted. I found a batch file calles “SEMS_InstallDB”. This batch file included this line:

It seems that the user group “Users” was hardcoded. In a german-language OS this group is called “Benutzer”. With knowing this, the solution was easy (and the same as for the Command View EVA installation problem): Create a group with the name “Users” and add “Domain Admins” and “Domain Users” into it. Then rerun the setup and it should finish without problems.

Final words

Even if this failure is caused by bad coding habits, it confirms my personal recommendation to always deploy english operating systems.

Data Protector: Exchange 2010 database recovery from copy session fails

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

The recovery of an Exchange mailbox using a recovery database is usually no big deal. Simply restore the database, create a recovery database and recover the mailbox or items from the mailbox. Sometimes you have the luck that the customer has licensed the Data Protector Exchange 2010 Granular Recovery for Exchange (GRE). This was unfortunately not true in my case. Okay, so let’s do it the old way. The needed tape was available in the library and luckily it was a full backup. So I quickly added a disk to the VM and started the recovery of the database to a temporary location. At this point, the disaster took its course…

Constraints

This isn’t a general problem of Data Protector, it’s a bug. The following constraints met in my case:

  • Recovery source is a copy session (e.g. post-backup object copy to tape)
  • Data Protector 7.03
  • Microsoft Exchange 2010 SP2
  • Recovery to temporary location

This is the output of “omnidb -session 2014/08/01-6 -detail” for the wanted session. As you can see, this is a copy of session 2014/08/01-3, which was without protection and therefore was removed by Data Protector.

When trying to recover the DB to a temporary location, I ran into this error:

The session I tried to recover was 2014/08/01-06, the session that was chosen by Data Protector for recovery was 2014/08/01-03. To make the long story short: You can fix it with a site specific path for Data Protector 7.03. Log a call at the HP Support and ask for SSPNT700_038. Plase note that you need a valid HP Software Support contract to get this patch! The patch delivers fixes for the three defects QCCR2A51280, QCCR2A53555 and QCCR2A46724. The patch is delivered as a ZIP file and contains binaries and libraries which has to be installed on the Cell Manager and the Exchange server. The patch contains files for Windows on x64, HP-UX 11.23, 11.31 on IA64, HP-UX 11.23, 11.31 on PA RISC and Linux on x64. When running Data Protector on Windows, you have to replace the following files:

On the Exchange server

e2010_bar.exe
PSHAPI.dll
vssbar.exe

On the Data Protector Cell Manager

asm.exe
bsm.exe
csm.exe
dbsm.exe
libob2eadm.dll
libob2ecdb.dll
libob2ecmn.dll
libob2emmdb.dll
mmd.exe
msm.exe
rsm.exe

You have to stop the Data Protector services on the Cell Manager and the Data Protector Inet service on the Exchange server before you can replace the files. Make a backup of the files before you replace them. After the file replacement start the services and try the restore again. In my case the restore didn’t worked after applying the patch: It failed with the same error. I opened a case at HP and after a few day I got the notice, that lab engineering was involved in the case. Short after that notice, the support sent me two files (QCCR2A54842_TM1) that I had to replace on the Data Protector Cell Manager (libob2ecdb.dll & libob2ecmn.dll). Both files were part of SSPNT700_038, so you can interprete this as a patch for a patch. ;) This patch did the trick and the restore was successful. The root cause was, that a database query returns the wrong session for the recovery. So if you face the same problems, ask HP for the site-specific patch. If the problem remains, ask for QCCR2A54842_TM1.