Tag Archives: bug

Data Protector: Exchange 2010 database recovery from copy session fails

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

The recovery of an Exchange mailbox using a recovery database is usually no big deal. Simply restore the database, create a recovery database and recover the mailbox or items from the mailbox. Sometimes you have the luck that the customer has licensed the Data Protector Exchange 2010 Granular Recovery for Exchange (GRE). This was unfortunately not true in my case. Okay, so let’s do it the old way. The needed tape was available in the library and luckily it was a full backup. So I quickly added a disk to the VM and started the recovery of the database to a temporary location. At this point, the disaster took its course…

Constraints

This isn’t a general problem of Data Protector, it’s a bug. The following constraints met in my case:

  • Recovery source is a copy session (e.g. post-backup object copy to tape)
  • Data Protector 7.03
  • Microsoft Exchange 2010 SP2
  • Recovery to temporary location

This is the output of “omnidb -session 2014/08/01-6 -detail” for the wanted session. As you can see, this is a copy of session 2014/08/01-3, which was without protection and therefore was removed by Data Protector.

When trying to recover the DB to a temporary location, I ran into this error:

The session I tried to recover was 2014/08/01-06, the session that was chosen by Data Protector for recovery was 2014/08/01-03. To make the long story short: You can fix it with a site specific path for Data Protector 7.03. Log a call at the HP Support and ask for SSPNT700_038. Plase note that you need a valid HP Software Support contract to get this patch! The patch delivers fixes for the three defects QCCR2A51280, QCCR2A53555 and QCCR2A46724. The patch is delivered as a ZIP file and contains binaries and libraries which has to be installed on the Cell Manager and the Exchange server. The patch contains files for Windows on x64, HP-UX 11.23, 11.31 on IA64, HP-UX 11.23, 11.31 on PA RISC and Linux on x64. When running Data Protector on Windows, you have to replace the following files:

On the Exchange server

e2010_bar.exe
PSHAPI.dll
vssbar.exe

On the Data Protector Cell Manager

asm.exe
bsm.exe
csm.exe
dbsm.exe
libob2eadm.dll
libob2ecdb.dll
libob2ecmn.dll
libob2emmdb.dll
mmd.exe
msm.exe
rsm.exe

You have to stop the Data Protector services on the Cell Manager and the Data Protector Inet service on the Exchange server before you can replace the files. Make a backup of the files before you replace them. After the file replacement start the services and try the restore again. In my case the restore didn’t worked after applying the patch: It failed with the same error. I opened a case at HP and after a few day I got the notice, that lab engineering was involved in the case. Short after that notice, the support sent me two files (QCCR2A54842_TM1) that I had to replace on the Data Protector Cell Manager (libob2ecdb.dll & libob2ecmn.dll). Both files were part of SSPNT700_038, so you can interprete this as a patch for a patch. ;) This patch did the trick and the restore was successful. The root cause was, that a database query returns the wrong session for the recovery. So if you face the same problems, ask HP for the site-specific patch. If the problem remains, ask for QCCR2A54842_TM1.

DataCore In SANsymphony-V 10: Potential for data corruption

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

This is only a short blog post. Just got an e-mail from the DataCore Support. They found a critical bug in SANsymphony-V 10.0.0.0 which should be fixed with Update 1. Only VMware customers are affected, because the bug is related to VMware Thin Provisioning Thresholds. Update 1 is planned for early September 2014. If you’re running SANsymphony-V 10.0.0.0 open an incident at the DataCore Support to get an available hotfix. If you have planned to update to SANsymphony-V 10, delay this update until the release of SANsymphony-V 10 Update 1.

Users on Exchange 2013 can’t open public folders or shared mailboxes on an Exchange 2007/ 2010

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

When moving users to Exchange 2013 it can happen, that they can’t access public folders housed on the old Exchange 2010 or 2007 server. The same can happen to shared mailboxes (mailboxes with Full Access permissions). The users are constantly prompted for credentials or they get this message:

This can be a huge problem during a migration. Microsoft described this in KB2834139. This error is caused by a misconfigured Outlook security setting, called “Logon network security”. If you experience this issue, check the “Logon network security” setting. If it’s set to “Anonymous Authentication”, then you experience the in KB2834139 described problem. Otherwise you have another problem. Check the “Logon network security” settings in your Outlook client. I took this screenshots from a Outlook 2013, but it looks the same in Outlook 2010. With this setting you will have a problem:

outlook_settings_login_security_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

With one of these settings, it will work.

outlook_settings_login_security_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

outlook_settings_login_security_3

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

You can change the setting and try the access to a public folder or shared mailbox. If you can access the public folder or shared mailbox, then you have to change some settings on the Exchange Server 2013 Client Access Server (CAS).

Open an Exchange Management Shell:

I would recommend to execute “iisreset” after changing the settings. Please note, that this interrupts the client access for a short moment! After a restart of the Outlook client or during the next Autodiscover, the client should get the correct settings and the access to the public folders and shared mailboxes should work.

Active Directory property ‎’homeMDB‎’ is not writeable on recipient

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

During an Exchange 2013 migration project the  first attempt to migrate a mailbox failed with the following error:

The error message clearly stated, that this was a permission issue. A quick search pointed me to the right direction. I found a thread in the TechNet forums, in which the same error message were discussed. This error occurs, if the Exchange Trusted Subsystem group is missing in the ACL of the user object. This group contains the exchange server and it’s usually inherited from the domain object to all child containers and objects. I checked the ACL of the user and the Exchange Trusted Subsystem group was missing in the ACL. This was caused by the disabled permissions inheritance. An object ACL with disabled permissions inheritance is sometimes called a protected ACL. Bill Long wrote a nice Power Shell script to search for object which have permissions inheritance disabled.

How to enable permissions inheritance?

Start the Active Directory Users and Computers MMC. Click “View” and enable “Advanced Features”. Otherwise the needed tabs will not shown.

permissions_inheritance_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Open the user that can’t be migrated. Switch to the “Security” tab. Click “Advanced” and enable the “Include inheritable permissions from this object’s parent” check box. Click “OK” and confirm popups regarding added ACL entries.

permissions_inheritance_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Check if the Exchange Trusted Subsystem group is now part of the ACL. If yes, try the migration of the mailbox again.

Why is  permissions inheritance disabled?

Good question, the customer asked me the same question. But the answer is easy. As above mentioned, is a ACL with disabled permission inheritance called a protected ACL. When should a ACL be protected? If the ACL of the associated object shouldn’t be changed, when a permission from a parent object is inherited. A good example for such an object is the Administrator account. Active Directory protects members of the following groups

  • Enterprise Admins
  • Schema Admins
  • Domain Admins
  • Administrators
  • Account Operators
  • Server Operators
  • Print Operators
  • Backup Operators
  • Cert Publishers

and the following users

  • Administrator
  • krbtgt

with three simple steps:

  1. Set the value of the AdminCount attribute to 1.
  2. Disable permission inheritance
  3. Apply permissions from the AdminSDHolder template

This process runs once an hour on a Active Directory Domain Controller and recovers the ACL of the two user accounts and the member accounts of the mentioned groups to a protected state and well-known permissions. What happens if you add an user account to the Domain Admins group? AdminCount is set to 1, the permission inheritance is disabled and the permissions specified in the AdminSDHolder template are applied. If you remove the user account from the Domain Admins group, the changes are not rolled back and so  permission inheritance stays disabled! Changes to parent ACLs are not applied to the user account and, in the case of my customer, the Exchange Trusted Subsystem group wasn’t added to the user objects ACL.

You can search for accounts with AdminCount=1 easily with the PowerShell:

Why is this a problem? With AdminCount=1 and disabled permission inheritance you will face a lot of problems:

  • Changes to an object ACLs are rolled back after an hour
  • Send-As doesn’t work
  • ActiveSync doesn’t work
  • Language settings in Outlook Web Access isn’t saved
  • ..and some other nasty effects

I have seen this error so often. Mostly at customers, that use normal work accounts for administrative tasks, or that add normal user accounts to admin groups during the setup of new profiles or clients. The solution is easy: Don’t do it!

VM shows alarm – but no alarm triggered

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today I observed a strange behaviour of several VMs at a customer. Several VMs in a cluster showed an alarm, but neither on the alarm tab of the VM, nor the alarm section at the bottom of the C# client showed an error.

vm_alarm_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The customer still uses vSphere 5.0. An upgrade to 5.5 is on the roadmap. The symptoms:

  • not all VMs in the cluster were affected
  • all, except one VM, were running on one specific host
  • the alarm on a VM disappeared after a vMotion
  • no trigger for the alarm could be found
  • vSphere HA status “protected”

The similar behaviour could be observed, if a VM is moved to another cluster using VMware vMotion technology. In this case, a VM isn’t protected by vSphere HA after it’s moved to another HA protected cluster. This is described in KB2012682. I ran a small PowerCLI script, which I found in this VMTN thread.

But I couldn’t find any hint pointing me to the root cause. At this point I was able to rule out several things:

  • Problem with the VM
  • Issue with HA/ FT
  • Problem is host related
  • Resource usage of a VM

At the end the workaround was easy: Restart of the management agents. After the restart, the alarm icons disappeared. Please note, that the customer was using VMware vSphere vCenter Server 5.0 U1a (latest release is 5.0 U3) and ESXI 5.0 U3 (latest release is 5.0 Patch 7). Maybe it’s a bug and it’s fixed in a newer release.

Patch available: VMware vSphere 5.5 U1 NFS APD bug

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

In April 2014 was a bug in vSphere 5.5 U1 discovered, which can lead to APD events with NFS datastores.iSCSI, FC or FCoE aren’t affected by this bug, but potentially every NFS installation running vSphere 5.5 U1 was at risk. This bug is described in KB2076392. Luckily none of my customers ran into this bug, but this is more due to the fact, that most of my customers use FC/ FCoE or iSCSI. Until today the only solution was to avoid the upgrade to U1 and to use vSphere 5.5 GA (with some patches to fix the Heartbleed bug).

On 10. June 2014 VMware released the patch ESXi550-201406401-SG, which fixes the NFS APD bug. This patch also includes an updated OpenSSL library to address CVE-2014-0224. CVE-2014-0224 is not the Heartbleed bug! This is a new vulnerability in OpenSSL, which was discovered on 05. June 2014.

You can install the patch using the VMware Update Manager (VUM) or you can download the ZIP file and install it using the “esxcli software vib" command. No matter which way you choose to install the patch, you should install the patch! Especially if you already use vSphere 5.5 U1 and NFS (most NetApp or Nutanix customers…).