Tag Archives: veeam

Backup from a secondary HPE 3PAR StoreServ array with Veeam Backup & Replication

When taking a backup with Veeam Backup & Replication, a VM snapshot is created to get a consistent state of the VM. The snapshot is taken prior the backup, and it is removed after the successful backup of the VM. The snapshot grows during its lifetime, and you should keep in mind, that you need some free space in the datastore for snapshots. This can be a problem, especially in case of multiple VM backups at a time, and if the VMs share the same datastore.

Benefit of storage snapshots

If your underlying storage supports the creation of storage snapshots, Veeam offers an additional way to create a consistent state of the VMs. In this case, a storage snapshot is taken, which is presented to the backup proxy, and is then used to backup the data. As you can see: No VM snapshot is taken.

Now one more thing: If you have a replication or synchronous mirror between two storage systems, Veeam can do this operation on the secondary array. This is pretty cool, because it takes load from you primary storage!

Backup from a secondary HPE 3PAR StoreServ array

Last week I was able to try something new: Backup from a secondary HPE 3PAR StoreServ array. A customer has two HPE 3PAR StoreServ 8200 in a Peer Persistence setup, a HPE StoreOnce, and a physical Veeam backup server, which also acts as Veeam proxy. Everything is attached to a pretty nice 16 Gb dual Fabric SAN. The customer uses Veeam Backup & Replication 9.5 U3a. The data was taken from the secondary 3PAR StoreServ and it was pushed via FC into a Catalyst Store on a StoreOnce. Using the Catalyst API allows my customer to use Synthetic Full backups, because the creation is offloaded to StoreOnce. This setup is dramatically faster and better than the prior solution based on MicroFocus Data Protector. Okay, this last backup solution was designed to another time with other priorities and requirements. it was a perfect fit at the time it was designed.

This blog post from Veeam pointed me to this new feature: Backup from a secondary HPE 3PAR StoreServ array. Until I found this post, it was planned to use “traditional” storage snapshots, taken from the primary 3PAR StoreServ.

With this feature enabled, Veeam takes the snapshot on the 3PAR StoreServ, that is hosting the synchronous mirrored virtual volume. This graphic was created by Veeam and shows the backup workflow.

Veeam/ Backup process/ Copyright by Veeam

My tests showed, that it’s blazing fast, pretty easy to setup, and it takes unnecessary load from the primary storage.

In essence, there are only three steps to do:

  • add both 3PARs to Veeam
  • add the registry value and restart the Veeam Backup Server Service
  • enable the usage of storage snapshots in the backup job

How to enable this feature?

To enable this feature, you have to add a single registry value on the Veeam backup server, and afterwards restart the Veeam Backup Server service.

  • Location: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
  • Name: Hp3PARPeerPersistentUseSecondary
  • Type: REG_DWORD (0 False, 1 True)
  • Default value: 0 (disabled)

Thanks to Pierre-Francois from Veeam for sharing his knowledge with the community. Read his blog post Backup from a secondary HPE 3PAR StoreServ array for additional information.

Veeam backups fails because of time differences

Last week I had an interesting incident at a customer. The customer reported that one of multiple Veeam backup jobs jobs constantly failed.

jarmoluk/ pixabay.com/ Creative Commons CC0

The backup job included two VMs, and the backup of one of these VMs failed with this error:

The verified the used credentials for that job, but re-entering the password does not solved the issue. I then checked the Veeam backup logs located under %ProgramData%\Veeam\Backup (look for the Agent.Job_Name.Source.VM_Name.vmdk.log) and found VDDK Error 3014:

The user, that was used to connect to the vCenter, was an Active Directory located account. The account were granted administrator privileges root of the vCenter. Switching from an AD located account to Administrator@vsphere.local solved the issue. Next stop: vmware-sts-idmd.log on the vCenter Server appliance. The error found in this log confirmed my theory, that there was an issue with the authentication itself, not an issue with the AD located account.

To make a long story short: Time differences. The vCenter, the ESXi hosts and some servers had the wrong time. vCenter and ESXi hosts were using the Domain Controllers as time source.

This is the ntpq  output of the vCenter. You might notice the jitter values on the right side, both noted in milliseconds.

After some investigation, the root cause seemed to be a bad DCF77 receiver, which was connected to the domain controller that was hosting the PDC Emulator role. The DCF77 receiver was connected using an USB-2-LAN converter. Instead of using a DCF77 receiver, the customer and I implemented a NTP hierarchy using a valid NTP source on the internet (pool.ntp.org).

Consider the Veeam Network transport mode if you use NFS datastores

I’m using Veeam Backup & Replication (currently 8.0 Update 3) in my lab environment to backup some of my VMs to a HP StoreOnce VSA. The VMs reside in a NFS datastore on a Synology DS414slim NAS, the StoreOnce VSA is located in a local datastore (RAID 5 with SAS disks) on one of my ESXi hosts. The Veeam backup server is a VM and it’s also the Veeam Backup Proxy. The transport mode selection is set to “Automatic selection”.

Veeam Backup & Replication offers three different backup proxy transport modes:

  • Direct SAN Access
  • Virtual Appliance
  • Network

The Direct SAN Access transport mode is the recommended mode, if the VMs are located in shared datastores (connected via FC or iSCSI). The Veeam Backup Proxy needs access to the LUNs, so the Veeam Backup Proxy is mostly a physical machine. The data is directly read by the backup proxy from the LUNs. The Virtual Appliance mode uses the SCSI hot-add feature, which allows the attachment of disks to a running VM. In this case, the data is read by the backup proxy VM from the directly attached SCSI disk. In contrast to the Direct SAN Access mode, the Virtual Appliance mode can only be used if the backup proxy is a VM. The third transport mode is the Network transport mode. It can be used in any setup, regardless if the backup proxy is a VM or a physical machine. In this mode, the data is retrieved via the ESXi management network and travels over the network using the Network Block Device protocol (NBD or NBDSSL, latter is encrypted). This is a screenshot of the transport mode selection dialog of the backup proxy configuration.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

As you can see, the transport mode selection will happen automatically if you doesn’t select a specific transport mode. The selection will occur in the following order: Direct SAN Access > Virtual Appliance > Network. So if you have a physical backup proxy without direct access to the VMFS datastore LUNs, Veeam Backup & Replication will use the Network transport mode. A virtual backup proxy will use the Virtual Appliance transport. This explains why Veeam uses the Virtual Appliance transport mode in my lab environment.

Some days ago, I configured E-Mail notifications for some vCenter alarms. During the last nights I got alarm messages: A host has been disconnected from the vCenter. But the host reconnected some seconds later. Another observation was, that a running vSphere Client lost the connection to the vCenter Update Manager during the night. After some troubleshooting, I found indications, that some of my VMs became unresponsive. With this information, I quickly found the VMware KB article “Virtual machines residing on NFS storage become unresponsive during a snapshot removal operation (2010953)“. Therefore I switched the transport from Virtual Appliance to Network.

I recommend to use Network transport mode instead Virtual Appliance transport mode, if you have a virtual Veeam Backup Proxy and NFS datastores. I really can’t say that it’s running slower as the Virtual Appliance transport mode. It just works.

Important note for PernixData FVP customers

Remember to exclude the Veeam Backup Proxy VM from acceleration, if you use Virtual Appliance or NBD transport mode. If you use datastore policies, blacklist the VM or configure it as VADP appliance. If you use VM policies, simply doesn’t configure a policy for the Veeam Backup Proxy VM. If you use Direct SAN access, you need a pre- and a post-backup script to suspend the cache population during the backup. Check Frank Dennemans blog post about “PernixData FVP I/O Profiling PowerCLI commands“.

Error 1325: VBRCatalog is not a valid short file name

While upgrading a rather old (but very stable) Veeam Backup & Replication 6.1 installation to 8.0 Update 3 (with intermediate step to 6.5), I ran into a curious error. Right after the welcome screen, this error message


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

appeared. A closer look into the BackupSetup.log (you can find this log in the %temp% dir. Just enter %temp% into the Explorer address bar) resulted in this very interesting log entry:

First, the VBRCatalog folder was located under D:\Veeam, so why the hell was the CATALOGPATH property changed to E:\VBRCatalog? I searched the registry for for E:\VBRCatalog and found multiple entries for it. One of the entries was located under “HKLM\SOFTWARE\Wow6432Node\Veeam\Veeam Backup Catalog”. The entry under “HKLM\SOFTWARE\Veeam\Veeam Backup Catalog” pointed to the correct path. I found some other entries, e.g. in connection with Windows Installer.

After changing all found entries to the correct path, the update went smooth. The reason for this error was that the VBRCatalog was moved after the installation. I did this more than 3 years ago and followed Veeam KB1453. But this article only describes the change of the CatalogPath entry under “HKLM\SOFTWARE\Veeam\Veeam Backup Catalog”. You have to change all references to the old VBRCatalog path! Otherwise you will run into the same error as I.

Protection of virtual machines with HP StoreOnce VSA & Veeam Backup & Replication v7

HP StoreOnce Appliances or VSA offers three different types of backup destinations:

  • Virtual Tape Library (VTL)
  • NAS (CIFS or NFS)
  • StoreOnce Catalyst

If you use Veeam Backup & Replication, the NAS feature is possibly worth a try. Using the NAS feature, the StoreOnce appliance or VSA offers a CIFS or NFS share, which can be used as a backup destionation. Today I want to show you how you can use a NAS share of a StoreOnce VSA with Veeam Backup & Replication.To backup virtual maschines with Veeam Backup & Replication to a HP StoreOnce VSA you need at least three things:

  • a HP StoreOnce VSA
  • a backup server with Veeam Backup & Replication
  • at least one VM

I have built such an environment in my lab. I described the process how to get and deploy StoreOnceVSA in this article. I will not cover the installation of Veeam Backup & Replication, because this is really easy. This article only covers the configuration of the StoreOnce VSA in terms of the backup of VMs, the configuration of a Veeam backup job as well as some backup tests.

Configure StoreOnce VSA

The first step is the configuration of a NAS share. To do so, login into the StoreOnce Management Console. If you haven’t changed the default login credentials, you can login with:

Username: Admin
Password: admin

Click on “NAS” and then on “Shares”. Click “Create” on the upper-right.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Name your CIFS share. Click on “Create”. If you like you can enable authentication, so that you must provide a username and password to access the share. You can enable this option under “NAS”. By this the configuration of the StoreOnce VSA has finished.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Configure Veeam Repository

First we need to add a new backup repository. Name the repository and click “Next”.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The repository type is “Shared folder”. Click “Next”.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Enter the UNC path to the StoreOnce VSA share. If you have configured authentication, you need to provide credentials in ordner to access the share. Depending on your environment, you can use the IP or the FQDN.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click on “Advanced” lower-right.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Compression reduces the efficiency of deduplication. So enable the checkbox “Decompress backup data bl ocks before storing”. This ensures that the data blocks are decompressed before written to the StoreOnce VSA.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I have disabled the vPower NFS. Depending on your needs you can leave this option enabled.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Check the made settings and click “Next”.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Congratulations. You now have a CIFS repository which points to your StoreOnce VSA.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Configure a backup job

Now it’s time to create a backup job. Create a new job and give it a name. Click “Next”.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Add the backup objects. This can be for example a cluster, a host, vApps or one or more VMs.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Select your newly created repository. Click on “Advanced”.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

It’s recommended to use the “Incremental” backup mode when using backup appliances like HP StoreOnce or EMC Data Domain. This backup mode has a lower performance impact on the backup appliance, but it needs more disk space, because of regular full backups. This backup mode starts with a full backup and makes subsequently incremental backups, until a new full backup is created. Because the backup appliance does deduplication, additional disk space due to regular full backups doesn’t use much additional disk space.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Switch to the “Storage” tab. It’s a good idea to use deduplication on the backup proxy and the backup appliance. But the deduplication should be optmized for “Local target”. Veeam uses then a block size of 1 MB. Compression should be disabled to optimizes the deduplication ratio. This will result into a higher network load!


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Switch to the “vSphere” tab and enable the checkbox “Enable VMware Tools quiescence”. Please note, that this does not support log truncation for applications like Exchange or SQL Server! Click “OK”, then “Next”.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

If you want to backup applications like Exchange or SQL Server, tick the “Enable application-aware image processing” checkbox. Click “Next”.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Configure the job scheduling depending on your needs. Click “Create”


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Check the summary and then click “Finish”.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The job is now ready to run, either by starting the job manually or wait until the scheduled job starts.

Backup tests

I’ve done some tests. During the first full backup, Veeam Backup & Replication processed 17 GB and transferred 10 GB.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The StoreOnce VSA wrote 7,4 GB to disk.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

A second full backup showed similar results as the first full backup


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

But only additional 0,2 GB were written to disk, and the deduplication ratio raised from 2.4 to 4,7.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

You can access the CIFS share using the Windows Explorer. You can see, that the stored files doesn’t differ from a “normal” CIFS repository.


Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Final words

Using a HP StoreOnce VSA as CIFS repository for Veeam Backup & Replication is really easy and doesn’t need much configuration. But some points should be considered. The settings that I have used are recommended for maximizing the backup capacity and retention time. If you focus on RTO, you should consider backing up critical VMs to a physical backup proxy with local disks (or access to a fast storage system) in addition to a backup to a StoreOnce appliance or VSA. The backup and restore performance depends on the backup target (StoreOnce VSA) and the backup proxy (in my case a VM). Depending on your environment and the number of backup proxies, backup targets and repositories you have to make additional decisions.