Tag Archives: veeam

Veeam B&R: “Rescan of Manually Added” failed

I got this error in a new deployment of Veeam Backup & Replication 9.5 Update 4. The error occured every day at 9 pm.

The solution to this issue is pretty simple. Make sure that you allow the consumption of licenses for free agents. You will find this option under General > License.

Another workaround is to disable the protection group. Right click “Manually Added” under “Physical & Cloud Infrastructure” and click “Disable”.

Let me know if one of these workarounds worked for you. :)

Veeam Backup & Replication: Backup of Microsoft Active Directory Domain Controller VMs

To backup a virtual machine, Veeam Backup & Replication needs two permissions:

  • permission to access and backup the VM, as well as the
  • permission to do specific tasks inside the VM

to guarantee a consistent backup. The former persmission is granted by the user account that is used to access the VMware vCenter server (sorry for the VMW focust at this point). Usually, this account has the Administrator role granted at the vCenter Server level. The latter permission is granted by a user account that has permissions inside the guest operating system.

geralt / pixabay.com/ Creative Commons CC0

Something I often see in customer environments is the usage of the Domain Administrator account. But why? Because everything works when this account is used!

There are two reasons for this:

  • This account is part of the local Administrator group on every server and client
  • customers tend to grant the Administrator role to the Domain Admins group on vCenter Server level

In simple words: Many customers use the same account to connect to the vCenter, and for the application-aware processing of Veeam Backup & Replication. At least for Windows servers backups.

Houston, we have a problem!

Everything is fine until customers have to secure their environments. One of the very first things customers do, is to protect the Administrator account. And at this point, things might go wrong.

Using a service account to connect to the vCenter server is easy. This can be any account from the Active Directory, or from the embedded VMware SSO domain. I tend to create a dedicated AD-based service account. For the necessary permissions in the vCenter, you can grant this account Administrator permissions, or you can create a new user role in the vCenter. Veeam offers a PDF document which documents the necessary permissions for the different Veeam tasks.

The next challenge is the application-aware processing. For Microsoft SQL Server, the user account must have the sysadmin privileges on the Microsoft SQL Server. For Microsoft Exchange, the user must be member of the local Administrator group. But in case of a Active Directory Domain Contoller things get complicated.

A Domain Controller does not have a local user database (SAM). So what user account or group membership is needed to backup a domain controller using application-aware processing?

This statement is from a great Veeam blog post:

Permissions: Administrative rights for target Active Directory. Account of an enterprise administrator or domain administrator.

So the service account used to backup a domain controller is one of the most powerful accounts in the active directory.

There is no other way. You need a Domain or Enterprise Administrator account. I tend to create a dedicated account for this task.

I recommend to create a service account to connect the vCenter, and which is added to the local Administrator group on the servers to backup, and I create a dedicated Domain/ Enterprise Administrator account to backup the virtual Domain Controllers.

The advantage is that I can change apply different fine-grained password policies to this accounts. Sure, you can add more security by creating more accounts for different servers, and applications, add a dedicated role to the vCenter for Veeam etc. But this apporach is easy enough to implement, and adds a significant amount of user account security to every environment that is still using DOMAIN\Administrator to backup their VMs.

Veeam and StoreOnce: Wrong FC-HBA driver/ firmware causes Windows BSoD

One of my customers bought a very nice new backup solution, which consists of a

  • HPE StoreOnce 5100 with ~ 144 TB usable capacity,
  • and a new HPE ProLiant DL380 Gen10 with Windows Server 2016

as new backup server. StoreOnce and backup server will be connected with 8 Gb Fibre-Channel and 10 GbE to the existing network and SAN. Veeam Backup & Replication 9.5 U3a is already in use, as well as VMware vSphere 6.5 Enterprise Plus. The backend storage is a HPE 3PAR 8200.

This setup allows the usage of Catalyst over Fibre-Channel together with Veeam Storage Snapshots, and this was intended to use.

I wrote about a similar setup some month ago: Backup from a secondary HPE 3PAR StoreServ array with Veeam Backup & Replication.

The OS on the StoreOnce was up-to-date (3.16.7), Windows Server 2016 was installed using HPE Intelligent Provisioning. Afterwards, a drivers and firmware were updated using the latest SPP 2018.11 was installed. So all drivers and firmware were also up-to-date.

After doing zoning and some other configuration tasks, I installed Veeam Backup and Replication 9.5 U3, configured my Catalyst over Fibre-Channel repository. I configured a test backup… and the server failed with a Blue Screen of Death… which is pretty rare since Server 2008 R2.

geralt / pixabay.com/ Creative Commons CC0

I did some tests:

  • backup from 3PAR Storage Snapshots to Catalyst over FC repository – BSoD
  • backup without 3PAR Storage Snapshots to Catalyst over FC repository – BSoD
  • backup from 3PAR Storage Snapshots to Catalyst over LAN repository – works fine
  • backup without 3PAR Storage Snapshots to Catalyst over LAN repository – works fine
  • backup from 3PAR Storage Snapshots to default repository – works fine
  • backup without 3PAR Storage Snapshots to default repository – works fine

So the error must be caused by the usage of Catalyst over Fibre-Channel. I filed a case at HPE, uploaded gigabytes of memory dumps and heard pretty less during the next week.

HPE StoreOnce Support Matrix FTW!

After a week, I got an email from the HPE support with a question about the installed HBA driver and firmware. I told them the version number and a day later I was requested to downgrade (!) drivers and firmware.

The customer has got a SN1100Q (P9D93A & P9D94A) HBA in his backup server, and I was requested to downgrade the firmware to version 8.05.61, as well as the driver to 9.2.5.20. And with this firmware and driver version, the backup was running fine (~ 750 MB/s hroughput).

I found the HPE StoreOnce Support Matrix on the SPOCK website from HPE. The matrix confirmed the firmware and driver version requirement (click to enlarge).

Fun fact: None of the listed HBAs (except the Synergy HBAs) is supported with the latest StoreOnce G2 products.

Lessons learned

You should take a look at those support matrices – always! HPE confirmed that the first level recommendation “Have you trieed to update to the latest firmware” can cause similar problems. The fact, that the factory ships the server with the latest firmware does not make this easier.

Backup from a secondary HPE 3PAR StoreServ array with Veeam Backup & Replication

When taking a backup with Veeam Backup & Replication, a VM snapshot is created to get a consistent state of the VM. The snapshot is taken prior the backup, and it is removed after the successful backup of the VM. The snapshot grows during its lifetime, and you should keep in mind, that you need some free space in the datastore for snapshots. This can be a problem, especially in case of multiple VM backups at a time, and if the VMs share the same datastore.

Benefit of storage snapshots

If your underlying storage supports the creation of storage snapshots, Veeam offers an additional way to create a consistent state of the VMs. In this case, a storage snapshot is taken, which is presented to the backup proxy, and is then used to backup the data. As you can see: No VM snapshot is taken.

Now one more thing: If you have a replication or synchronous mirror between two storage systems, Veeam can do this operation on the secondary array. This is pretty cool, because it takes load from you primary storage!

Backup from a secondary HPE 3PAR StoreServ array

Last week I was able to try something new: Backup from a secondary HPE 3PAR StoreServ array. A customer has two HPE 3PAR StoreServ 8200 in a Peer Persistence setup, a HPE StoreOnce, and a physical Veeam backup server, which also acts as Veeam proxy. Everything is attached to a pretty nice 16 Gb dual Fabric SAN. The customer uses Veeam Backup & Replication 9.5 U3a. The data was taken from the secondary 3PAR StoreServ and it was pushed via FC into a Catalyst Store on a StoreOnce. Using the Catalyst API allows my customer to use Synthetic Full backups, because the creation is offloaded to StoreOnce. This setup is dramatically faster and better than the prior solution based on MicroFocus Data Protector. Okay, this last backup solution was designed to another time with other priorities and requirements. it was a perfect fit at the time it was designed.

This blog post from Veeam pointed me to this new feature: Backup from a secondary HPE 3PAR StoreServ array. Until I found this post, it was planned to use “traditional” storage snapshots, taken from the primary 3PAR StoreServ.

With this feature enabled, Veeam takes the snapshot on the 3PAR StoreServ, that is hosting the synchronous mirrored virtual volume. This graphic was created by Veeam and shows the backup workflow.

Veeam/ Backup process/ Copyright by Veeam

My tests showed, that it’s blazing fast, pretty easy to setup, and it takes unnecessary load from the primary storage.

In essence, there are only three steps to do:

  • add both 3PARs to Veeam
  • add the registry value and restart the Veeam Backup Server Service
  • enable the usage of storage snapshots in the backup job

How to enable this feature?

To enable this feature, you have to add a single registry value on the Veeam backup server, and afterwards restart the Veeam Backup Server service.

  • Location: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
  • Name: Hp3PARPeerPersistentUseSecondary
  • Type: REG_DWORD (0 False, 1 True)
  • Default value: 0 (disabled)

Thanks to Pierre-Francois from Veeam for sharing his knowledge with the community. Read his blog post Backup from a secondary HPE 3PAR StoreServ array for additional information.

Veeam backups fails because of time differences

Last week I had an interesting incident at a customer. The customer reported that one of multiple Veeam backup jobs jobs constantly failed.

jarmoluk/ pixabay.com/ Creative Commons CC0

The backup job included two VMs, and the backup of one of these VMs failed with this error:

The verified the used credentials for that job, but re-entering the password does not solved the issue. I then checked the Veeam backup logs located under %ProgramData%\Veeam\Backup (look for the Agent.Job_Name.Source.VM_Name.vmdk.log) and found VDDK Error 3014:

The user, that was used to connect to the vCenter, was an Active Directory located account. The account were granted administrator privileges root of the vCenter. Switching from an AD located account to Administrator@vsphere.local solved the issue. Next stop: vmware-sts-idmd.log on the vCenter Server appliance. The error found in this log confirmed my theory, that there was an issue with the authentication itself, not an issue with the AD located account.

To make a long story short: Time differences. The vCenter, the ESXi hosts and some servers had the wrong time. vCenter and ESXi hosts were using the Domain Controllers as time source.

This is the ntpq  output of the vCenter. You might notice the jitter values on the right side, both noted in milliseconds.

After some investigation, the root cause seemed to be a bad DCF77 receiver, which was connected to the domain controller that was hosting the PDC Emulator role. The DCF77 receiver was connected using an USB-2-LAN converter. Instead of using a DCF77 receiver, the customer and I implemented a NTP hierarchy using a valid NTP source on the internet (pool.ntp.org).

Consider the Veeam Network transport mode if you use NFS datastores

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

I’m using Veeam Backup & Replication (currently 8.0 Update 3) in my lab environment to backup some of my VMs to a HP StoreOnce VSA. The VMs reside in a NFS datastore on a Synology DS414slim NAS, the StoreOnce VSA is located in a local datastore (RAID 5 with SAS disks) on one of my ESXi hosts. The Veeam backup server is a VM and it’s also the Veeam Backup Proxy. The transport mode selection is set to “Automatic selection”.

Veeam Backup & Replication offers three different backup proxy transport modes:

  • Direct SAN Access
  • Virtual Appliance
  • Network

The Direct SAN Access transport mode is the recommended mode, if the VMs are located in shared datastores (connected via FC or iSCSI). The Veeam Backup Proxy needs access to the LUNs, so the Veeam Backup Proxy is mostly a physical machine. The data is directly read by the backup proxy from the LUNs. The Virtual Appliance mode uses the SCSI hot-add feature, which allows the attachment of disks to a running VM. In this case, the data is read by the backup proxy VM from the directly attached SCSI disk. In contrast to the Direct SAN Access mode, the Virtual Appliance mode can only be used if the backup proxy is a VM. The third transport mode is the Network transport mode. It can be used in any setup, regardless if the backup proxy is a VM or a physical machine. In this mode, the data is retrieved via the ESXi management network and travels over the network using the Network Block Device protocol (NBD or NBDSSL, latter is encrypted). This is a screenshot of the transport mode selection dialog of the backup proxy configuration.

veeam_transport_mode_selection

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

As you can see, the transport mode selection will happen automatically if you doesn’t select a specific transport mode. The selection will occur in the following order: Direct SAN Access > Virtual Appliance > Network. So if you have a physical backup proxy without direct access to the VMFS datastore LUNs, Veeam Backup & Replication will use the Network transport mode. A virtual backup proxy will use the Virtual Appliance transport. This explains why Veeam uses the Virtual Appliance transport mode in my lab environment.

Some days ago, I configured E-Mail notifications for some vCenter alarms. During the last nights I got alarm messages: A host has been disconnected from the vCenter. But the host reconnected some seconds later. Another observation was, that a running vSphere Client lost the connection to the vCenter Update Manager during the night. After some troubleshooting, I found indications, that some of my VMs became unresponsive. With this information, I quickly found the VMware KB article “Virtual machines residing on NFS storage become unresponsive during a snapshot removal operation (2010953)“. Therefore I switched the transport from Virtual Appliance to Network.

I recommend to use Network transport mode instead Virtual Appliance transport mode, if you have a virtual Veeam Backup Proxy and NFS datastores. I really can’t say that it’s running slower as the Virtual Appliance transport mode. It just works.

Important note for PernixData FVP customers

Remember to exclude the Veeam Backup Proxy VM from acceleration, if you use Virtual Appliance or NBD transport mode. If you use datastore policies, blacklist the VM or configure it as VADP appliance. If you use VM policies, simply doesn’t configure a policy for the Veeam Backup Proxy VM. If you use Direct SAN access, you need a pre- and a post-backup script to suspend the cache population during the backup. Check Frank Dennemans blog post about “PernixData FVP I/O Profiling PowerCLI commands“.

Error 1325: VBRCatalog is not a valid short file name

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

While upgrading a rather old (but very stable) Veeam Backup & Replication 6.1 installation to 8.0 Update 3 (with intermediate step to 6.5), I ran into a curious error. Right after the welcome screen, this error message

veeam_error_1325

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

appeared. A closer look into the BackupSetup.log (you can find this log in the %temp% dir. Just enter %temp% into the Explorer address bar) resulted in this very interesting log entry:

First, the VBRCatalog folder was located under D:\Veeam, so why the hell was the CATALOGPATH property changed to E:\VBRCatalog? I searched the registry for for E:\VBRCatalog and found multiple entries for it. One of the entries was located under “HKLM\SOFTWARE\Wow6432Node\Veeam\Veeam Backup Catalog”. The entry under “HKLM\SOFTWARE\Veeam\Veeam Backup Catalog” pointed to the correct path. I found some other entries, e.g. in connection with Windows Installer.

After changing all found entries to the correct path, the update went smooth. The reason for this error was that the VBRCatalog was moved after the installation. I did this more than 3 years ago and followed Veeam KB1453. But this article only describes the change of the CatalogPath entry under “HKLM\SOFTWARE\Veeam\Veeam Backup Catalog”. You have to change all references to the old VBRCatalog path! Otherwise you will run into the same error as I.

Protection of virtual machines with HP StoreOnce VSA & Veeam Backup & Replication v7

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

HP StoreOnce Appliances or VSA offers three different types of backup destinations:

  • Virtual Tape Library (VTL)
  • NAS (CIFS or NFS)
  • StoreOnce Catalyst

If you use Veeam Backup & Replication, the NAS feature is possibly worth a try. Using the NAS feature, the StoreOnce appliance or VSA offers a CIFS or NFS share, which can be used as a backup destionation. Today I want to show you how you can use a NAS share of a StoreOnce VSA with Veeam Backup & Replication.To backup virtual maschines with Veeam Backup & Replication to a HP StoreOnce VSA you need at least three things:

  • a HP StoreOnce VSA
  • a backup server with Veeam Backup & Replication
  • at least one VM

I have built such an environment in my lab. I described the process how to get and deploy StoreOnceVSA in this article. I will not cover the installation of Veeam Backup & Replication, because this is really easy. This article only covers the configuration of the StoreOnce VSA in terms of the backup of VMs, the configuration of a Veeam backup job as well as some backup tests.

Configure StoreOnce VSA

The first step is the configuration of a NAS share. To do so, login into the StoreOnce Management Console. If you haven’t changed the default login credentials, you can login with:

Username: Admin
Password: admin

Click on “NAS” and then on “Shares”. Click “Create” on the upper-right.

storeonce_create_cifs_share_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Name your CIFS share. Click on “Create”. If you like you can enable authentication, so that you must provide a username and password to access the share. You can enable this option under “NAS”. By this the configuration of the StoreOnce VSA has finished.

storeonce_create_cifs_share_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Configure Veeam Repository

First we need to add a new backup repository. Name the repository and click “Next”.

veeam_config_repo_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The repository type is “Shared folder”. Click “Next”.

veeam_config_repo_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Enter the UNC path to the StoreOnce VSA share. If you have configured authentication, you need to provide credentials in ordner to access the share. Depending on your environment, you can use the IP or the FQDN.

veeam_config_repo_3

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click on “Advanced” lower-right.

veeam_config_repo_4

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Compression reduces the efficiency of deduplication. So enable the checkbox “Decompress backup data bl ocks before storing”. This ensures that the data blocks are decompressed before written to the StoreOnce VSA.

veeam_config_repo_5

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I have disabled the vPower NFS. Depending on your needs you can leave this option enabled.

veeam_config_repo_6

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Check the made settings and click “Next”.

veeam_config_repo_7

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Congratulations. You now have a CIFS repository which points to your StoreOnce VSA.

veeam_config_repo_8

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Configure a backup job

Now it’s time to create a backup job. Create a new job and give it a name. Click “Next”.

veeam_config_job_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Add the backup objects. This can be for example a cluster, a host, vApps or one or more VMs.

veeam_config_job_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Select your newly created repository. Click on “Advanced”.

veeam_config_job_3

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

It’s recommended to use the “Incremental” backup mode when using backup appliances like HP StoreOnce or EMC Data Domain. This backup mode has a lower performance impact on the backup appliance, but it needs more disk space, because of regular full backups. This backup mode starts with a full backup and makes subsequently incremental backups, until a new full backup is created. Because the backup appliance does deduplication, additional disk space due to regular full backups doesn’t use much additional disk space.

veeam_config_job_4

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Switch to the “Storage” tab. It’s a good idea to use deduplication on the backup proxy and the backup appliance. But the deduplication should be optmized for “Local target”. Veeam uses then a block size of 1 MB. Compression should be disabled to optimizes the deduplication ratio. This will result into a higher network load!

veeam_config_job_5

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Switch to the “vSphere” tab and enable the checkbox “Enable VMware Tools quiescence”. Please note, that this does not support log truncation for applications like Exchange or SQL Server! Click “OK”, then “Next”.

veeam_config_job_6

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

If you want to backup applications like Exchange or SQL Server, tick the “Enable application-aware image processing” checkbox. Click “Next”.

veeam_config_job_7

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Configure the job scheduling depending on your needs. Click “Create”

veeam_config_job_8

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Check the summary and then click “Finish”.

veeam_config_job_9

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The job is now ready to run, either by starting the job manually or wait until the scheduled job starts.

Backup tests

I’ve done some tests. During the first full backup, Veeam Backup & Replication processed 17 GB and transferred 10 GB.

veeam_job_result_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The StoreOnce VSA wrote 7,4 GB to disk.

veeam_job_result_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

A second full backup showed similar results as the first full backup

veeam_job_result_3

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

But only additional 0,2 GB were written to disk, and the deduplication ratio raised from 2.4 to 4,7.

veeam_job_result_4

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

You can access the CIFS share using the Windows Explorer. You can see, that the stored files doesn’t differ from a “normal” CIFS repository.

veeam_job_result_5

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Final words

Using a HP StoreOnce VSA as CIFS repository for Veeam Backup & Replication is really easy and doesn’t need much configuration. But some points should be considered. The settings that I have used are recommended for maximizing the backup capacity and retention time. If you focus on RTO, you should consider backing up critical VMs to a physical backup proxy with local disks (or access to a fast storage system) in addition to a backup to a StoreOnce appliance or VSA. The backup and restore performance depends on the backup target (StoreOnce VSA) and the backup proxy (in my case a VM). Depending on your environment and the number of backup proxies, backup targets and repositories you have to make additional decisions.