Tag Archives: backup

Using HP StoreOnce as target for Windows Server Backup (WSB)

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Some days ago, I blogged about the new HP StoreOnce software release 3.13.0. This release included several fixes. One fix wasn’t mentioned by me, although it’s interesting.

  • Fixed issue where Windows 2012 R2 built-in native backup was not supported with 3.12.x software (BZ 61232)

Windows Server Backup (WSB) is part of Windows Server since Windows Server 2008. WSB can create bare metal backups and recover those backups. The same applies to system state backups, file level backups, Hyper-V VMs, Exchange etc. Very handy for small environmens. Backup can be stored on disk or on a file share. With Server 2012, the file share must be SMB3 capable. So if it’s not a Windows file server, the NAS that offers the file share has to be SMB3 capable. This doesn’t apply to Windows Server 2008 (R2).

With StoreOnce 3.13.0, HP has fixed this. Starting with 3.13.0, you can use a CIFS share on a StoreOnce appliance as a target for Windows Server Backup. This allows you to take advantage of the benefits of StoreOnce, like industry-leading deduplication and replication technology.

I was able to test this new feature with StoreOnce VSA appliances in my lab, as well as with a customers StoreOnce 4700 appliance.

Download you free copy of the HP StoreOnce Free 1 TB VSA today and give it a try!

HP StoreOnce Backup System software version 3.13.0 is available

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Since september 2015, the latest version of HP StoreOnce backup system software is available. The latest release 3.13.0 is available for HP StoreOnce VSA, 6500, B6200 multi-node and all single node systems running software version 3.x. This also applies to some D2D 2500, 4100 and 4300 single-node backup systems running software versions 2.x. Make sure that you take a look into customer notice c03729283 for details on performing the conversion.

This release comes with some nice enhancements, e.g.

  • support for jumbo frames
  • 1 TB, 5 TB and 10 TB thin provisioned disks are now supported for the VSA
  • VSA Hyper-V PowerShell Installer script
  • HP StoreOnce VSA Ubuntu KVM bash Installer
  • Single Entitlement page on the HP StoreOnce GUI

Two fixes caught my attention, because I saw both of them in the wild:

  • Fixed issue where running NAS Replication data jobs are cancelled when files on target share are simultaneously accessed (BZ 63232)

I saw this error in environments, where customers used StoreOnce CIFS shares as backup target with Veeam Endpoint Backup together with StoreOnce Replication.

  • Fixed issue where NAS CIFS shares are inaccessible when special characters are in the share description (BZ 62263)

I ran into this issue some months ago and wrote about it (HP StoreOnce: Avoid special characters in NAS share description). I was in contact with the StoreOnce engineering because of this issue. Cool that this has been fixed!

I strongly recommend to update to 3.13.0! You can download the software from the HP Software Depot (make sure that you download the right StoreOnce software for your HW appliance or VSA!). All you need is a HP Passport login. To update the software, upload the RPM into the repository folder on the appliance with SFTP. Login using SSH and execute three simple commands.

The update takes only a few minutes. Subsequently, the appliance reboots.

Error 1325: VBRCatalog is not a valid short file name

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

While upgrading a rather old (but very stable) Veeam Backup & Replication 6.1 installation to 8.0 Update 3 (with intermediate step to 6.5), I ran into a curious error. Right after the welcome screen, this error message

veeam_error_1325

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

appeared. A closer look into the BackupSetup.log (you can find this log in the %temp% dir. Just enter %temp% into the Explorer address bar) resulted in this very interesting log entry:

First, the VBRCatalog folder was located under D:\Veeam, so why the hell was the CATALOGPATH property changed to E:\VBRCatalog? I searched the registry for for E:\VBRCatalog and found multiple entries for it. One of the entries was located under “HKLM\SOFTWARE\Wow6432Node\Veeam\Veeam Backup Catalog”. The entry under “HKLM\SOFTWARE\Veeam\Veeam Backup Catalog” pointed to the correct path. I found some other entries, e.g. in connection with Windows Installer.

After changing all found entries to the correct path, the update went smooth. The reason for this error was that the VBRCatalog was moved after the installation. I did this more than 3 years ago and followed Veeam KB1453. But this article only describes the change of the CatalogPath entry under “HKLM\SOFTWARE\Veeam\Veeam Backup Catalog”. You have to change all references to the old VBRCatalog path! Otherwise you will run into the same error as I.

HP StoreOnce: Avoid special characters in NAS share description

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

While I was playing with my shiny, new HP StoreOnce VSA in my lab, I noticed a curious behavior. I created a NAS share for some tests with Veeam Backup & Replication. Creating a new share is nothing fancy. You can create a share in two ways:

  • using the GUI, or
  • using the CLI

So I created a new share:

storeonce_create_share_gui_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Nothing special, as you can see. I opened up a Explorer, typed in the IP address of my StoreOnce VSA and… saw no share.

storeonce_access_share_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I repeated this process a couple of times, always with the same result. Then I went to the CLI and checked the newly created share:

So far, so good. I removed the share and tried to create the share using the CLI:

The command failed, no share was created. I verified the syntax, but the syntax of the command was correct. I started to simplify the command and removed the description.

The share was added with the default description. I removed the share and tried it again with my description. The command failed again. After removing the ampersand (&) from the description, the share could be added. I tried the same from the GUI. Using the GUI, a share with a ampersand (&) in the description field could be added, but it wasn’t accessible. Even if I removed the ampersand (&) from the share description. I had to remove and re-create the share with a valid description. Unfortunately the GUI allows you to create the share, even if the CLI command fails with the same settings. The GUI also doesn’t allow you to create the share with an empty description.

At this point, I can’t say if this is a bug or a known behaviour. I’m in contact with HP to clarify this. But you should avoid the usage of special characters in the NAS share description.

EDIT

Today, I got an e-mail from the HP StoreOnce Engineering. They informed me, that it’s not only the ampersand (&) you should avoid. You should avoid a set of special characters

  • `
  • *
  • &
  • %
  • +
  • multiple space in a row

These characters can cause minor issues with Windows tools, like the Explorer. As a result, these special characters were banned in the latest 3.12.x CIFS server code. However this ban was not messaged in the GUI. As a fix, this ban will be lifted from 3.12.2 software to allow the use of the above mentioned special characters.

VMware publishes patch for ESXi 6.0 CBT bug (KB2114076)

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today, VMware has published the long-awaited patch for the ESXi 6.0 CBT bug. This patch is the result of a problem, which is described in KB2114076 (Backing up a virtual machine with Change Block Tracking (CBT) enabled fails after upgrading to or installing VMware ESXi 6.0). All customers that upgraded to ESXi 6.0 or installed ESXi 6.0 were affected.

Symptoms of this bug were:

  • Powering on virtual machines fails
  • Expanding the size of a virtual disk fails
  • Taking virtual machine quiesced snapshots fails
  • Error messages like “An error occurred while taking a snapshot: msg.snapshot.error-QUIESCINGERROR” (vSphere Client), “WARNING: CBT: 191: No memory available! Called from 0x4180219af50e” (vmkernel.log) or “Creating cbt node 92b78c-cbt failed with error Cannot allocate memory (0xbad0014, Out of memory)” in the vmware.log of the affected virtual machine

A workaround was to disable CBT, which resulted in longer running backups.

You can download the patch (ESXi600-201505401-BG) from VMware to manually update your ESXi hosts. Otherwise you can use VMware Update Manager to patch your hosts. A reboot is necessary!

If you decide to manually update your hosts with this patch, you have to use the “esxcli software vib” command to install the patch.

  • Download the ZIP file and upload it to a datastore that is reachable for the host you want to patch (e.g. a local datastore)
  • Bring the host into the maintenance mode
  • Connect with SSH to your ESXi host and run:

  • Reboot the host and leave the maintenance mode

If you have disabled CBT for all or some of your VMs, make sure that you re-enable CBT.

Safe (or safer) than backup to tape: HP StoreOnce

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

When talking to SMB customers, most of them don’t want to talk about their backup strategy. It’s paradox: They know that data loss can ruin their business, but they don’t want to invest money into a fully tested recovery concept (I try to avoid the word “backup concept” – Recovery is the key). Because of tight budgets and lacking knowledge, many customers use traditional concepts in a virtualized world. This often ends  in traditional backup applications with agents deployed into guest OS, and backups that are written to tape (or worse: On USB disks). If you ask a customer “Why do you store your data on tape?”, only a few argue with costs per GB or performance. Most the customer argue with something like

  • “We’re doing this for years, so why we should change it?”
  • “We have to store our tapes offsite”
  • “There is a corporate policy that forces us to store our backups on tape”

In most cases, the attempt to sell a backup-to-disk appliance (like HP StoreOnce backup system) dies with the last arguments. Customers tend to doesn’t trust designs in which they don’t have a backup on tape. Some customers have a strong desire to have a tape which is labled with “MONDAY” or “FRIDAY FULL”. To be honest: Usually I see this behaviour only at SMB customers. Backup-to-disk appliances are often described as

  • expensive,
  • complex, and
  • vulnerable

None of them applies to a HP StoreOnce backup system. Not even expensive, if you not only focus on CAPEX.

HP StoreOnce

Please allow me to write some sentences about HP StoreOnce.

A HP StoreOnce backup system is available as physical or virtual appliance. HP offers a broad range of physical appliances that can store between 5,5 TB and 1.728 TB BEFORE deduplication. The virtual StoreOnce VSA is available with a capacity of 4 TB, 10 TB and 50 TB before deduplication. And don’t forget the free 1 TB StoreOnce VSA! All HP StoreOnce backup systems, regardless if physical appliance or VSA, share the same StoreOnce deduplication technology, as well as the same replication and security features. In fact, the StoreOnce VSA runs the same (linux based) software as the physical applanices and vice versa. You can add features by adding software options:

  • HP StoreOnce Catalyst
  • HP StoreOnce Replication
  • HP StoreOnce Security Pack
  • HP StoreOnce Enterprise Manager

HP StoreOnce Catalyst allow the seamless movement of deduplicated data across StoreOnce capable devices. This means, that a HP Data Protector media agent can deduplicate data during a backup, write the data to a HP StoreOnce backup system, and then the data can replicated to another HP StoreOnce backup system. All without the need to rehydrate on the source, and deduplicate it on the destionation again. The StoreOnce VSA includes a HP StoreOnce Catalyst license!

HP StoreOnce Replication enables an appliance or a VSA to act as a target in a replication relationship. Only the target needs to be licensed. Fan-in describes the number of possible source appliances.

ModelFan-in
StoreOnce VSA8
StoreOnce 27008
StoreOnce 290024
StoreOnce 450024
StoreOnce 470050
StoreOnce 490050
StoreOnce 6200384

As you can see, even the StoreOnce VSA can used as a target for up to 8 source appliances. Replication is a licensable feature, except for the StoreOnce VSA. The StoreOnce VSA includes the replication license!

HP StoreOnce Enterprise Manager can be obtained for free and allows you to monitor up to 400 physical appliances or StoreOnce VSAs. It provides monitoring, reporting, trend analysis and forcasting. It integrates with the StoreOnce GUI for single pane-of-glass management for physical appliances and VSA.

HP StoreOnce Security Pack enables data-at-rest and data-in-flight encryption (using IPsec and only for StoreOnce Catalyst), as well as secure data deletion. Here applies the same as for the HP StoreOnce Catalyst and Replication license: The StoreOnce VSA includes this license already.

HP StoreOnce Deduplication

Deduplication is nothing really new. In simple terms it’s a technique to reduce the amount of stored data by removing redundancies. Data that is being detected as redundant, isn’t stored again on the disks. Only a pointer to the stored data is set. This runs the risk of potential data loss. What if the original block gets corrupted? Grist to the mill of the tape lovers (Tapes never fail… for sure…).

Integrity Plus

Don’t worry. I won’t bore you with stuff about a dead (or nearly dead) CPU architecture. Integrity Plus is HPs approach for an end-to-end verification process. Let’s take a look on how data comes into a StoreOnce backup system. From a client perspective, you can choose between Virtual Tape Library (VTL), NAS emulation (CIFS or NFS) and StoreOnce Catalyst.

When data is written to a VTL, a CRC is computed for each block and it’s stored together with the data block on disk. During a restore, a CRC is computed for every block that is read from disk and it’s compared to the initial stored CRC. If it differs, a SCSI check condition is reported. Because NAS emulation and StoreOnce Catalyst doesn’t use SCSI protocol, no CRC is computed and stored to disk. The integrity of the written data is guaranteed in other ways.

At the beginning of the deduplication process, the incoming data is divided into chunks. HP uses a variable length for each data chunk, but in average a data chunk is 4 KB. A smaller chunk size leads to better deduplication results. A SHA-1 (AFAIK 160 bit) hash is computed for each data chunk. This chunk hash is used to identify duplicate data by comparing it to other chunk hashes. At this point, a sparse index is used to find possible candidates of redundant data chunks. Instead of holding all chunk hashes in the memory, only a few hashes are stored in the RAM. The remaining chunk hashes are stored as metadata on disk. The container index contains a list of chunk hashes and a pointer to the data container where the data chunk is stored. Before data chunks are stored on disk, multiple chunks are compressed (using LZO) and a SHA-1 checksum is computed for the compressed chunks. This checksum is stored on disk. When the compressed data is decompressed, a new checksum is computed and it’s compared to the stored SHA-1 checksum. Metadata and container index files are protected with MD5 checksums. In addition, a transaction log file is maintained for the whole process and the sparse index is frequently flushed to disk.

When data is coming into the StoreOnce backup system, a match with a chunk hash in the memory can lead the system (using the sparse index, metadata and container index files) to containers with associated data chunk (e.g. data chunks that represent a backup VM). And if a data chunk of the incoming data is a duplicate, it is very likely that many of the following data chunks are also duplicates.

All physical appliances use RAID 6 to protect data in case of disk failures. Only the HP StoreOnce 2700 uses a RAID 5, because the appliance can only hold 4 SAS-NL disks. When using StoreOnce VSA, you can use any RAID level for the underlying storage. But you should use something above RAID 0…

Conclusion

Let’s summarize:

  • RAID
  • Supercapacitors on RAID controllers to protect write cache in case of power loss
  • ECC memory
  • Integrity Plus to protect the data within the StoreOnce backup system
  • StoreOnce Replication to replicate data to another HP StoreOnce backup systems
  • data-at-rest, data-in-flight encryption and secure deletion with StoreOnce Security Pack

Sounds very safe to me. Tape isn’t dead. Tape has its right to exist. But backup to tape isn’t safer than a backup to a StoreOnce backup system. Latter can offer you faster backups AND restores, new backup and recovery options (e.g. backups in RoBo offices that are replicated to the central datacenter). Think about the requirements for storing tapes (temperature, humidity, physical access), regular recovery tests, copy tapes to newer tapes etc. Consider not only CAPEX. Also remember OPEX.

A HP StoreOnce backup system is perfect for SMBs. It simplifies backup and recovery and it can offer new opportunities. Testdrive it using the free 1 TB StoreOnce VSA! Remember: The StoreOnce VSA includes StoreOnce Replication, Catalyst and the Security Pack! Even the free 1 TB StoreOnce VSA.

HP offers 1TB StoreOnce VSA for free

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

A free StoreOnce VSA, like the well known 1 TB StoreVirtual VSA? That would be too cool to be real. But it is real! Since February, HP offers a free 1 TB version of their StoreOnce VSA. I totally missed this announcement, but thanks to Calvin Zito I noticed it today:

The link leads to another blog post from Ashwin Shetty (Can you protect your data for free? Introducing the new free 1TB StoreOnce VSA), in which he provides more information about the free 1 TB StoreOnce VSA.

HP StoreOnce VSA

HP StoreOnce VSA runs with the same software as the hardware-based StoreOnce appliances, but it’s delivered as a VM. You can run the VM on top of VMware ESXi, Microsoft Hyper-V or KVM. Beside the free 1 TB license, the StoreOnce VSA can purchased with 4 TB, 10 TB or 50 TB capacity (usable, non-deduplicated). In contrast to the hardware-based appliances, the StoreOnce VSA comes with licenses for replication and StoreOnce Catalyst. This makes the StoreOnce VSA a perfect fit for remote and branch offices. You can quickly deploy the StoreOnce VSA and replicate the backuped data to the central datacenter. But you can also deploy the VSA with the 4 TB, 10 TB or 50 TB license in your central datacenter and use it as a replication target for StoreOnce VSAs in the remote and branch offices (the replication target needs the replication license). A single VSA can act as replication target for up to 8 StoreOnce VSA and/ or StoreOnce appliances. You can scale the free 1 TB license with license upgrades to 4 TB, 10 TB and 50 TB. The StoreOnce VSA supports Catalyst, VTL (iSCSI) and as NAS (CIFS or NFS) backup targets. Take a look into the QuickSpecs for more information. I also recommend to read the two blog posts from Ashwin Shetty on Around the Storage Block:

Last year I’ve published several posts about the StoreOnce VSA. I recommend to download the free 1 TB StoreOnce VSA and to play with it. Some of my blog posts should help you get started.

HP Data Protector: JSONizer error when restoring from StoreOnce

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

After installing the Data Protector patch bundle 8.13, you may ran into this error when trying to restore data from a HP StoreOnce appliance.

This problem is known and it is described in QCCR2A56465. A fix is available (new BMA, CMA, MMA and RMA binaries). Simply open a service request and ask for the fix. Make sure that you add a copy of the session messages or a screenshot to the service request.

HP Data Protector: Can’t delete old DCBF directories

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
This applies to upgrades from Data Protector 6.x and 7.x to 8.x and 9.x.

It seems that today is my debugging day… Yesterday I performed a Data Protector update from 7.03 to 8.13. During this update, the Data Protector IDB is migrated to another database format. Last night the backups went smoothly, but today I noticed that two old Detail Catalog Binary File (DCBF) directories were still referenced in the HP Data Protector IDB.

The two directories with the “db40” inside the path are old DCBF directories. Because the directories contained actively used DCBF files, I relocated the files and did a “omnidbutil -remap_dcdir”:

A quick check after the relocation showed no errors.

Looking good. Time to remove the old DCBF directories:

Did I mentioned that today was my debugging day? To make a long story short: HP switched the path separator character for the Data Protector IDB. They are using now a / instead a \ on both platforms (Windows & UNIX). During the update, this change is not performed correctly. Sebastian Koehler wrote a small SQL script that fixes this problem. Check his blog post (he had the same problem as me).

This is the script (you can also find it on Sebastian Koehlers blog):

This is the output of the script when I run it.

You can clearly see that the wrong path separator is used for the old DB40 directories (the upper part of the output). Compare it to the output of omnidbutil -list_dcdirs! The lower part shows that the correct path separator was set. After the run of the script I was able to delete the old DCBF directories.

Thanks to Sebastian, who described this bug.

HP StoreOnce Enterprise Manager v1.3 installation fails on non-English OS

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Sometimes the easy jobs seems to be the hardest. Especially if you have to deal with high-quality software… As part of a project I had to install and configure a HP StoreOnce 4500 appliance in combination with HP Data Protector 8.12 and a StoreEver MSL2024 G3 tape-library. No big deal – until I hit the part, when I had to install HP StoreOnce Enterprise Manager v1.3 (SEM) on the new backup server. The installation failed with this error:

hp_sem_installation_error

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Setup could not provide access privileges to “C:\Program Files\Hewlett-Packard\HP StoreOnce Enterprise Manager\RMSDataStore\Postgres\data” directory of PostgreSQL

First I blamed UAC for this. I disabled it and re-ran the setup with “Run as administrator…”. The setup failed again. I tried the setup on my “rack ‘n stack” laptop (Windows 8.1 Enterprise) and it fails with exactly the same error. I was puzzled, because I had SEM running on Windows 8.1 – until I decided to re-install this laptop with a german-language Windows 8.1. At this point it dawned on me a sense of foreboding. I remembered a bug in a HP Command View EVA release, which couldn’t be installed on non-Englisch operating systems, because the setup relied on hardcoded, english group names. A quick cross-check with a english Windows Server 2008 R2 VM confirmed this and I was able to install SEM without problems.

I checked the temporary folder in which the setup files were extracted. I found a batch file calles “SEMS_InstallDB”. This batch file included this line:

It seems that the user group “Users” was hardcoded. In a german-language OS this group is called “Benutzer”. With knowing this, the solution was easy (and the same as for the Command View EVA installation problem): Create a group with the name “Users” and add “Domain Admins” and “Domain Users” into it. Then rerun the setup and it should finish without problems.

Final words

Even if this failure is caused by bad coding habits, it confirms my personal recommendation to always deploy english operating systems.