Category Archives: HPE

Data Protector Exchange GRE and IP-less Exchange DAG

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

When dealing with Microsoft Exchange restore requests, you will come across three different restore situations:

  • a database
  • a single mailbox
  • a single mailbox item (mail, calendar entry etc.)

Restoring a complete database is not a complicated task, but restoring a single mailbox, or a single mailbox item, is. First, you need to restore the mailbox, that includes the desired mailbox, into a recovery database. Then you can restore the mailbox, or the mailbox items, from the recovery database. Some of the tasks can only be done with the Exchange Management Shell.

The HPE Data Protector Granular Recovery Extension (GRE) for Microsoft Exchange helps you to simplify the necessary steps to recover a single mailbox, or mailbox items. But the GRE can only assist you during the restore. It hids the above described tasks behind a nice GUI. The backup of Microsoft Exchange is still something you have to do with HPE Data Protector.

Database Availability Group without an Administrative Access Point

With Exchange 2013 SP1, Microsoft introduced the IP-less Database Availability Group (DAG). This type of DAG does not need a Cluster Name Object (CNO), and therefore has no IP address. With Exchange 2016, the IP-less DAG is the default DAG configuration.

But how to backup a DAG, that has no IP address and no name? It is easier than imagined. You have to create a DNS A-Record that includes all IP addresses of the cluster nodes, resulting in a DNS round-robin A-Record. You also have to install the Data Protector Disk Agent and On-line Extension on all cluster nodes. After that, you simply import the DAG by using the DNS A-Record into Data Protector. Then you can proceed with the creation and configuration of a backup job, that uses the newly imported cluster.

Backup runs fine, but the GRE fails

During the test phase of a new Exchange 2016 cluster, a customer of mine discovered a strange error, when he tried to restore a mailbox, or mailbox item, using the Exchange GRE.

Either parsing of command builder output failed or no databases were backed up.
Data Protector Exchange GRE Error

Patrick Terlisten/ Creative Commons CC0

The customer and I double-checked the installation of the GRE on both nodes. Everything was fine. We also found out, that Data Protector was able to list the backup objects. This is a shortened output of the command.

Object Name                                                     Object type
exchange-2010.domain.tld:/25e0d4ac-5a63-4035-b718-09d07bbbba47/DB3 E2010
dag-backup.domain.tld:/c7982a7e-76cd-4759-9d03-019c0c410957/DB03 E2010
dag-backup.domain.tld:/5d3faa67-6fd9-493e-bf56-5044f75620a8/DB01 E2010

As you can see, dag-backup.domain.tld is the DNS A-Record, that was created to backup the DAG with Data Protector.

Connection between A-Record and DAG name

It took some time to get this sorted, but at the end, a new A-Record was the key. The DAG has a name, e.g. customer-dag1.domain.tld. But there is no matching A-Record, and the DAG has no IP address.

When the GRE searches for available database backups, it stumbles over the mismatch between the DAG name, that is reported by the Exchange organization, and the name of the Data Protector client that was used to backup the databases.

The key to success was to change the DNS A-Record from dag-backup.domain.tld to customer-dag1.domain.tld. Latter is the name of the DAG, that is given during DAG creation. After removing the Data Protector client, the re-import of the DAG with the new A-Record, and a successful backup, the customer was able to restore mailboxes and mailbox items using the GRE for Microsoft Exchange.

This process is not described in detail in the Data Protector documentation. All you find is this foot note in the Data Protector Platform Integration Matrix (page 12, foot note 19):

Microsoft Exchange Server DAG configured without a Cluster Administrator Access Point is supported with Round Robin DNS mapping of DAG name to all the node IPs.

Make sure that the DNS round-robin A-Record matches your DAG name.

Wrong iovDisableIR setting on ProLiant Gen8 might cause a PSOD

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

TL;DR: There’s a script at the bottom of the page that fixes the issue.

Some days ago, this HPE customer advisory caught my attention:

Advisory: (Revision) VMware – HPE ProLiant Gen8 Servers running VMware ESXi 5.5 Patch 10, VMware ESXi 6.0 Patch 4, Or VMware ESXi 6.5 May Experience Purple Screen Of Death (PSOD): LINT1 Motherboard Interrupt

And there is also a corrosponding VMware KB article:

ESXi host fails with intermittent NMI PSOD on HP ProLiant Gen8 servers

It isn’t clear WHY this setting was changed, but in VMware ESXi 5.5 patch 10, 6.0  patch 4, 6.0 U3 and, 6.5 the Intel IOMMU’s interrupt remapper functionality was disabled. So if you are running these ESXi versions on a HPE ProLiant Gen8, you might want to check if you are affected.

To make it clear again, only HPE ProLiant Gen8 models are affected. No newer (Gen9) or older (G6, G7) models.

Currently there is no resolution, only a workaround. The iovDisableIR setting must set to FALSE. If it’s set to TRUE, the Intel IOMMU’s interrupt remapper functionality is disabled.

To check this setting, you have to SSH to each host, and use esxcli  to check the current setting:

[[email protected]:~] esxcli system settings kernel list -o iovDisableIR

Name          Type  Description                                 Configured  Runtime  Default
------------  ----  ---------------------------------------     ----------  -------  -------
iovDisableIR  Bool  Disable Interrupt Routing in the IOMMU...   FALSE       FALSE    TRUE

I have written a small PowerCLI script that uses the Get-EsxCli cmdlet to check all hosts in a cluster. The script only checks the setting, it doesn’t change the iovDisableIR setting.

Here’s another script, that analyzes and fixes the issue.

Checking the 3PAR Quorum Witness appliance

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Two 3PAR StoreServs running in a Peer Persistence setup lost the connection to the Quorum Witness appliance. The appliance is an important part of a 3PAR Peer Persistence setup, because it acts as a tie-breaker in a split-brain scenario.

While analyzing this issue, I saw this message in the 3PAR Management Console:

3PAR Quorum Witness Status

Patrick Terlisten/ Creative Commons CC0

In addition to that, the customer got e-mails that the 3PAR StoreServ arrays lost the connection to the Quorum Witness appliance. In my case, the CouchDB process died. A restart of the appliance brought it back online.

How to check the Quorum Witness appliance?

You can check the status of the appliance with a simple web request. The documentation shows a simple test based on curl. You can run this direct from the BASH of the appliance.

[[email protected] ~]# curl
[[email protected] ~]#

But you can also use the PowerShell cmdlet Invoke-WebRequest.

PS C:\Users\patrick> Invoke-WebRequest -Uri

StatusCode        : 200
StatusDescription : OK
Content           : {"couchdb":"Welcome","version":"1.0.4"}

RawContent        : HTTP/1.1 200 OK
                    Content-Length: 40
                    Cache-Control: must-revalidate
                    Content-Type: text/plain;charset=utf-8
                    Date: Mon, 30 Jan 2017 08:31:37 GMT
                    Server: CouchDB/1.0.4 (Erlang OTP/R14B04)

Forms             : {}
Headers           : {[Content-Length, 40], [Cache-Control, must-revalidate], [Content-Type, text/plain;charset=utf-8],
                    [Date, Mon, 30 Jan 2017 08:31:37 GMT]...}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : mshtml.HTMLDocumentClass
RawContentLength  : 40

If you add /witness to the URL, you can test the access to the database, which is used for Peer Persistence.

PS C:\Users\patrick> Invoke-WebRequest -Uri

StatusCode        : 200
StatusDescription : OK
Content           : {"db_name":"witness","doc_count":5,"doc_del_count":4,"update_seq":149557915,"purge_seq":0,"compact_
RawContent        : HTTP/1.1 200 OK
                    Content-Length: 234
                    Cache-Control: must-revalidate
                    Content-Type: text/plain;charset=utf-8
                    Date: Mon, 30 Jan 2017 08:36:38 GMT
                    Server: CouchDB/1.0.4 (Erlang OTP/R14B04)

Forms             : {}
Headers           : {[Content-Length, 234], [Cache-Control, must-revalidate], [Content-Type,
                    text/plain;charset=utf-8], [Date, Mon, 30 Jan 2017 08:36:38 GMT]...}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : mshtml.HTMLDocumentClass
RawContentLength  : 234

If you get a connection error, check if the beam process is running.

[[email protected] ~]# netstat -tulpen |grep 8080
tcp        0      0      *                   LISTEN      495        10726      1643/beam
[[email protected] ~]#

If not, reboot the appliance. This can be done without downtime. The appliance comes only into play, if a failover occurs.

HPE ProLiant PowerShell SDK

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Some days ago, my colleague Claudia and I started to work on a new project: A greenfield deployment consisting of some well known building blocks: HPE ProLiant, HPE MSA, HPE Networking (now Aruba) and VMware vSphere. Nothing new for us, because we did this a couple times together. But this led us to the idea, to automate some tasks. Especially the configuration of the HPE ProLiants: Changing BIOS settings and configuring the iLO.

Do not automate what you have not fully understood

Some of the wisest words I have ever said to a customer. Modifying the BIOS and iLO settings is a well understood task. But if you have to deploy a bunch of ProLiants, this is a monotonous, and therefore error prone process. Perfect for automation!

Scripting Tools for Windows PowerShell

To support the automation of HPE ProLiant deployments, HPE offers the Scripting Tools for Windows PowerShell. HPE offers the PowerShell modules free for charge. There are three different downloads:

  • iLO cmdlets
  • BIOS cmdlets
  • Onboard Administrator (OA) cmdlets

The iLO cmdlets include PowerShell cmdlets to configure and manage iLO on HPE ProLiant G7, Gen8 or Gen9 servers. The BIOS cmdlets does not support G7 servers, so you can only configure and manage legacy and UEFI BIOS for Gen8 (except DL580) and all Gen9 models. The OA cmdlets support the configuration and management of the HPE Onboard Administrator, which is used with HPEs well known ProLiant BL blade servers. The OA cmdlets need at least  OA v3.11, whereby v4.60 is the latest version available.  All you need to get started are

  • Microsoft .NET Framework 4.5, and
  • Windows Management Framework 3.0 or later

If you are using Windows 8 or 10, you already have PowerShell 4 respectively PowerShell 5.

Support for HPE ProLiant Gen9 iLO RESTful API

If you have ever seen a HPE ProLiant Gen9 booting up, you might have noticed the iLO RESTful API icon down right. Depending on the server model, the BIOS cmdlets utilize the ILO4 RESTful API. But the iLO RESTful API ecosystem is it worth to be presented in an own blog post. Stay tuned.

Documentation and examples

HPE offers a simple documentation for the BIOS, iLO and OA cmdlets. You can find the documentation in HPEs Information Library. Documentation is important, but sometimes example code is necessary to quickly ramp up code. Check HPEs PowerShell SDK GitHub repository for examples.

Time to code

I’m keen on it and curious to automate some of my regular deployment tasks with these PowerShell modules. Some of these tasks are always the same:

  • change the power management and other BIOS settings
  • change the network settings of the iLO
  • change the initial password of the iLO administrator account and create additional iLO user accounts

Further automation tasks are not necessarily related to the HPE ProLiant PowerShell SDK, but to PowerShell, respectively VMware PowerCLI. PowerShell is great to automate the different aspects and modules of an infrastructure deployment. You can use it to build your own tool box.

Enable IPv6 SLAAC on HPE OfficeConnect 1920 switches

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

The HPE OfficeConnect 1920 switch series is designed for SMBs. The switch is perfect for small environments, that require features like VLANs, routing or 802.1x. This switch is smart-managed, so it has “only” a web interface and only a limited CLI.

I have two switches in my lab: A 1910-8G and the successor, a 1920-24G. Although the device supports IPv6, it doesn’t support SLAAC (Stateless Address Autoconfiguration) by default. The switch does not send router advertisements (RA). I’m using IPv6 in my lab (Stateless DHCPv6 + SLAAC), so the missing RAs were a problem for me, or at least, annoying. Fortunately you can change the default behaviour.

Enable router advertisements (RA)

To change the default behaviour of the HPE 1920, you have to use the CLI. The CLI is very limited, but there’s a hidden CLI command, which enables access to nearly all available features. If you are familiar with HPEs Comware based switches, you will notice, that the switch is a Comware-based device.

login as: admin
[email protected]'s password:

* Copyright (c) 2010-2016 Hewlett Packard Enterprise Development LP          *
* Without the owner's prior written consent,                                 *
* no decompiling or reverse-engineering shall be allowed.                    *

<1920-24G>_cmdline-mode on
All commands can be displayed and executed. Continue? [Y/N]y
Please input password: Jinhua1920unauthorized
Warning: Now you enter an all-command mode for developer's testing, some commands may affect operation by wrong use, please carefully use it with our engineer's direction.
System View: return to User View with Ctrl+Z.

After switching to the system-view, we can change the default behaviour for each VLAN interface. I have multiple VLAN interfaces and each VLAN interface has an IPv4 and an unique local address (ULA) IPv6 address.

[1920-24G]interface Vlan-interface 3
[1920-24G-Vlan-interface3]display this
interface Vlan-interface3
 ipv6 address FDDA:28AD:487:3:FFFF:FFFF:FFFF:FFFE/64
 ip address
 dhcp select relay
 dhcp relay server-select 0

[1920-24G-Vlan-interface3]undo ipv6 nd ra halt
[1920-24G-Vlan-interface3]ipv6 nd ra prefix FDDA:28AD:487:3::/64 86400 3600
[1920-24G-Vlan-interface3]display this
interface Vlan-interface3
 ipv6 nd ra prefix FDDA:28AD:487:3::/64 86400 3600
 undo ipv6 nd ra halt
 ipv6 address FDDA:28AD:487:3:FFFF:FFFF:FFFF:FFFE/64
 ip address
 dhcp select relay
 dhcp relay server-select 0

The first command enables router advertisements. The second command adds the prefix which should be announced. That’s it. Don’t forget to save the changed configuration with “save force”. If you have more than one VLAN interface, enter this command in each VLAN interface context you wish to change.

HPE Data Protector 9.08 is available

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

3 days ago, on 13th October 2016, HPE has released patch bundle 9,08 for Data Protector 9. A patch bundle isn’t a directly installable version, instead it’s a bundle of patches and enhancements for a specific version of Data Protector, in this case Data Protector 9.

Beside fixes for discovered problems, a patch bundle includes also enhancements. There are some enhancements in this patch bundle, that have caught my attention particularly.

QCCR2A64053: Support for object copy of file system data to Microsoft Azure. Data Protector now supports the creation of a special backup device, which can be used together with Data Protector object copies, to copy Data Protector file system backups to Azure Backup Vaults. This is an easy way to create copies of important data on Microsoft Azure.

Contemporaneous with the announcement of Data Protector 9.08, I got an e-mail of HPE with the information, that one of my change request has made it into the latest patch bundle:

QCCR2A68100: VMWARE GRE stays in debug mode. I have observed this behaviour in different Data Protector installations: If debugging isn’t explicitly disabled (OB2DBG=0 in the omnirc), the VMware GRE always writes debug logs. Regardless if debugging is enabled or disabled in the GRE configuration.

Because of some security related changes and fixes in Data Protector 9.08, HPE has marked this patch bundle as critical.

Download Data Protector patch bundle 9.08:

Data Protector 9.08 for Windows

Data Protector 9.08 for HP-UX/IA

Data Protector 9.08 for Linux/64

HPE 3PAR OS updates that fix VMware VAAI ATS Heartbeat issue

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Customers that use HPE 3PAR StoreServs with 3PAR OS 3.2.1 or 3.2.2 and VMware ESXi 5.5 U2 or later, might notice one or more of the following symptoms:

  • hosts lose connectivity to a VMFS5 datastore
  • hosts disconnect from the vCenter
  • VMs hang during I/O operations
  • you see the messages like these in the vobd.log or vCenter Events tab
Lost access to volume <uuid><volume name> due to connectivity issues. Recovery attempt is in progress and the outcome will be reported shortly
  • you see the following messages in the vmkernel.log
ATS Miscompare detected beween test and set HB images at offset XXX on vol YYY

2015-11-20T22:12:47.194Z cpu13:33467)ScsiDeviceIO: 2645: Cmd(0x439dd0d7c400) 0x89, CmdSN 0x2f3dd6 from world 3937473 to dev &#34;naa.50002ac0049412fa&#34; failed H:0x0 D:0x2 P:0x0 Valid sense data: 0xe 0x1d 0x0.

Interestingly, not only HPE is affected by this. Multiple vendors have the same issue. VMware described this issue in KB2113956. HPE has published a customer advisory about this.


If you have trouble and you can update, you can use this workaround. Disable ATS heartbeat for VMFS5 datastores. VMFS3 datastores are not affected by this issue. To disable ATS heartbeat, you can use this PowerCLI one-liner:

Get-AdvancedSetting -Entity hostname -Name VMFS3.UseATSForHBOnVMFS5 | Set-AdvancedSetting -Value 0 -Confirm:$false


But there is also a solution. Most vendors have published firwmare updates for their products. HPE has released

  • 3PAR OS 3.2.2 MU3
  • 3PAR OS 3.2.2 EMU2 P33, and
  • 3PAR OS 3.2.1 EMU3 P45

All three releases of 3PAR OS include enhancements to improve ATS heartbeat. Because 3PAR OS 3.2.2 has also some nice enhancements for Adaptive Optimization, I recommend to update to 3PAR OS 3.2.2.

Data Protector: Copy sessions to encrypted devices fail after update to 9.07

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Recently, a customer has informed me, that copy sessions to encrypted devices failed, after he has made an update to Data Protector 9.07. The copy sessions failed with this error:

|Critical| From: [email protected]<hostname> "" Time: <Date><Time>
|90:6111| Error retrieving encryption key.

The customer uses tape encryption. The destination for the backups is a HPE StoreOnce, and a post-backup copy creates a copy of the data on tape. Backup to disk was running fine, but the copy to tape failed immediately.

The customer has opened a ticket at the HPE support and got instantly a hotfix to resolve this issue. HPE has documented this error in QCCR2A69192. If you run into the same issue, please request hotfix QCCR2A69802. This hotfix consolidates QCCR2A69192 and QCCR2A69318 (The BMA ends abnormally during backup/copy to tape).

Thanks to Stefan for the hint!

HPE StoreVirtual – Managers and Quorum

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

HPE StoreVirtual is a scale-out storage platform, that is designed to meet the needs of virtualized environments. It’s based on LeftHand OS and because the magic is a piece of software, HPE StoreVirtual is available as HPE ProLiant/ BladeSystem-based hardware, or as Virtual Storage Appliance (VSA) for VMware ESXi, Microsoft Hyper-V and KVM. It comes with an all-inclusive enterprise feature set. This feature set provides

  • Storage clustering
  • Network RAID
  • Thin Provisioning (with support for space reclamation)
  • Snapshots
  • Asynchronous and synchronous replication across multiple sites
  • Automated software upgrades and self-healing storage
  • Adaptive Optimization (Tiering)

The license is alway all-inclusive. There is no need to license individual features.

HPE StoreVirtual is not a new product. Hewlett-Packard has acquired LeftHand Networks in 2008. The product had several names since 2008 (HP LeftHand, HP P4000 and since a couple of years it’s StoreVirtual), but the core intelligence, LeftHand OS, was constantly developed by HPE. There are rumours that HPE StoreOnce Recovery Manager Central will be available for StoreVirtual soon.

Management Groups & Clusters

A management group is a collection of multiple (at least one) StoreVirtual P4000 storage systems or StoreVirtual VSA. A management group represents the highest administrative domain. Administrative users, NTP and e-mail notification settings are configured on management group level. Clusters are created per management group. A management group can consist of multiple clusters. A cluster represents a pool of storage from which volumes are created. A volume spans all nodes of a cluster. Depending on the Network RAID level, multiple copies of data are distributed over the storage systems in a cluster. Capacity and IO are expanded by adding more storage systems to a cluster.

As in each cluster, there are aids to ensure the function of the cluster in case of node failes. This is where managers and quorums comes into play.

Managers & Quorums

HPE StoreVirtual is a scale-out storage platform. Multiple storage systems form a cluster. As in each cluster, availability must be maintained if one or more cluster nodes fail. To maintain availability, a majority of managers must be running and be able to communicate with each other. This majority is called “a quorum”. This is nothing new. Windows Failover Clusters can also use a majority of nodes to gain a quorum. The same applies to OpenVMS clusters.

A manager is a service running on a storage system. This service is running on multiple storage systems within a cluster, and therefore in a management group. A manager has several functions:

  • Monitor the data replication and the health of the storage systems
  • Resynchronize data after a storage system failure
  • Manage and monitor communication between storage systems in the cluster
  • Coordinate configuration changes (one storage system is the coordinating manager)

This manager is called a “regular manager”. Regular managers are running on storage systems. The number of managers are counted per management group. You can have up to 5 managers per management group. Even if you have multiple storage systems and clusters per management group, you can’t have more than 5 managers running on storage systems. Sounds like a problem, but it’s not. If you have three 3-node clusters in a single management group, you can start managers on 5 of the 6 storage systems. Even if two storage systems fail, the remaining three managers gain a quorum. But if the quorum is lost, all clusters in a management group will be unavailable.

I have two StoreVirtual VSA running in my lab. As you can see, the management group contains two regular managers and vsa1 is the coordinating manager.


Patrick Terlisten/ Creative Commons CC0

There are also specialized manager. There are three types of specialized managers:

  • Failover Manager (FOM)
  • Quorum Witness (NFS)
  • Virtual Manager

A FOM is a special version of LeftHand OS and its primary function is to act as a tie breaker in split-brain scenarios. it’s added to a management group. It is mainly used if an even number of storage systems is used in a cluster, or in case of multi-site deployments.

The Quorum Witness was added with LeftHand OS 12.5. The Quorum Witness can only be used in 2-node cluster configurations. It’s added to the management group and it uses a file on a NFS share to provide high availability. Like the FOM, the Quorum Witness is used as the tie breaker in the event of a failure.

The Virtual Manager is the third specialized managers. It can be added to a management group, but its not active until it is needed to regain quorum. It can be used to regain quorum and maintain access to data in a disaster recovery situation. But you have to start it manually. And you can’t add it, if the quorum is lost!

As you can see in this screenshot, I use the Quorum Witness in my tiny 2-node cluster.


Patrick Terlisten/ Creative Commons CC0

Regardless of the number of storage systems in a management group, you should use an odd number of managers. An odd number of managers ensures, that a majority is easily maintained. In case of a even number of manager, you should add a FOM. I don’t recommend to add a Virtual Manager.

# of storage systems# of Manager
11 regular manager
22 regular manager + 1 specialized manager
33 regular manager or 2 + 1 FOM or Virtual Manager
43 regular manager or 4 + 1 FOM or Virtual Manager
> 55 regular manager or 4 + 1 FOM or Virtual Manager

In case of a multi-site deployment, I really recommend to place a FOM at a third site. I know that this isn’t always possible. If you can’t deploy it to a third site, place it at the “primary site”. A multi-site deployment is characterized by the fact, that the storage systems of a cluster are located in different locations. But it’s still a single cluster! This might lead to the situation, where a site failure causes the quorum gets lost. Think about a 4-node cluster with two nodes per site. In this case, the remaining two nodes wouldn’t gain quorum (split-brain situation). In this case, a FOM at a third site would help to gain quorum in case of a site failure. If you have multiple clusters in a management group, balance the managers across the clusters. I recommend to add a FOM. If you have a clusters at multiple sites, (primary and a DR site with remote copy), ensure that the majority of managers are at the primary site.

Final words

It is important to understand how managers, quorum, management groups and clusters are linked. Network RAID protects the data by storing multiple copies of data across storage systems in a cluster. Depending on the chosen Network RAID level, you can lose disks or even multiple storage systems. But never forget to have a sufficient number of managers (regular and specialized). If the quorum can’t be maintained, the access to the data will be unavailable. It’s not sufficient to focus on data protection. The availability of, or more specifically, the access to the data is at least as important. If you follow the guidelines, you will get a rock-solid, high performance scale-out storage.

I recommend to listen to Calvin Zitos podcast (7 Years of 100% uptime with StoreVirtual VSA) and to read Bart Heungens blog post about his experience with HPE StoreVirtual VSA (100% uptime for 7 years with StoreVirtual VSA? Check!).

HPE StoreVirtual REST API

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Representational State Transfer (REST) APIs are all the rage. REST was defined by Roy Thomas Fielding in his PhD dissertation “Architectural Styles and the Design of Network-based Software Architectures“. The architectural style of REST describes six constraints:

  • Uniform interface
  • Stateless
  • Cacheable
  • Client – Server communication
  • Layered system
  • Code on demand

RESTful APIs typically use HTTP and HTTP verbs (GET, POST, PUT, DELETE, etc.) to send data to, or retrieve data from remote systems. To do so, REST APIs use Uniform Resource Identifiers (URIs) to interact with remote systems. Thus, a client can interact with a remote system over a REST API using standard HTTP URIs and HTTP verbs. For the data transfer, common internet media types, like JSON or XML are used. It’s important to understand that REST is not a standard per se. But most implementations make use of standards such as HTTP, URI, JSON or XML.

Because of the uniform interface, you have different choices in view of a client. I will use PowerShell and the Invoke-RestMethod cmdlet in my examples.

HPE StoreVirtual REST API

With the release of LeftHand OS 11.5 (the latest release is 12.6), HPE added a REST API for management and storage provisioning. Due to a re-engineered management stack, the REST API is significantly faster than the same task processed on the CLI or using the  Centralized Management Console (CMC). It’s perfect for automation and scripting. It allows customers to achieve a higher level of automation and operational simplicity. The StoreVirtual REST API is using JavaScript Object Notation (JSON) for data transfer between client and the StoreVirtual management group. With the REST API, you can

  • Read, create, and modify volumes
  • Create and delete snapshots
  • Create, modify, and delete servers
  • Grant and revoke access of servers to volumes

I use two StoreVirtal VSA (LeftHand OS 12.6) in my lab. Everything I show in this blog post is based on LeftHand OS 12.6.

The REST API in LeftHand OS 12.6 uses:

  • HTTPS 1.1
  • media types application/JSON
  • Internet media types application/schema+JSON
  • UTF-8 character encoding

RESTful APIs typically use HTTP and HTTP verbs (GET, POST, PUT, DELETE, etc.). I case of the StoreVirtual REST API:

  • GET is used to retrieve an object. No body is necessary.
  • PUT is used to update an object. The information to update the object is sent within the body.
  • POST is used to create of an object, or to invoke an action or event. The necessary information are sent within the body.
  • DELETE is used to delete an object.

Entry point for all REST API calls is /lhos, starting from a node, eg.


Subsequent resources are relative to this base URI. Resources are:

Resource pathDescription
/lhos/managementGroupManagement group entity
/lhos/clustersCluster collection
/lhos/cluster/<id>Cluster entity
/lhos/credentialsCredentials collection
/lhos/credentials/<session token>Credentials entity
/lhos/serversServer collection
/lhos/servers/<id>Server entity
/lhos/snapshotsSnapshot collection
/lhos/snapshots/<id>Snapshot entity
/lhos/volumesVolume collection
/lhos/volumes/<id> Volume entity

The object model of the StoreVirtual REST API uses

  • Collections, and
  • Entities

to address resources. An entity is used to address individual resources, whereas a collection is a group of individual resources. Resources can be addressed by using a URI.

Exploring the API

First of all, we need to authenticate us. Without a valid authentication token, no REST API queries can be made. To create a credential entity, we have to use the POST method.

$cred = @{

$body = $cred | ConvertTo-Json

$a = Invoke-RestMethod -Uri https://vsa1.lab.local:8081/lhos/credentials -Method Post -Body $body -ContentType 'application/JSON'

$cred is a hash table which includes the username and the password. This hash table is converted to the JSON format with the ConvertTo-Json cmdlet. The JSON data will be used as body for our query. The result is an authentication token.

PS C:\Users\p.terlisten> $a


This authentication token must be used for all subsequent API queries. This query retrieves a collection of all valid sessions.

$b = Invoke-RestMethod -Uri https://vsa1.lab.local:8081/lhos/credentials -Method Get -Headers @{'Authorization'=$a.authToken}

The GET method is used, and the authentication token is sent with the header of the request.

PS C:\Users\p.terlisten> $b

name        : REST Sessions Collection
description : Collection of authentication sessions used by the REST server.
type        : RESTSession
uri         : /lhos/credentials
total       : 1
members     : {@{name=fa0a7b56-0134-400f-9d62-79b3071c950a; description=REST Session; type=RESTSession; id=0; uri=/lhos/credentials/fa0a7b56-0134-400f-9d62-79b3071c950a; 
              created=2016-06-07T08:38:06.426241Z; modified=2016-06-07T08:44:28.255283Z; userName=admin; clientIP=}}

To retrieve an individual credential entity, the URI of the entity must be used.

$b = Invoke-RestMethod -Uri https://vsa1.lab.local:8081/lhos/credentials/fa0a7b56-0134-400f-9d62-79b3071c950a -Method Get -Headers @{'Authorization'=$a.authToken}

The result of this query is the individual credential entity

PS C:\Users\p.terlisten> $b

name        : fa0a7b56-0134-400f-9d62-79b3071c950a
description : REST Session
type        : RESTSession
id          : 0
uri         : /lhos/credentials/fa0a7b56-0134-400f-9d62-79b3071c950a
created     : 2016-06-07T08:38:06.426241Z
modified    : 2016-06-07T08:51:56.358096Z
userName    : admin
clientIP    :

It’s important to know, that if a session has not been used for 15 minutes, it is automatically removed. The same applies to constantly active sessions after 24 hours. After 24 hours, the credential entity will be automatically removed.

Let’s try to create a volume. The information about this new volume has to be sent within the body of our request. We use again the ConvertTo-Json cmdlet to convert a hash table with the necessary information to the JSON format.

$vol = @{
            description='Volume created via REST API';

$body = $vol | ConvertTo-Json

Invoke-RestMethod -Uri https://vsa1.lab.local:8081/lhos/volumes -Method Post -Headers @{'Authorization'=$a.authToken} -Body $body -ContentType 'application/JSON'

The size must be specified in bytes. As a result, Invoke-RestMethod will output this:

name                          : api-vol
description                   : Volume created via REST API
type                          : volume
id                            : 3058
uri                           : /lhos/volumes/3058
created                       : 2016-06-07T08:55:41Z
modified                      : 
friendlyName                  : 
transport                     : 0
isThinProvisioned             : True
size                          : 1073741824
serialNumber                  : f9df3e8bb0a160f269027ecc0371884e0000000000000bf2
provisionedSpace              : 1073741824
numberOfReplicas              : 2
dataProtectionLevel           : 2
iscsiIqn                      :
isPrimary                     : True
isDeleting                    : False
bytesWritten                  : 0
isAvailable                   : True
clusterName                   : CLUSTER
clusterId                     : 28
isVIPRebalancing              : False
isAdaptiveOptimizationEnabled : True
isMigrating                   : False
scsiLUNStatus                 : available
hasUnrecoverableIOErrors      : False
restripePendingStatus         : none
replicationStatus             : normal
resynchronizationStatus       : none
migrationStatus               : none
isLicensed                    : True
transportServerId             : 0
fcTransportStatus             : 0
createdBy                     : Unknown
iscsiSessions                 : 
fibreChannelPaths             : 
snapshots                     : @{name=snapshots; type=snapshot; uri=/snapshots?volumeName=api-vol; resource=}

Using the CMC, we can confirm that the volume was successfully created.


Patrick Terlisten/ Creative Commons CC0

Since we have a volume, we can create a snapshot. To create a snapshot, we need to invoke an action on the volume entity. We have to use the POST method and the URI of our newly created volume.

$snapshot = @{
                parameters = @{
                                description='Volume snapshot created via REST API';

$body = $snapshot | ConvertTo-Json

Invoke-RestMethod -Uri https://vsa1.lab.local:8081/lhos/volumes/3058 -Method Post -Headers @{'Authorization'=$a.authToken} -Body $body -ContentType 'application/JSON'

In case of a successful query, Invoke-RestMethod will give us this output.

name                     : vol-api-snap
description              : Volume snapshot created via REST API
type                     : snapshot
id                       : 3060
uri                      : /lhos/snapshots/3060
created                  : 2016-06-07T09:01:12Z
modified                 : 
friendlyName             : 
transport                : 0
isThinProvisioned        : True
size                     : 1073741824
serialNumber             : f9df3e8bb0a160f269027ecc0371884e0000000000000bf4
provisionedSpace         : 8388608
iscsiIqn                 :
isPrimary                : True
isDeleting               : False
bytesWritten             : 0
isAvailable              : True
clusterName              : CLUSTER
clusterId                : 28
iscsiSessions            : 
fibreChannelPaths        : 
snapshotACL              : 
writableSpaceUsed        : 0
managedBy                : 0
isAutomatic              : False
isMigrating              : False
scsiLUNStatus            : available
hasUnrecoverableIOErrors : False
restripePendingStatus    : none
replicationStatus        : normal
resynchronizationStatus  : none
migrationStatus          : none
isLicensed               : True
transportServerId        : 0
fcTransportStatus        : 0
createdBy                : Unknown

Again, we can use the CMC to confirm the success of our operation.


Patrick Terlisten/ Creative Commons CC0

To delete the snapshot, the DELETE method and the URI of the snapshot entity must be used.

Invoke-RestMethod -Uri https://vsa1.lab.local:8081/lhos/snapshots/3060 -Method Delete -Headers @{'Authorization'=$a.authToken}

To confirm the successful deletion of the snapshot, the GET method can be used. The GET method will retrieve a collection of all snapshot entities.

Invoke-RestMethod -Uri https://vsa1.lab.local:8081/lhos/snapshots -Method Get -Headers @{'Authorization'=$a.authToken}

The result will show no members inside of the snapshot collection.

name        : Snapshots Collection
description : Collection of Snapshot objects
type        : snapshot
uri         : /lhos/snapshots
total       : 0
members     : {}

At the end of the day, we remove our credential entity, because it’s not longer used. To delete the credential entity, we use the DELETE method with the URI of our credential entity.

Invoke-RestMethod -Uri https://vsa1.lab.local:8081/lhos/credentials/fa0a7b56-0134-400f-9d62-79b3071c950a -Method Delete -Headers @{'Authorization'=$a.authToken}

The next query should fail, because the credential entity is no longer valid.

PS C:\Users\p.terlisten> Invoke-RestMethod -Uri https://vsa1.lab.local:8081/lhos/credentials -Method Get -Headers @{'Authorization'=$a.authToken}
The remote server returned an error: (401) Unauthorized. (raised by: Invoke-RestMethod)

HTTPS workaround

The StoreVirtual API is only accessable over HTTPS. By default, the StoreVirtual nodes use an untrusted HTTPS certifificate. This will cause Invoke-RestMethod to fail.

[10,6: Invoke-RestMethod] The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.

After a little research, I found a workaround. This workaround uses the System.Security.Cryptography.X509Certificates namespace. You can use this snippet to build a function or add it to a try-catch block.

add-type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCertsPolicy : ICertificatePolicy {
        public bool CheckValidationResult(
            ServicePoint srvPoint, X509Certificate certificate,
            WebRequest request, int certificateProblem) {
            return true;

[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy

Final words

The StoreVirtual REST API is really handy. It can be used to perform all important tasks. It’s perfect for automation and it’s faster than the CLI. I’ve used PowerShell in my examples, but I’ve successfully tested it with Python. Make sure to take a look in to the HPE StoreVirtual REST API Reference Guide.