Tag Archives: ha

VMware vSphere Metro Storage Cluster with HP 3PAR Peer Persistence – Part II

This posting is ~9 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

The first part of this (short) blog series covered the basics of VMware vSphere Metro Storage Cluster (vMSC) with HP 3PAR Peer Persistence. This, the second, part will cover the basic tasks to configure Peer Persistence. Please note that this blog post relies on the features and supported configurations of 3PAR OS 3.1.3! This is essential to know, because 3.1.3 got some important enhancements in respect of 3PAR Remote Copy.

Fibre-Channel zoning

On of the very first tasks is to create zones with between the Remote Copy Fibre Channel (RCFC) ports. I used two ports from a quad-port FC Adapter for Remote Copy. This matrix shows the zone members in each Fibre Channel fabric. 3PAR OS 3.1.3 supports up to four RCFC ports per node. Earlier versions of 3PAR OS only support one RCFC port per node.

N:S:P	0:2:1	0:2:2	1:2:1	1:2:2
0:2:1	Fabric 1
0:2:2		Fabric 2
1:2:1			Fabric 1
1:2:2				Fabric 2

RCFC port setup

After the zoning it’s time to setup the RCFC ports. In this case the RCFC ports will detect the partnering port by itself. I assume that the ports are unconfigured. Otherwise it’s necessary to take the ports offline. The command controlport is used to configure a port with a specific port role.

controlport config rcfc -ct point -f 0:2:1
controlport config rcfc -ct point -f 0:2:2
controlport config rcfc -ct point -f 1:2:1
controlport config rcfc -ct point -f 1:2:2

You can do the same with the 3PAR Management Console. After the RCFC port configuration the success of this procedure can checked with doing this on both StoreServs, you can check your success with showrctransport

showrctransport -rcfc

or with the 3PAR Management Console.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Remote Copy setup

Now it’s time to create the Remote Copy configuration. The screenshots below are schowing the configuration of a bidirectional 1-to-1 Remote Copy setup. Start the wizard and select the configuration.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

In the next step, the RCFC ports have to be configured and paired together. Simply connect the ports by selecting a port and pull a connection to the other port. Both ports have to be in the same zone.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

A Remote Copy Group groups Virtual Volumes (VV) together to ensure I/O consistency. To create a bidirectional Remote Copy configuration we need two Remote Copy Groups. One from A > B and a second from B > A. I recommend to enable the “Auto Recover” option. This option is only visible, if the “Show advanced options” tickbox is enabled.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

This screenshot shows the bidirectional Remote Copy configuration. Each StoreServ acts as primary arry for a Remove Copy Group and as secondary array for a primary Remote Copy Group on the other StoreServ.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

If you already created volumes, you can add the volumes in this step. I will show this step later.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The last page shows a summary of the configured options. Simply click “Finish” and proceed with the next step.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

After creating the volumes it’s necessary to add them to the Remote Copy groups. Right click the Remote Copy Group and select “Edit Remote Copy Group…”.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click “Next”.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Select the volumes to add and check the box “Create new volume”. I recommend to use CPGs with the same characteristics as on the source system. I also recommend to use the same CPG as User and Copy CPG. Click “Add” and repeat this step for each volume that should belong to the Remote Copy Group. At the end click “Next”…

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

… and “Finish”.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Repeat the steps for the second Remote Copy Group and the volumes on the secondary StoreServ.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

This screenshot shows the result of the configuration process.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

A very handy feature of 3PAR OS 3.1.3 is that it creates a Virtual Volume Set for each Remote Copy Group. When a VV is added to the Remote Copy Group, it belongs automatically to the Virtual Volume Set and will be exported to the hosts. This screenshots shows the Virtual Volume Sets on both StoreServs.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Please ensure that both Virtual Volume Sets on both StoreServs are exported to all hosts (I recommend using Host Sets). If everything has been correctly presented 8 paths should be visible for each VMFS datastore: 4 active paths to the primary, and 4 standby paths to the secondary StoreServ.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Automate the failover

There are two requirements to automate the failover:

Quorum Witness
Enabled “Auto Failover” for Remote Copy Groups

The Quorum Witness is a VMware Appliance that needs to be deployed at a third site. The setup is really easy. Simply deploy the OVA and power it on. A short menu guides you through some setup tasks, like setting a password, assigning an IP address etc. When the Quorum Witness is available on the network, create a Peer Persistence configuration. Enter the IP address and select the targets, for which the Quorum Witness should act as a witness.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

If everything went fine, the “Quorum Status” should be “Started”.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Now the automatic failover for the Remote Copy Groups can be enabled.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Select the groups and click the right arrow to enable automatic failover for the selected Remote Copy Groups.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

That’s it! To test the failover you can use the 3PAR Management Console or this CLI command:

setrcopygroup switchover -t 7200-EDV1

With this command all secondary Remote Copy Groups on StoreServ 7200-EDV2 will become primary Remote Copy groups. If everything is configured accordingly, you will notice no or only a short IO interruption during the failover. An automatic failover will only occur, if a StoreServ loses all RCFC links AND the connection to the Quorum Witness. Otherwise there will be no automatic failover! The parameter “switchover” is only used for transparent and controlled failovers. It’s issued on the primary storage array. The parameter “failover” is automatically issued from the secondary storage system in case of a failover situation.

Finaly words

The basics tasks are:

create zones for the RCFC ports
configure the RCFC ports on each node
create a bidirectional 1-to-1 Remote Copy setup with Remote Copy Groups on each StoreServ
add volumes to the Remote Copy Groups
present Virtual Volume Sets (that were automatically created based on the Remote Copy Groups) to the hosts
deploy Quorum WItness
create a Peer Persistence configuration and configure Quorum Witness for the StoreServs that belong to the Peer Persistence Configuration
Enable “Automatic Failover” for the presented Remote Copy Groups

This is only a very rough overview about the configuration of a 3PAR Peer Persistence setup. I strongly recommend to put some brain into the design and the planning of the Peer Persistence setup.

VMware vSphere Metro Storage Cluster with HP 3PAR Peer Persistence – Part I

This posting is ~9 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

The title of this blog post mentions two terms that have to be explained. First, a VMware vSphere Metro Storage Cluster (or VMware vMSC) is a configuration of a VMware vSphere cluster, that is based on a a stretched storage cluster. Secondly, HP 3PAR Peer Persistence adds functionalities to HP 3PAR Remote Copy software and HP 3PAR OS, that two 3PAR storage systems form a nearly continuous storage system. HP 3PAR Peer Persistence allows you, to create a VMware vMSC configuration and to achieve a new quality of availability and reliability.

VMware vSphere Metro Storage Cluster

In a vMSC, server and storage are geographically distributed over short or medium-long distances. vMSC goes far beyond the well-known synchronous mirror between two storage systems. Virtualization hosts and storage belong to the same cluster, but they are geographically dispersed: They are stretched between two sites. This setup allows you to move virtual machines from one site to another (vMotion and Storage vMotion) without downtime (downtime avoidance). With a stretched cluster, technologies such as VMware HA can help to minimize the time of a service outage in case of a disaster (disaster avoidance).

The requirements for a vMSC are:

Storage connectivity using Fibre Channel/ Fibre Channel over Ethernet (FCoE), NFS or iSCSI
max. 10 ms round-trip time (RTT) for the ESXi management network (> 10 ms is supported with vSphere Enterprise Plus – Metro vMotion)
max. 5 ms round-trip time RTT for the synchronous storage replication links
at leat 250 Mbps per concurrent vMotion on the vMotion network

The complexity of the storage requirements is not the maximum round-trip time – It’s the requirement that a datastore must be accessible from both sites. This means, that a host in Site A must be able to access /read & write) a datastore on a storage in Site B and vice versa. vMSC knows to different methods of host access configuration:

Uniform host access configuration
Non-Uniform host access configuration

With a uniform host access configuration, the storage on both sides can accessed by all hosts. LUNs from both storage systems are zoned to all hosts and the Fibre-Channel fabric is stretched across the site-links. The following figure was taken from the “Implementing vSphere Metro Storage Cluster using HP 3PAR Peer Persistence” technical whitepaper and shows a typical uniform host access configuration.

HPE/ hpe.com

The second possible configuration is non-uniform host access configuration, in which the hosts only access the site-local storage system. The Fibre Channel fabrics are not stretched across the inter-site links. The following figure was taken from the “Implementing vSphere Metro Storage Cluster using HP 3PAR Peer Persistence” technical whitepaper and shows a typical non-uniform host access configuration. If a storage system fails, the ESXi in a datacenter will lose the connectivity and the virtual machine will fail. VMware HA will take care that the VM is restarted in other datacenter.

HPE/ hpe.com

Another possible non-uniform setup uses stretched Fibre Channel fabrics and some kind of virtual LUN. A LUN is mirrored between two storage systems and can be accessed from both sites. The storage systems take care of the consistency of the data. This figure was taken from the “VMware vSphere Metro Storage Cluster Case Study” technical whitepaper.

non-uniform-host-access-stretched-fabric

VMware/ vmware.com

The uniform host access configuration is currently used most frequently.

Regardless of the implementation it’s useful to think about the data locality. Let’s assume, that a host in datacenter A is running a VM, that is housed in a datastore on a storage system in datacenter B. As long as you’re using a stretched fabric between the sites, this is a potential scenario. What happens to the storage I/O of this VM? Right, it will travel across the inter-site links from datacenter A to datacenter B. To avoid this, you can use DRS groups and rules.

Examples for uniform host access configuration are:

Uniform host access configuration	Non-Uniform host access configuration
vSphere 5.x support with NetApp MetroCluster (2031038)	Implementing vSphere Metro Storage Cluster (vMSC) using EMC VPLEX (2007545)
Implementing vSphere Metro Storage Cluster (vMSC) using EMC VPLEX (2007545)
Implementing vSphere Metro Storage Cluster using HP 3PAR StoreServ Peer Persistence (2055904)
Implementing vSphere Metro Storage Cluster using HP LeftHand Multi-Site (2020097)
Implementing vSphere Metro Storage Cluster using Hitachi Storage Cluster for VMware vSphere (2073278)
Implementing vSphere Metro Storage Cluster using IBM System Storage SAN Volume Controller (2032346)

HP 3PAR Remote Copy, Peer Persistence & the Quorum Witness

HP 3PAR Peer Persistence uses synchronous Remote Copy and Asymmetric Logical Unit Access (ALUA) to realize a metro cluster configuration that allows host access from both sides. 3PAR Virtual Volumes (VV) are synchronous mirrored between two 3PAR StoreServs in a Remote Copy 1-to-1 relationship. The relationship may be uni- or bidirectional, which allows the StoreServs to act mutually as a failover system. To create a vMX configuration with HP 3PAR StoreServ storage systems, some requirements have to be fulfilled.

Firmware on both StoreServ storage systems must be 3.1.2 MU2 or newer (I recommend 3.1.3)
a remote copy 1-to-1 synchronous relationship
2.6 ms or less round-trip time (RTT)
Quorum Witness VM must run at a 3rd site and must be reachable from each 3PAR StoreServ
same WWN and LUN ID for each source and target virtual volume
VMware ESXi 5.0, 5.1 or 5.5
Hosts must be created with Hostpersona 11
Hosts must be zoned to both 3PAR StoreServ storage systems (this requires a stretched Fibre Channel fabric between the sites)
iSCSI or FCoE for host connectivity is supported with 3PAR OS 3.2.1. Versions below 3PAR OS 3.2.1 only support FC for host connectivity with Peer Persistence
Both 3PAR StoreServ storage systems must be licensed for Remote Copy and Peer Persistence (I recommend to license the Replication Suite)

A VV can be a source or a target volume. Source VV belong to the primary remote copy group, target virtual volumes belong to a secondary remote copy groups. VV are grouped to remote copy groups to ensure I/O consistency. So all VV that require write order consistency should belong to a remote copy group. Even VV that don’t need write order consistency should belong to a remote copy group, just to simplify administration tasks. A typical uniform vMSC configuration with 3PAR StoreServs will have remote copy groups replicating in both directions. So both StoreServs act as source and target in a bi-directional synchronous remote copy relationship. It’s important to understand that the source and target volumes share the same WWN and they are presented using the same LUN ID. The ESXi hosts must use Hostpersona 11. During the process of creating the remote copy groups, the target volumes can be created automatically. This ensures that the source and target volumes use the same WWN. When the volumes from the source and target StoreServ are presented, the paths to the target StoreServ are marked as “Stand by”. In case of a failover the paths will become active and the I/O will continue. The Quorum Witness is a RHEL appliance that communicates with the StoreServs and triggers the failover in some specific scenarios. This table was taken from the “Implementing vSphere Metro Storage Cluster using HP 3PAR Peer Persistence” technical whitepaper. As you can see, the automatic failover is only triggered in one specific scenario.

	Replication stopped	Automatic failover	Host I/O impacted
Array to Array remote copy links failure	Y	N	N
Single site to Quorum Witness network failure	N	N	N
Single site to Quorum Witness network and Array to Array remote copy link failure	Y	Y	N
Both sites to Quorum Witness network failure	N	N	N
Both sites to Quorum Witness network and Array to Array remote copy link failure	Y	N	Y

Summary

VMware vSphere Metro Storage Cluster (vMSC) is a special configuration of a stretched compute and storage cluster. A vMSC is usually implemented to avoid downtime. A vMSC configuration makes it possible to move virtual machine, and thus workloads, between sites. Beyond this, vMSC can avoid downtime caused due to a failed storage system. Using HP 3PAR Remote Copy, 3PAR Peer Persistence and the Quorum Witness, two HP 3PAR StoreServ storage systems can form a uniform vMSC configuration. This allows movement of VMs/ workloads between sites and also a transparent failover between storage systems in case of a failure of one of the StoreServs.

Part II of this small series will cover the configuration of Remote Copy and Peer Persistence.

Windows Server 2012 Cluster with VMware vSphere 5.1/ 5.5

This posting is ~10 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

While I was poking around in my Twitter timeline, a tweet from Victor van den Berg (VCDX #121) got my attention.

@Mandivs I'm suprised W2K12R2 MSCS doesn't support in guest iSCSI anymore…You have to use vSphere 5.5 RDM's
— Viktor van den Berg (@viktoriousss) February 11, 2014

My first though “What a step backwards!”. I have installed a bunch of Microsoft clusters in Virtual Infrastructure and vSphere enviroments and most times it was PITA. Especially with Raw Device Mappings (RDM) and bus sharing, which prevents vMotion a VM to another host (regardless of this: it’s not supported!). It’s ironic to invest a significant amount of money into a technology, which increases availability and manageability, and another technology lowers availability due additional maintenance windows for cluster failovers. But that’s exactly what you get, when you use MSCS with SCSI bus sharing (RDM or VMFS). A way to address this issue is to use in-guest iSCSI. This means that you access the shared disks directly from the VM due a iSCSI initiator running in the VM. To do so, you have to present the disks for the cluster to the VMs, not to the ESXi hosts. To be honest: In-guest increases complexity. Especially then, when the customer doesn’t have a iSCSI infrastructure. A second method is in-guest SMB, which is currently only supported with Windows Server 2012. Just to clear up the matter with in-guest iSCSI and W2K12(R2):

@Mandivs @viktoriousss Ah, just found it in the KB article. In-guest iSCSI with W2K12 is only supported with vSphere 5.1 or earlier
— Patrick Terlisten (@PTerlisten) February 11, 2014

Mostafa Khalil /VCDX #002) provided the crucial information:

@viktoriousss @Mandivs @vPatrickT it is an old footnote prior to 5.5. It is still supported. I will get the KB updated.
— Mostafa Khalil (@MostafaVMW) February 11, 2014

In-guest iSCSI is supported with W2K12 on vSphere 5.5, which also supportes W2K12 R2 failover clustering. But there’s another interesting fact, that was new to me: Windows Server 2012 failover clustering isn’t supported with ESXi provided shared disks! I found a hint in VMware KB1037959.

Windows Server 2012 failover clustering is not supported with ESXi-provided shared storage (such as RDMs or virtual disks) in vSphere 5.1 and earlier. For more information, see the Miscellaneous Issues section of the vSphere 5.1 Release Notes. VMware vSphere 5.5 provides complete support for 2012 failover clustering.

This means, that you can run Windows Server 2012 failover cluster on vSphere 5.1, but only with in-guest iSCSI or in-guest SMB.

What’s new in vSphere 5.5?

Windows 2012 R2 failover clustering is now supported. But much more significant are changes regarding the storage protocols. With vSphere 5.5 RDMs (shared disks for quorum or data) can be on iSCSI and Fibre Channel over Ethernet (FCoE). Until vSphere 5.5 only Fibre-Channel (FC) was supported. iSCSI and FCoE are supported for cluster-in-a-box (CIB) and cluster across boxes (CAB). Just to make it clear: NFS isn’t supported. Neither for RDM nor for VMFS! Furthermore software and hardware initiator, and mixed setups are supported for iSCSI and FCoE. With vSphere 5.5 the VMW_PSP_RR can be used for the RDMs. There’s no need to change the PSP for the RDMs. VMware KB2052238 summarizes the changes in vSphere 5.5 together.

Final words

Clusters are never an easy thing. The complex support matrix does not make it easier. If you’re using Microsoft clusters, be sure to check the above mentioned knowledge base articles before you make a update of your vSphere enviroment or of your Microsoft cluster.

Enable VMware Fault Tolerance in nested enviroments

This posting is ~10 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

While playing around in my lab, I wanted to enable VMware Fault Tolerance (FT) for a VM. In the absence of physical HW I use a nested enviroment, which is running on a HP ProLiant DL160 G6 (2x Intel Xeon E5520, 32G RAM, a RAID 0 with 4 SATA drives). FT isn’t available in nested enviroments, because HW virtualization features are required. This screenshot was taken from the web client.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

But “isn’t available” doesn’t mean that you can’t enable it. ;) As always this isn’t supported by VMware. It’s for lab enviroments, trainings etc., but not for production. You have to set three configuration parameters for the VM that you want to use with FT. If you use the web client, you can set the configuration parameters as follows:

Edit Settings… > VM Options > Advanced > Edit Configuration…

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

For easy copy ‘n paste here the parameters:

replay.supported = "true"
replay.allowFT = "true"
replay.allowBTOnly = "true"

After setting the parameters you can enable FT and start the VM.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Please note that you can only run 32-bit guests! This is due to the binary translation (replay.allowBTOnly). A FT-secured 64-bit Windows 2008 R2 won’t be possible. But to show the configuration and use of FT a 32-bit Windows 2003 should be sufficient. :) I have configured FT in my lab and I took the screenshots from the vSphere 5.5 Web Client. I use VM HW 9 for the nested ESXi and VHV is enabled in the CPU section. If you are searching for more: virtuallyGhetto is an awesome source, especially for nested virtualization and everything around automation using various API/ SDK and CLI. Kudos to William Lam for the work he puts in there.