Tag Archives: datacore

Automating ESXi configuration for DataCore SANsymphony-V

DataCore describes in their Host Configuration Guide for VMware ESXi some settings that must be adjusted before storage from DataCore SANsymphony-V storage servers will be assigned to the ESXi hosts. Today, for ESXi 5.x and 6.0, you have to add a custom rule and adjust the advanced setting DiskMaxIOSize. For ESX(i) 4 more parameters had to be adjusted. But I will focus on  ESXi 5.x and 6.0. You need to adjust these settings for each host that should get storage mapped from a DataCore storage server. If you have more then one host, you may have the wish to automate the necessary steps. The check the current value of DiskMaxIOSize, you can use this lines of PowerCLI code.

So set DiskMaxIOSize to the recommended value of 512 and to create the custom SATP rule, use this lines:

Please make sure that you adjust the $esxCluster variable.  Nothing fancy, but it can save some time – even if you only have two hosts. ;)

Use Windows MPIO for DataCore backend storage connections

When you install DataCore SANsymphony-V (SSV), you will be asked during the setup to allow the installation of some special drivers. DataCore SANSymphony-V needs this drivers to act as a storage target for hosts and other storage servers. Usually you have three different port roles in a DataCore SSV setup:

  • Frontend Ports (FE)
  • Mirror Ports (MR)
  • Backend Ports (BE)

Frontend (FE) ports act only in target-only mode. These ports will be disabled, if you stop a DataCore storage server. Mirror (MR) ports (can) act as target AND initiator. You can set (if you like) a mirror port to a specific mode (target or initiator), but I wouldn’t recomment this. Theoretically you can set one MR port to act as initiator, and a second to target-only mode. If the port is set to target-only, the port is also stopped when the DataCore storage server is stopped. A backend (BE) port acts as initiator for backend storage. Usually the FE ports act as target-only, the MR as target/ initiator and the BE ports as initiator-only. If you use local storage (or SAS connected), there will be no BE ports.

When it comes to ALUA-capable backend storage, DataCore SSV is a bit old-school. But stop! What is ALUA? Asymmetric Logical Unit Access (ALUA), somtimes also known as Target Port Groups Support (TPGS), is a set of commands that allows to define priorized paths for SCSI devices. With ALUA, a dual-controller storage system is capable to “tell” the server which of the controllers is the “owning” controller for a specific volume. All paths are active, but only a subset of paths to a volume is active and optimized. This is often refered as “Active/ Optimized” and “Active/ Non-Optimized”. Using the non-optimized path is not a problem when it comes to write IO (due to the fact that IO must travel the mirror link between the controllers to the other controllers cache), but when it comes to read IO, using a non-optimized path is a mess. And at this point, DataCore SSV is a bit old-school. Even if the backend storage is ALUA-capable (all new storage systems should be ALUA-capable), DataCore SSV doesn’t care about the optimized and non-optimized paths. It simply chooses one paths and uses it. And this can cause performance problems. There are two solutions for this problem:

  1. You check the chosen backend paths and manually select one (!) active and optimized path.
  2. You replace the DataCore drivers for the backend ports and use Microsoft MPIO to handle the backend paths.

Solution 1 is okay if you only have a few backend disks. DataCore also uses the Microsoft Windows MPIO framework, so it’s available by default if you install DataCore SANsymphony-V. DataCore allows the usage of 3rd party MPIO software (like EMC PowerPath), but they will never support it. If you have trouble with you backend connection, you will be on your own.

Using a Third Party Failover product

Using a Third Party Failover product (such as EMC’s PowerPath or the many MPIO variants from Storage Vendor) can be used directly on the DataCore Server. In the case where Storage Arrays are attached by Fibre Channel connections, do not use the DataCore Fibre Channel back-end driver when using any Third Party Failover product – use the Third Party’s preferred Fibre Channel Driver instead.

This statement was taken from FAQ 1302 (Storage Hardware Guideline for use with DataCore Servers). Some things won’t work with the native backend drivers. You won’t be able to see the backend paths or monitor the performance using the DataCore SSV GUI. Backend storage that is handled by native drivers and 3rd party MPIO products will be treated as “local” storage. DataCore describes this also in FAQ 1302:

A Storage Array that is connected to a DataCore Server but without using the DataCore Fibre Channel back-end driver can still be used. This includes SAS or SATA-attached, all types of SSD, iSCSI connections and any Fibre Channel connection using the Vendor’s own driver.

Any storage that is connected in this way will appear to the SANsymphony-V software as if it were an ‘Internal’ or ‘direct-attached’ Storage Array and some SANsymphony-V functionality will be unavailable to the user such not being able to make use of SANsymphony-V’s performance tools to get some of the available performance counters related to Storage attached to DataCore Servers. Also some potentially useful logging information regarding connections between the DataCore Server and the Storage Array is lost (i.e. not being able to monitor any SCSI connection alerts or errors directly) and may hinder some kinds of troubleshooting, should any be needed.

You should replace the backend drivers AFTER the installation of SSV, but BEFORE you map storage to the newly installed storage server. To identify the correct FC-HBA or NIC, take a look at the PCI bus number. You can find this information on the info tab of the server port details in the DataCore SSV GUI. Then check the FC-HBA/ NIC in the Windows device manager for the same PCI bus number.

devicemgr_datacore_hba_driver_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Switch to the “Driver” tab and replace the DataCore driver with the native HBA driver.

devicemgr_datacore_hba_driver_install_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

devicemgr_datacore_hba_driver_install_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

devicemgr_datacore_hba_driver_install_3

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

After replacing the drivers you will notice that the backend ports are marked as missing in the DataCore SSV GUI. Simply remove them. You don’t need them anymore. Now you can map storage to the storage server. Make sure that you scan for new devices in the MPIO console or that you run

to claim new devices. You can check the path status using the device manager (check the properties of the disk device, then switch to the MPIO tab) or you can use this command

Somethimes the storage vendor offers monitoring tool (this is from Nexsan) that provide information about path state, mpio policy and statistics.

nexsan_msio_setup_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I can’t see much reasons to use the DataCore drivers and the native backend path handling from SSV. But you should be clear about the limitations when it somes to support!

First experience: Nexsan E-Series

One of my longtime DataCore customers has started a project to replace their current DataCore storage servers and backend storage with new hardware. In opposite of the current setup, the newly installed backend storage is now FC-attached. The customer has selected Nexsan E-Series E32V, E32XV and E48V storage systems in combination with DataCore SANsymphony-V10.

Who is Nexsan?

The question should be: Who is Imation? Nexsan was founded in 1999 in Derby, England, but was aquired by Imation in December 2012. Since December 2012, Nexsan is one of Imations brands and offers, as a storage-only company, three different product lines: Assureon Secure StorageE-Series High Density Storage and NST Hybrid Storage.

Assureon Secure Storage is amied to customers that needs to implement storage optimization, regulatory and corporate compliance, and/ or long-term archiving of unstructured data.

NST Hybrid Storage offers unified storage and access to it using standard NAS and SAN protocols (CIFS, NFS, FTP FC, iSCSI). It also offers high scalability and Imation’s FASTier caching technology to provide performance for mixed application workloads.

E-Series High Density Storage is aimed to customers that needs a lot capacity on a small footprint with low costs. Typically customers use the E-Series for backup-2-disk or as a high-capacity storage. The E-Series offers four different models:

ModelDisksForm factor 
E18V18 disks2U
E32V32 disks2U
E48V48 disks4U
E60V60 disks4U

The E18XV, E32XV, E48XV and E60XV are the corresponding expansion enclosures. You can add up to 2 enclosures to a so called head unit. All models support 3,5″ disks (or 2,5″ in a 3,5″ cage), except the E32(X)V, which offers only support for 2,5″ disks. You can mix different disk technologies (NL, SAS, SSD) in a single system. All controllers offer 1 GbE iSCSI ports (up to 8 ports per controller pair). There is also support for 6G SAS, 8 Gb FC, 16 Gb FC and 10 GbE iSCSI (up to 4 ports per controller pair). All E-Series support array-based snapshots and replication. The E60V with two enclosures can hold up to 1.44 PB (!) on 12U.

Why choose Nexsan E-Series storage as a backend for DataCore SANsymphony-V?

My first two thoughts were “Where are the cool features, like thin-provisioning, tiering etc.?” and “Oh cool, 32 disk on only 2U/ 48 disks on 4U”. And I think this ultimately describes the benefits of the Nexsan E-Series: No unnecessary frills and high capacity. To be honest: The main reason for using dumb SAS JBODs in the past was, that they doesn’t offer unnecessary frills. I don’t need snapshots, replication or thin-provisioning on array level. These features are offered by DataCore SANsymphony-V. In this case, the customer has chosen to switch from a SAS to a FC-attached storage backend. And in this case, a high performance and high density storage without unnecessary frills is the best thing that can happen.

Disclaimer: I’ve never worked with Nexsan before, and they do not pay me for this blog post.

Delivery and first impression

The delivery came directly from Nexsan, Derby. Six pallets, one pallet for each system (2x E32V, 2x E32XV and 2x E48V). The enclosure and disks were packed separately, but in the same package. The packaging was adequate and it was all neatly packed and well padded. Each system was intensively tested by Nexsan before the shipment.

The first step was to mount the rail kits. The rail kits are very solidly built and partially milled from aluminum. Mounting the rail kits was an easy thing using the included template. You have to use the enclosed screws. Unfortunately no screwdriver-less mounting. The rails were anything but smoothly. And it was a precision job to mount the enclosures. But to be honest: How often do you rack the enclosures? One time.

Installation

Each controller has a management port. The first controller has the ip address 10.11.12.13/8, the second controller 10.11.12.14/8. Simply hook up your laptop and connect to one of these IPs. You don’t need login credentials at this point. But you better should assign a password for the ADMIN (case-sensitive!) after finishing the installation.

nexsan_gui_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

If you have multiple Nexsan storage arrays in your network, please make sure that have connected only one system at one time. All Nexsan E-Series storage arrays use the same default ip addresses.

The hardware

The Nexsan E-Series is a classy dual-controller storage. This is a picture of a E32V. As you can see, 16 hard drives are mounted in one drawer. Nexsan’s so called ActiveDrawer Technology allows you to service disks and both fan units (one at the front and one at the end of the drawer) online. All important components are hot-swappable.

nexsan_e32_pod

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The expansion unit E32XV uses the same technology as the E32V. But instead of two controllers, the expansion enclosures has two IO modules. This is the back of the E32V with the SAS-connected E32XV.

nexsan_e32_controller

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

This is a E48V with one pulled out drawer. It also uses the the ActiveDrawer Technology. The controllers are nearly the same, but they have 8 GB cache instead of 4 GB.

nexsan_e48_pod

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The GUI

The GUI is really puristic and fast. Pure HTML with a bit Java Script. You can enable HTTPS if you like. I haven’t noticed any problems with different browsers. The home page gives you a brief overview about the hardware status.

nexsan_webui_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The storage arrays are coming with pre-build RAID arrays and volumes. If you want another setup, simply use the Quick Start feature. Simply enter the number of RAID sets, spares and volumes and the wizard make the rest.

nexsan_webui_3

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

One very cool feature is the multi view. Multi view allows you get a quick overview over multiple E-Series storage arrays.

nexsan_webui_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The GUI is very talkative. Lots of useful information. Very clear and a short help on every page. This is the system info page:

nexsan_webui_4

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The same applies to the RAID info page. Nice: Disk and hosts stats! Very handy.

nexsan_webui_5

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

For our DevOps friends: Each controller offers a config dump page which is plain text and can parsed by a monitoring system or script. The E-Series offers all necessary monitoring features, like e-mail notification, SNMP, Syslog etc. If technical support is needed, the GUI offers a “Technical Support” page which can be used to open a support ticket right from the GUI.

nexsan_webui_6

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

nexsan_webui_7

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Final words

I can’t say anything bad about the Nexsan E-Series. Sure, this is not a HP 3PAR, but should it? It’s a solid block storage and it’s a perfect fit with DataCore SANsymphony-V. The look and feel of the hardware is quite good. Good quality designed in UK. Make sure that you take a look at Nexsan if you’re searching for a solid DataCore backend storage. Patrick Schulz has also some experience with Nexsan and DataCore. Check out his blog posts about DataCore SANsymphony-V and Nexsan!

DataCore mirrored virtual disks full recovery fails repeatedly

Last sunday a customer suffered a power outage for a few hours. Unfortunately the DataCore Storage Server in the affected datacenter weren’t shutdown and therefore it crashed. After the power was back, the Storage Server was started and the recoveries for the mirrored virtual disks started. Hours later, three mirrored virtual disks were still running full recoveries and the recovery for each of them failed repeatedly.

virtual_disk_error_ds10_mirror

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The recovery ran until a specific point, failed and started again. When the recovery failed, several events were logged on the Storage Server in the other datacenter (the Storage Server that wasn’t affected from the power outage):

Source: DcsPool, Event ID: 29

Source: disk, Event ID: 7

Source: Cissesrv, Event ID: 24606

The DataCore support quickly confirmed what we already knew: We had trouble with the backend storage on the DataCore Storage Server that was serving the full recovies for the recovering Storage Server. The full recoveries ran until the point at which a non-readable block was hit. Clearly a problem with the backend storage.

Summary

To summarize this very painful situation:

  • VMFS datastore with productive VMs on DataCore mirrored virtual disks with no redundancy
  • Trouble with the backend storage on the DataCore Storage Server, that was serving the mirrored virtual disks with no redundancy

Next steps

The customer and I decided to evacuate the VMs from the three affected datastores (each mirrored virtual disks represents a VMFS datastore). To avoid more trouble, we decided to split the unhealthy mirrors. So we had three single virtual disks. After the shutdown of the VMs on the affected datastores, we started a single storage vMotions at a time to move the VMs to other datastores. This worked until the storage vMotion hit the non-readable blocks. The storage vMotions failed and the single virtual disks went also into the status “Failed”. After that, we mounted the single virtual disks from the other DataCore Storage Server (that one, that was affected from the power outage and which was running the full recoveries). We expected that the VMFS on the single virtual disks was broken, but to our suprise we were able to mount the datastores. We moved the VMs from the datastores to other datastores. This process was flawless. Just to make this clear: We were able to mount the VMFS on virtual disks, that were in the status “Full Recovery pending”. I was quite sure that there was garbage on the disks, especially if you consider, that there was a full recovery running that never finished.

The only way to remove the logical block errors is to rebuild the logical drive on the RAID controller. This means:

  • Pray for good luck
  • Break all mirrored virtual disks
  • Remove the resulting single virtual disks
  • Remove the disks from the DataCore disk pool
  • Remove the DataCore disk pool
  • Remove the logical drives on the RAID controller
  • Remove the arrays on the RAID controller
  • Replace the faulty physical disks
  • Rebuild the arrays
  • Rebuild the logical drives
  • Create a new DataCore disk pool
  • Add disks to the DataCore disk pool
  • Add mirrors to the single virtual disks
  • Wait until the full recoveries have finished
  • Treat yourself to a beer

Final words

This was very, very painful and, unfortunately, not the first time I had to do this for this customer. The customer is in close contact to the vendor of the backend storage to identify the root cause.

Tiering? Caching? Why it’s important to differ between them.

Some days ago I talked to a colleague from our sales team and we discussed different solutions for a customer. I will spare you the details, but we discussed different solutions and we came across PernixData FVP, HP 3PAR Adaptive OptimizationHP 3PAR Adaptive Flash Cache and DataCore SANsymphony-V. And then the question of all questions came up: “What is the difference?”.

Simplify, then add Lightness

Lets talk about tiering. To make it simple: Tiering moves a block from one tier to another, depending on how often a block is accessed in a specific time. A tier is a class of storage with specific characteristics, for example ultra-fast flash, enterprise-grade SAS drives or even nearline drives. Characteristics can be the drive type, the used RAID level or a combination of characteristics. A 3-tier storage design can consist of only one drive type, but they can be organized in different RAID levels. Tier 1 can be RAID 1 and tier 3 can be RAID 6, but all tiers use enterprise-grade 15k SAS drives. But you can also mix drive types and RAID levels, for example tier 1 with flash, tier 2 with 15k SAS in a RAID 5 and tier 3 with SAS-NL and RAID 6. Each time a block is accessed, the block “heats up”. If it’s hot enough, it is moved one tier up. If it’s less often accessed, the block “cools down” and at a specific point, the block is moved a tier down. If a tier is full, colder blocks will to be moved down and hotter block have to be moved up. It’s a bit simplified, but products like DataCore SANsymphony-V with Auto-Tiering or HP 3PAR Adaptive Optimization are working this way.

Lets talk about caching. With caching a block is only copied to a faster region, which can be flash or even DRAM. The original block isn’t moved, only a copy of the accessed block is copied to a faster medium. If this block is accessed, the data is served from the faster medium. This also works for write I/O. If a block is written, the data is written to the faster medium and will be moved later to the underlying, slower medium. You can’t store block copies until infinity, so less accessed blocks have to be removed from cache if they are not accessed, or if the cache fills up. Examples for caching solutions are PernixData FVP, HP 3PAR Adaptive Flash Cache or NetApp Flash Pool (and also Flash Cache). I lead storage controller cache explicitly not appear in this list. All of the listed caching technologies (except NetApp Flash Cache) can do write-back caching. I wouldn’t recommend read-cache only solutions like VMware vSphere Flash Read Cache, except two situations: Your workload is focused on read I/O and/ or you already own a vSphere Enterprise Plus license, and you do not want to spend extra money.

Tiering or caching? What to choose?

Well… it depends. What is the main goal when using these techniques? Accelerate workloads and making best use of scarce and expensive storage (commonly flash storage).

Regardless of the workload, tiering will need some time to let the often accessed blocks heat up. Some vendors may anticipate this partially by writing data always to the fastest tier. But I don’t think that this is what I would call efficient. One benefit of tiering is, that you can have more then two tiers. You can have a small flash tier, a bigger SAS tier and a really big SAS-NL tier. Usually you will see a 10% flash / 40% SAS / 50% SAS-NL distribution. But as I also mentioned: You don’t have to use flash in a tiered storage design. That’s a plus. On the downside tiering can make mirrored storage designs complex. Heat maps aren’t mirrored between storage systems. If you failover your primary storage, all blocks need to be heaten up again. I know that vendors are working on that. HP 3PAR and DataCore SANsymphony-V currently have a “performance problem” after a failover. It’s only fair to mention it. Here are two examples of products I know well and both offer tiering: In a HP 3PAR Adaptive Optimization configuration, data is always written to the tier, from which the virtual volume was provisioned. This explains the best practice to provision new virtual volumes from the middle tier (Tier 1 CPG). DataCore SANsymphony-V uses the performance class in the storage profile of a virtual disk to determine where data should be written. Depending on the performance class, data is written to the highest available tier (tier affinity is taken into account). Don’t get confused with the tier numbering: Some vendors use tier 0 as the highest tier, others may start counting at tier 1.

Caching is more “spontaneous”. New blocks are written to the cache (usually flash storage, but it can also be DRAM). If a block is read from disks, it’s placed in the cache. Depending on the cache size, you can hold up a lot data. You can lose the cache, but you can’t lose the data ins this case. The cache only holds block copies (okay, okay, written blocks shouldn’t be acknowledged until they are in a second cache/ hose/ $WHATEVER). If the cache is gone, it’s relatively quickly filled up again. You usually can’t have more then two “tiers”. You can have flash and you can have rotating rust. Exception: PernixData FVP can also use host memory. I would call this as an additional half tier. ;) Nutanix uses a tiered storage desing in ther hyper-converged platform: Flash storage is used as read/ write cache, cost effective SATA drives are used to store the data. Caching is great if you have unpredictable workloads. Another interesting point: You can cache at different places in the stack. Take a look at PernixData FVP and HP 3PAR Adaptive Flash Cache. PernixData FVP is sitting next to the hypervisor kernel. HP 3PAR AFC is working at the storage controller level. FVP is awesome to accelerate VM workloads, but what if I have physical database servers? At this point, HP 3PAR AFC can play to its advantages. Because you usually have only two “tiers”, you will need more flash storage as compared to a tiered storage design. Especially then, if you mix flash and SAS-NL/ SATA.

Final words

Is there a rule when to use caching and when to use tiering? I don’t think so. You may use the workload as an indicator. If it’s more predictable you should take a closer look at a tiered storage design. In particular, if the customer wants to separate data from different classes. If you have more to do with unpredictable workloads, take a closer look at caching. There is no law that prevents combining caching and tiering. At the end, the customer requirements are the key. Do the math. Sometimes caching can outperform tiering from the cost perspective, especially if you mix flash and SAS-NL/ SATA in the right proportion.

Update OS or reinstall DataCore SANsymphony-V Storage Server

Sometimes you have to update the OS of your DataCore Storage Server, or the server is crashed and you have to reinstall it. In both cases, a configuration backup is the starting point. The procedure remains the same, regardless if it’s an update or a reinstall after a server crash:

  • Install Windows Server OS
  • Copy configuration backup file to C:\Program Files\DataCore\SANsymphony\Recovery
  • Install DataCore SANsymphony-V

Take a backup

You can take the configuration backup on different ways:

  • Using the DataCore SANsymphony-V Management Console
  • Using the SANsymphony-V Cmdlets for Windows PowerShell

Regardless of how you take the backup, be sure that you have a valid backup! I recommend to take backups in a regular and automated fashion, e.g. with a PowerShell script. I have written such a script in the past: Backup DataCore SANsymphony-V config using PowerShell

You can take the backup with the DataCore SANsymphony-V Management Console by right clicking the server group and then select “Backup Configuration” from the context menu.

ssv_backup_configuration

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

When you use the PowerShell, you have to execute three different Cmdlets:

  • Connect-DcsServer to make a connection to the server group
  • Set-DcsBackupFolder to set the location of the backup folder
  • Backup-DcsConfiguration to backup the configuration and store it in the backup folder

The Cmdlets take the configuration backup for all servers in the server group! Make sure that you copy the configuration backups to a safe location. Make sure that you copy a valid backup of each server in the server group. On thing is important: If you take the backups online (DataCore Storage Server is running!), then you will see full recoveries in case of a restore. If you plan to reinstall a DataCore Storage Server, stop the DataCore Storage Server and then take the configuration backup. In case of a clean shutdown, only log recoveries will be necessary.

Restore the backup

Disconnect backend and mirror ports before you reinstall the OS! Install the Windows OS according to the DataCore guidelines (meet the prerequisites, read the know errors with 3rd party components PDF, check name resolution etc.). Make sure that you install the same build of DataCore SANsymphony-V that was used prior the reinstallation. Don’t install newer or older builds! Install exactly the build that was used when taken the configuration backup. Create the folder structure “C:\Program Files\DataCore\SANsymphony\Recovery” and copy the ZIP file into it. Start the DataCore SANsymphony-V installation. You will be prompted during the installation, that a saved configuration was found.

ssv_restore_config

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

If not, log a call at DataCore or follow the instructions in the DataCore online help. You also need the DcsAdmin password during the installation! I hope you have written it down somewhere. ;) After finishing the installation, shutdown the server and reconnect backend and mirror ports. Power-on the server and open the SANsymphony-V Management Console. If everything’s fine, start the DataCore Storage Server and watch the mirror recoveries. Take a configuration backup and support bundles. Proceed with the next servers in the server group.

Final words

The process is quite simple. If you’re unsure about the correct steps, log a call at the DataCore support or take a look into the DataCore online help. Don’t try in-place upgrades to update the OS. I also don’t recommend to take images of running storage servers. Just reinstall the OS and use configuration backups to restore the configuration.

DataCore In SANsymphony-V 10: Potential for data corruption

This is only a short blog post. Just got an e-mail from the DataCore Support. They found a critical bug in SANsymphony-V 10.0.0.0 which should be fixed with Update 1. Only VMware customers are affected, because the bug is related to VMware Thin Provisioning Thresholds. Update 1 is planned for early September 2014. If you’re running SANsymphony-V 10.0.0.0 open an incident at the DataCore Support to get an available hotfix. If you have planned to update to SANsymphony-V 10, delay this update until the release of SANsymphony-V 10 Update 1.

DataCore announces SANsymphony-V10

Today DataCore announced their latest SANsymphony-V release. After the merge of SANmelody & SANsymphony, SANsymphony-V10 is the 10th generation of DataCores flagship product. Interestingly DataCore uses the terms  “software-defined” and “Virtual SAN”. Whether the product of the definition of the terms corresponds everyone should decide for themselves. But this is another story.

What is DataCore SANsymphony-V?

What DataCore definitely does is automating and simplifying storage management and provisioning. I really like it the simplicity. DataCore SANsymphony-V can deliver enterprise-class functionality, like synchronous mirroring, replication, snapshots, clones, thin-provisioning and tiering . It runs on x86 hardware with Microsoft Windows Server 2008 or 2012. Multiple servers can grouped together for load balancing and redundancy. A storage pool can created out of the internal or external flash and roting rust. Single or mirrored virtual disks can be carved out of this storage pool. Hosts can access these virtual disks using iSCSI or Fibre-Channel. Because DataCore SANsymphony-V10 can use several different technologies as backend for storage pools, it’s easy to replace backend storage. You can add or remove disks to or from storage pools. If you backend storage is an old EMC CLARiiON and you get a new HP MSA 2040 Storage, you can replance the old storage without disruption.

What’s new in SANsymphony-V10?

I took this information directly from the DataCore SANsymphony-V10 announcement page:

  • Scalability has doubled from 16 to 32 nodes; Enables Metro-wide N+1 grid data protection
  • Supports high-speed 40/56 GigE iSCSI; 16Gbps Fibre Channel; iSCSI Target NIC teaming
  • Performance visualization/Heat Map tools add insight into the behavior of Flash and disks
  • New auto-tiering settings optimize expensive resources (e.g., flash cards) in a pool
  • Intelligent disk rebalancing, dynamically redistributes load across available devices within a tier
  • Automated CPU load leveling and Flash optimizations to increase performance
  • Disk pool optimization and self-healing storage; Disk contents are automatically restored across the remaining storage in the pool; Enhancements to easily select and prioritize order of recovery
  • New self-tuning caching algorithms and optimizations for flash cards and SSDs
  • ‘Click-simple’ configuration wizards to rapidly set up different use cases (Virtual SAN; High-Availability SANs; NAS File Shares; etc.)

Along with new feature DataCore has announced is a new licensing model. Aside the traditional server license, there will be a Virtual SAN license which includes tiering, adaptive caching, storage pooling, synchronous mirroring, thin provisioning and snapshots/ clones. Both variations, the traditional SANsymphony-V10 and the Virtual SAN, running on Windows Server 2012. So the Virtual SAN will not be a virtual appliance. AFAIK it will only be a special license.

The general availability is scheduled for May 30, 2014. So stay tuned. :) I hope to get SANsymphony-V10 as fast as it’s possible into my lab.

DataCore SANsymphony-V 9.0 PSP4 Update 3 – Update recommended

About two weeks after the release of DataCore SANsymphony-V 9.0 PSP4 Update 2, DataCore announces Update 3. This is a really short release cycle… DataCore fixed three issues in Update 3. This is an excerpt from the release notes:

Problem: SANsymphony 9.0 PSP4 Update2 failed to update configurations with shared pools on DataCore Servers running SANsymphony 9.0 PSP3, PSP3 U1 or PSP3 U2.
Cause: An upgrade script run during installation expected a cmdlet parameter that wasn‟t supported in these versions.
Resolution: Updated the script to no longer rely on this parameter.

Problem: A system crash in the storage pool driver could occur when using space reclamation on a virtual disk whose size had been reduced or in rare cases where certain I/O operations associated with space reclamation failed.
Cause: An I/O completion routine was implemented incorrectly.
Resolution: A code change was made to the completion routine to correct the issue.

Problem: In specific scenarios where a mirrored virtual disk is in an unhealthy condition and is the source disk of an in-progress ESX virtual machine migration, third party XCOPY commands could copy stale data to the destination disk.
Cause: XCOPY commands were incorrectly allowed to succeed even though a VM source or destination volume was marked as inaccessible to a host.
Resolution: Fail these commands appropriately when the virtual disk is not healthy.

Customers that are still run DataCore SANsymphony-V 9.0 PSP4 Update 1 or an earlier release are highly recommended to update to the latest release. Customers that run Update 2 can take actions to avoid two of the three issues. Customers that run update 2 shouldn’t resize volumes and they should disable VAAI before doint a svMotion. In my opinion both actions will not avoid to update to Update 3. The action can save up to over a certain time window, until the update 3 can be installed.

DataCore SANsymphony-V 9.0 PSP4 Update 2 – Update recommended

Yesterday I got an e-mail from DataCore in which Update 2 for DataCore SANsymphony-V PSP4 was announced. DataCore found a critical issue in all releases since SANsymphony-V 9.0 PSP3. According to the releases notes a situation can occur, in which storage space reclamation and migration can happen at the same time. This can lead to a situation in which two storage allocation unit (SAU) can point to the same disk offset. If this happens, the disk pool can be marked offline.

From my point of view only those customers are directly affected that use DataCore Auto-Tiering. Nevertheless I recommend to all DataCore customers to update to the latest SANsymphony-V release. You can download the latest release from the DataCore Support Portal. The update can be done online if you have a mirrored DataCore enviroment.

Update: Patrick mentioned in his blog, that SAU migration can also occur in other situations, e.g. adding or removing disks to or from a disk pool. The space reclamation starts as soon as a virtual disk is deleted or SSV detects a all zeroed virtual disks. So deleting a virtual and adding a physical disk to a pool can also lead to a offline disk pool. Thanks to Patrick for clarification.