Checking the 3PAR Quorum Witness appliance

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Two 3PAR StoreServs running in a Peer Persistence setup lost the connection to the Quorum Witness appliance. The appliance is an important part of a 3PAR Peer Persistence setup, because it acts as a tie-breaker in a split-brain scenario.

While analyzing this issue, I saw this message in the 3PAR Management Console:

3PAR Quorum Witness Status

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

In addition to that, the customer got e-mails that the 3PAR StoreServ arrays lost the connection to the Quorum Witness appliance. In my case, the CouchDB process died. A restart of the appliance brought it back online.

How to check the Quorum Witness appliance?

You can check the status of the appliance with a simple web request. The documentation shows a simple test based on curl. You can run this direct from the BASH of the appliance.

But you can also use the PowerShell cmdlet Invoke-WebRequest.

If you add /witness to the URL, you can test the access to the database, which is used for Peer Persistence.

If you get a connection error, check if the beam process is running.

If not, reboot the appliance. This can be done without downtime. The appliance comes only into play, if a failover occurs.

Checking the 3PAR Quorum Witness appliance
3.8 (75%) 4 votes
Patrick Terlisten
Follow me

Patrick Terlisten

vcloudnine.de is the personal blog of Patrick Terlisten. Patrick has a strong focus on virtualization & cloud solutions, but also storage, networking, and IT infrastructure in general. He is a fan of Lean Management and agile methods, and practices continuous improvement whereever it is possible.

Feel free to follow him on Twitter and/ or leave a comment.
Patrick Terlisten
Follow me

3 thoughts on “Checking the 3PAR Quorum Witness appliance

  1. Sergey

    Great! Useful information. And there is no information on how (for example using SNMP) to monitor the state of services on the arrays themselves. They tend to hang (sometime) when the connection is broken to the quorum.

    Reply
    1. Patrick Terlisten Post author

      Unfortunately, I don’t have any clue on how to monitor this. I know what you mean, because I have seen communication loss to the quorum a couple of times, especially if the quorum is located in another site. You can try something like SSH with expect and check the output of the command issued to the arrays.

      Reply
      1. Sergey

        Thank you for your feedback. Would like to use standard tools and not SSH, POWERSHELL or other crutches.

        Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

I accept!