Installing and Configuring a Quorum Server

Hammerspace requires using an external quorum server to improve resilience against certain failure conditions, especially in a two-node HA configuration. The supported external quorum server software is Qnetd, and it is available as an open-source package for most Linux distributions.

A single quorum server can serve multiple clusters.

External quorum server requirements:

Qnetd (version 3.0 or later) is installed.
Server resources are configured according to the vendor-recommended Linux requirements. A minimal server configuration is sufficient. The Qnetd software does not require significant resources to operate.
The choice of Linux distribution is up to the customer.
The CPU architecture can be either x86_64 or ARM.
Only one quorum server is required and supported per cluster.
The server must be able to connect to all Anvil nodes in the cluster.
Must be in a different failure domain than the Anvil nodes to ensure it has the impact that is expected.

The importance of the quorum server is its functionality. A quorum server enables a cluster to use a non-cluster node as a voting member. This is especially helpful when cluster nodes lose connectivity with each other, a condition that can lead to a split-brain.

When a cluster membership change occurs, the surviving side with the majority of votes will become (or remain) the primary node and continue to provide services to clients.

The quorum server enables automatic decision-making for the cluster in a split-brain situation, eliminating potential downtime caused by manual steps.

Installing Qnetd

The following instructions showing how to install Qnetd on a regular Linux node are provided for two different Linux distributions. Other Linux distributions can also be used.

On Red Hat and Red Hat-like Distributions

Use the following commands to install Qnetd on Red Hat or Red Hat-like distributions:

Installing Qnetd on Red Hat or Red Hat-like Linux

# dnf install -y corosync-qnetd

Disable TLS:

# sed -i 's/COROSYNC_QNETD_OPTIONS=""/COROSYNC_QNETD_OPTIONS="-s off"/g' /etc/sysconfig/corosync-qnetd

Enable and start the service:

# systemctl enable corosync-qnetd.service

# systemctl start corosync-qnetd.service

Open firewall ports (if needed):

# firewall-cmd --zone public --add-service=high-availability

# firewall-cmd --zone public --add-service=high-availability --permanent

On Ubuntu

Use the following commands to install Qnetd on Ubuntu Linux:

Installing Qnetd on Ubuntu Linux

# apt install -y corosync-qnetd

Disable TLS:

# sed -i 's/COROSYNC_QNETD_OPTIONS=""/COROSYNC_QNETD_OPTIONS="-s off"/g' /etc/default/corosync-qnetd

Enable and start the service:

# systemctl enable corosync-qnetd.service

# systemctl start corosync-qnetd.service

Open firewall ports (if enabled; 5403 is the default qnetd port):

# ufw allow 5403

Once the Qnetd software is installed on the Linux server, no further configuration is required on the Linux quorum server.

Configuring an External Quorum Server

After the quorum server is installed, the cluster must be configured to use it.

External quorum server functionality is configured using the sysadmin CLI, not the Hammerspace Admin CLI.

Configuring external quorum server

# ssh sysadmin@<ANVIL CLUSTER IP>

...

Run quorum-device-set to configure 192.0.2.50 as the quorum server. Sample output included below.

sysadmin> quorum-device-set --host 192.0.2.50

anvil2.example.com: Succeeded

anvil1.example.com: Succeeded

Corosync configuration has been synchronized, please reload corosync daemon using 'pcs cluster reload corosync' command.

Corosync reloaded

anvil2.example.com: Cluster Enabled

anvil1.example.com: Cluster Enabled

anvil2.example.com: Starting Cluster...

anvil1.example.com: Starting Cluster...

Troubleshooting

If you get an error like this while trying to configure a quorum server, then it is likely that one of the Anvil nodes is not reachable at the moment.

Sysadmin CLI

Unable to connect to anvil2.example.com, check if pcsd is running there or try setting higher timeout with --request-timeout option (Failed to connect to anvil2.example.com port 2224: Connection refused)

Error: Unable to set corosync config: Unable to connect to anvil2.example.com, check if pcsd is running there or try setting higher timeout with --request-timeout option (Failed to connect to anvil2.example.com port 2224: Connection refused)

Command '['pcs', 'cluster', 'sync']' returned non-zero exit status 1.

An external quorum server is required for high-availability (two-Anvil) deployments.

Checking Status of Configured Quorum Server

You can check the status of the external quorum server using the following command:

Check the status of external quorum server

# ssh sysadmin@<ANVIL CLUSTER IP>
...
Run quorum-device-status to verify the configuration.
sysadmin> quorum-device-status

Qdevice information
-------------------
Model: Net
Node ID: 2
Configured node list:
0 Node ID = 1
1 Node ID = 2
Membership node list: 1, 2

Qdevice-net information
----------------------
Cluster name: hs-0f9e8d7c-6b5a-4321-8000-000000000001
QNetd host: 192.0.2.50:5403
Algorithm: Fifty-Fifty split
Tie-breaker: Node with lowest node ID
State: Connected