Search the docs

Data Availability and Durability

This section describes objectives related to data availability and protection.

The system treats availability and durability in a very similar fashion, but there are some subtle differences. One way to think of it is protection against temporary vs permanent loss of data access.

Availability represents temporary loss of (access to) data, while durability represents permanent loss of data. As such, durability should always be greater than availability, since data that is permanently lost is, by definition, inaccessible.

One subtle way the difference comes into play is that if you want to temporarily take a storage node offline, you can lower its configured availability to match the time you expect to keep it offline. A short service window may mean you don’t lower the availability at all. A longer one means you lower it, possibly even all the way to zero. By lowering the availability, data that is resident there, and with availability objectives not otherwise met, will need to be moved (or multiply instantiated) onto other volumes.

In other words, to service storage, you lower availability. To decommission storage, the system lowers durability to zero.

This is why you need two measures to express the temporary vs. permanent nature of the loss.

The Availability Number-of-Nines Objective

Three availability objectives enable the admin to specify the number of nines of availability the data should have. Each storage volume is assigned an availability value when it is added. Data will be placed either on a storage volume that meets the availability number, or multiple file instances may be created to meet the requested number of nines.

If you configure a share with an availability objective that causes multiple file instances to be created, and the volume containing some or all those files becomes marked as SUSPECTED (unavailable) within the cluster, the cluster will re-silver the impacted files after 30 minutes by default. If the volume SUSPECTED status later clears, the cluster will automatically remove any excess file instances.

The default availability objectives allow you to specify either 1, 3, or 5 nines. Custom objectives can be created that allow for up to 11 nines in the GUI and up to 20 nines in the Admin CLI.

Best Practices

  • Use availability objectives to control how many file instances are created, so that if a storage volume becomes unavailable, a file can still be read from another volume. The availability objective will always try to place the file instances on different failure domains (meaning different hosts) to ensure this happens.

  • Double-check your volume availability and durability values! While Hammerspace does apply defaults, not all volumes are created equal. For example, a self-hosted object storage volume on a standalone S3 server is likely less durable and available than one managed by AWS, Google, or Microsoft Azure; or a DSX volume backed by a RAID is better than one that is not. To edit the settings, go to the Infrastructure page and click the pencil icon to the right of the volume name. In the Volume Properties Details tab, scroll down to the Capabilities section and edit the Availability and Durability values as needed.

  • The volume durability value should always be larger than the availability value.

Volume Availability - Default Values

Volumes are assigned the following default availability values that can be changed if required:

  • All DSX or remote NFS (online) volumes: 2 nines

  • All Object storage (offline) volumes: 4 nines

Based on these default values, the share default objectives will only create one copy of every file.

Mirroring Data Between NFS Volumes

You can mirror data across NFS volumes using the availability-3-nines objective. This objective creates two copies of your data on different storage volumes. This will protect your data if one of the storage volumes fails.

Complete the following steps to apply the availability-3-nines objective to a share in your environment.

  1. Enter the cluster IP in your browser to open the Management GUI.

  2. Select Data from the left-side panel and see the Shares tab.

  3. Locate the share you want to configure and click the pencil icon for that share in the Applied Objectives column.

  4. You can apply the availability-3-nines objective to the share or directories or files within the share.

  5. Optional: Navigate within the share to the directory or file that you want to mirror, and click the pencil icon in the Applied Objectives column for that object.

  6. In the popup, click Add Objective and in the Filter, type “availability” and press Enter.

  7. Click the box for availability-3 nines and click Apply.

  8. In the Objectives tab for the share or path, you can see the new availability-3-nines objective and verify that its applicability is TRUE, and it is Active. Click Close to finish.

Using the Durability Number-of-Nines Objective

Three durability objectives enable the admin to specify how many nines of durability the data should have. Each storage volume is assigned a durability value. Data will be placed either on a storage volume that meets the durability number (minimum is 1) or multiple file instances may be created to meet the requested number of nines.

If you configure a share with a durability objective that causes multiple file instances to be created, the cluster will only resilver the impacted files if the volume is decommissioned or failed. Decommissioning a volume gracefully moves the file instances it contains to other volumes, while failing a volume marks all file instances that it contains as permanently lost. Volumes are typically only failed if their underlying storage has failed.

The default durability objectives allow you to specify one, three, or five nines. Custom objectives can be created that allow for up to 11 nines in the GUI and up to 20 nines in the Admin CLI.

Durability and availability are calculated separately. Based on the cluster volume configuration, each may require a different number of file instances to be created.

Volume Durability – Default Values

Volumes are assigned the following default durability values that can be changed if required:

  • All DSX or remote NFS (online) volumes: three nines

  • All Object storage (offline) volumes: nine nines

Based on these default values, the share default objectives will only create one instance of every file.