Maintenance

Regardless of the failure type, once a node is determined to be non-recoverable (hard) or has exceeded the Max Suspected Time (soft-turned-hard failure), you must:

When taking Tier 0 nodes offline briefly for maintenance, whether for a simple reboot, replacing a failed component, or other brief task, it may not be desirable for files with instances on the down node to be resilvered (copied from other nodes that have instances to one or more nodes that don’t, in order to keep the instance count compliant with the policy).

Hammerspace refers to the state where a file has all the instances it is supposed to as "aligned to its objectives". To prevent data mobility from making more instances during a brief, planned outage, Hammerspace offers the ability to disable the function that determines a volume is not available, and the data requires resilvering. The per-volume parameter is called availability-drop.

Another important consideration before taking nodes offline is whether the data on them is currently aligned to its objectives. In other words, do the files with instances on the volumes of these nodes have all the instances they should have somewhere else? Here are two ways to see that.

In the Hammerspace GUI, go to Infrastructure > Volumes, then click the volume name, then click Alignment at the top.

In this example, 2 files are unaligned, which should be resolved before this node is taken offline. The mobility screen in the GUI may provide clues such as failed mobilities from the volume. The List view can show specific files that are failing mobility, along with their sources and destination volumes. The following HSTK command shows misaligned files for a specific volume in a specific share:

Note that you may need to run the above HSTK command for each Hammerspace share.

You can also query alignment of files with instances on a given volume using the API. This curl command queries the same volume in the GUI example above, referenced by its UUID.

Data must be removed and placed on other storage (evacuated) before a node can be taken offline long term or permanently. While Hammerspace will self-heal when there are objectives requiring multiple instances and "the plug is pulled" on a node, it is safer to decommission the volume(s) using the built-in functionality, either via GUI, CLI, or API, which can also remove the volume from Hammerspace after evacuation. The volume-decommission command evacuates the volume without removing it from Hammerspace. You can then use volume-remove to remove it once you’re satisfied that the data is evacuated. You can combine both of these by just running volume-remove which will do the decommission if it wasn’t already done. There is a --replace-with parameter for these two commands, but it is usually better to let Anvil figure out for itself where to place the evacuated instances.

It is also possible to use an exclude-from-node objective to evacuate data from all the volumes on the node. You still need to remove the volumes from Hammerspace.