Troubleshooting

Cannot Add Volume

Anvil may refuse to add volume possibly with error message "Cannot add a volume which has 0 bytes of storage capacity." This usually indicates a problem with exports on the Tier 0 node or LSS.

What to check:

Exports are correct, particularly that Anvil fixed and floating IPs are listed for each volume with no_root_squash

You may need to rescan the node (GUI node rescan button or CLI node-refresh --name <nodename> ) after changing exports, or in extreme cases, remove and re-add the node.

Volume Issues - Volumes Go "suspected"

Causes: Node gets rebuilt, NVME reformatted or remounted in a different path. The mount directory might still be exported, although the mp (mountpoint) export option should prevent the empty directory from being exported.

What to check:

Storage node (LSS or Tier 0) is up
NFS service is running
Exports are correct
Volumes or RAID arrays have not been reformatted or remounted
Hammerspace comb structure (/PrimaryData) is intact

If you see multiple volumes from a single storage server go suspected, and the FSID has changed, if the new FSIDs are the same on multiple volumes on the same server, this most likely indicates that it is the root filesystem being exported due to empty directory mount points.

In this example, it is fairly obvious that the volume is not properly exported.

anvil1> volume-list

ID: 190ad494-c79a-42e8-bac5-38587a198b19

Name: AZ3:node3::/mnt/hsvol1

Internal ID: 20

Discovered addresses:

[IP: 10.200.100.42/32, Port: 2049, NetId: tcp, NodeNum: 0]

Effective IPs:

[IP: 10.200.100.42/32, Port: 2049, NetId: tcp, NodeNum: 0]

Path: /mnt/hsvol1

State: OK

Access type: Read Write

Node: AZ3:node3

Oper state: Suspected

Oper state reason: path /mnt/hsvol1 is not exported at 10.200.100.42

Mobility Failures (Unsuccessful)

File instances aren’t where they should be (not enough instances, or not on the right volumes)

GUI Mobility screen shows unsuccessful mobilities that don’t resolve in short order.

Common causes include:

No DI (Data Instantiator or NFS Mover), or DI services are not running properly. The Hammerspace GUI and CLI alerts will indicate if there are no DIs. Check the status of the DIs.
DIs don’t have RW, no_root_squash on exports of the storage servers

Mobility Troubleshooting

If files don’t appear to have the right number of instances in the right places, especially after changing objectives or creating files with more than 3 instances, check the mobility screen in the Hammerspace GUI.

If you see any "Unsuccessful" mobilities, the most likely cause is one or more movers not having access to the exports on the source or destination storage node or volume. Check that all movers are listed in the exports of all volumes, and that they have no_root_squash.