Hammerspace Deployment and Integration

Hammerspace Installation

General installation of Hammerspace nodes is well-documented in the Hammerspace Installation and Licensing Guide. What follows here are specifics that may apply to Tier 0 in general, or your environment.

Anvils: If Anvils are to be installed in different subnets, BGP (Border Gateway Protocol) is required, using the ExaBGP tools in the Anvil nodes to communicate with connected switches in order to control which port routes the subnet for the Anvil floating management IP. Contact your Hammerspace tech team for architecture and deployment details.
DSXs: DSX may not be required for some Tier 0 installations. Mover (DI) may be deployed as RPM or container. See Appendix 1 for instructions on deploying the DI RPM.

Cluster Integration

LSS and Tier 0 storage services can be added using the Hammerspace GUI, CLI, or scripted with the API. For a small installation, it may be simple enough to add storage systems via GUI or CLI. For larger installations, setup should be automated using a script or configuration management or automation tools such as Ansible or Terraform.

Adding a Tier 0 Node or LSS as a Storage Node

Add a storage node using the node-add command:

anvil1> node-add --type OTHER --name lss001 --ip 10.1.2.2

If you plan to use per-node objectives (for example to achieve local write affinity), you can have the node-add command create them as part of adding the node:

anvil1> node-add --type OTHER --name lss001 --ip 10.1.2.2 --create-placement-objectives

If successful, the node-add command responds with a description of the node, with the exports listed as logical volumes.

Add Node Exports as Volumes

Add each export as a volume using the volume-add command:

anvil1> volume-add --name lss001::/hsvol1 --node-name lss001 --access-type read_write --logical-volume-id 123 --low-threshold 90 --high-threshold 95 --skip-performance-test

You can use either --logical-volume-name or --logical-volume-id to specify the volume. With scripting or other automation, it may be easier to use the logical volume name, since for simple NFS servers, the logical volume name is the same as the export. Alternatively, you can get the list of logical volume IDs from the output of the node-add command. When adding many volumes in rapid succession, you should skip the performance test.

If you plan to use availability zones, which are a best practice, the volume name must be prefixed with "AZ", a whole number, colon, then a unique name.

The example below shows adding a volume with the "AZ1:" prefix.

anvil1> volume-add --name AZ1:lss001::/hsvol1 --node-name lss001 --access-type read_write --logical-volume-id 123 --low-threshold 90 --high-threshold 95 --skip-performance-test

You can also name the node with the AZx: prefix, then by not specifying the --node-name parameter, with the volume-add command, the volume name will default to node::/path. With this trick, the volume name gets the AZx: prefix from the node name, since all volumes on a node would be in the same AZ.

Create Share(s)

You can create many shares for various purposes, such as checkpointing, to contain job-specific data, or results/output.

Run the following to create a share:

anvil1> share-create --name checkpoints --path /checkpoints --export-option 10.1.2.0/24,rw,root-squash

If you don’t specify --path, it will default to the share name. You can specify a size, but must plan this carefully so jobs and applications don’t run out of space unexpectedly due to a share quota. If size is not specified, the share size and free space will show as the total size and free space of all local (NFS, not object) volumes.

You must provide the necessary set of export options to include all clients needing access to the share, including all Tier 0 nodes. If LSS nodes are not used as clients, they do not need to be listed in the Hammerspace share exports. You may need to specify multiple --export-option parameters. If the command line gets too long, you can add export options using the share-update command. Clients can be specified by IP address, subnet in CIDR format, hostname, FQDN, or netgroup.

Hammerspace exports support ro or rw, root_squash or no_root_squash, and secure (default) or insecure. Generally, "rw,root_squash,secure" is the recommended set for Hammerspace exports to Tier 0 nodes.

You can specify objectives during share creation or add them later. See Data Placement Using Objectives.

Mounting Shares on Clients

Once shares are created, clients can mount them using NFS v4.2. The exact mount options may depend on a variety of factors including use case, model of NICs, and especially client kernel version, since the kernel version dictates which advanced NFS features are available.

# mkdir /mnt/checkpoints

# mount -o vers=4.2,nconnect=8 <anvil_IP>:/checkpoints /mnt/checkpoints

Mount Parameter and Option Notes

For NFSv4.2, the IP or other host specification used is for the floating data IP of the Anvil, even though the data resides on Tier 0 and/or LSS nodes.

If mount fails with "mount.nfs: Protocol not supported", first check firewall settings. If firewall is not the issue, your Linux kernel may be newer than the Hammerspace supported kernel whitelist allows. You can bypass this using the mount option port=20492, but please report the kernel version to your Hammerspace technical team.

Guidance for Setting NCONNECT with Respect to NFS Threads

When using NFS with nconnect in large-scale environments, it’s important to manage oversubscription of the NFS server’s knfsd threads. Excessive oversubscription can lead to contention, latency, or degraded performance.

Goal

Maintain a maximum 8:1 oversubscription ratio of client connections (nconnect) to server knfsd threads, while also enforcing a minimum of 2 and a maximum of 16 to ensure basic parallelism.

Formula

Given:

T = number of knfsd threads (from nfs.conf, e.g., threads=128)

N = number of client nodes expected to connect to the server

R = desired oversubscription ratio (e.g., R = 8)

C = number of nconnect connections each client should use

Then:

C = max(2, floorT * R) / N

Example

If:

T = 128 knfsd threads

N = 1023 clients

R = 8 (oversubscription target)

Then:

C = max(2, floor128 * 8) / 1023

= max(2, floor(1024 / 1023))

= max(2, 1)

= 2

You should configure nconnect=2. This results in a 16:1 oversubscription in this specific scenario, which exceeds the 8:1 target but honors the enforced minimum of 2.

Takeaway

To optimize performance, adjust nconnect downward as the cluster grows, using the formula above. Monitor actual server thread utilization to verify assumptions, especially under real workloads. Consider increasing threads in nfs.conf on high-demand servers if consistent oversubscription is unavoidable.

Noatime

With most AI and similar workloads, access time is not a critical file attribute to maintain. Hammerspace recommends setting noatime.