Search the docs

Validation

Configuration Validation

  • Using the GUI or CLI, check the storage system list and the volume list to see that the expected systems and volumes are added

  • Check that newly added volumes have 0 capacity used. If a volume has an unexpected used size, it is possible or even likely that the device is not mounted as intended and what is exported is the empty directory in the root volume of the server. Using the mp export option on the NFS exports should prevent this issue.

  • Check the root file handle on volumes added from the same server. If the root file handles are the same on different volumes, it is likely that the devices are not mounted as intended and what is exported are empty directories in the root volume of the server. Again, the mp export option should prevent this issue.

Command:

volume-list --name AZ2:node2::/mnt/hsvol1

Expected output:

ID: d0f22c08-4089-4a5c-b3ee-4bdc52523b21
Name: AZ2:node2::/mnt/hsvol1
Internal ID: 15
Discovered addresses:
[IP: 10.200.103.53/32, Port: 2049, NetId: tcp, NodeNum: 0]
Effective IPs:
[IP: 10.200.103.53/32, Port: 2049, NetId: tcp, NodeNum: 0]
Path: /mnt/hsvol1
State: OK
Access type: Read Write
Node: AZ2:node2
Oper state: Up
Admin state: Up
Capacity: [Total: 7TB, Used: 158.6MB (<1%), Free: 7TB]
Capabilities: Max read IOPs: 0
Max write IOPs: 0
Min read latency: 0.001 milliseconds
Min write latency: 0.001 milliseconds
Max read bandwidth: 0 KB/s
Max write bandwidth: 0 KB/s
NFS read request size: 1.0 MiB
NFS write request size: 1.0 MiB
Tolerable read latency: 100000 milliseconds
Tolerable write latency: 100000 milliseconds
High threshold: 95%
Low threshold: 91%
Availability: 99%
Availability (effective): 99%
Availability drop: Disabled
Durability: 99.9%
Durability (effective): 99.9%
Encryption: false
Clone status: Not supported
Online delay: Online
Created: 2025-07-10 18:13:02 UTC
Modified: 2025-07-10 18:13:02 UTC
Root file handle: 010006002ec5b990aceb436693c0d035b7325b53
Locations:
[Type: Node, ID: 74a111ce-6069-4427-8ebd-d75ddae8a63a, Name: AZ2:node2]
[Type: Storage Volume, ID: d0f22c08-4089-4a5c-b3ee-4bdc52523b21, Name: AZ2:node2::/mnt/hsvol1]
[Type: Volume Group, ID: 22c4be93-2717-4dd5-badd-345d1c912e62, Name: all]
Random writes: Enabled

Data Placement

Using one of the clients, create a number of files in a Hammerspace share, ideally more files than the number of volumes or a multiple of the number of volumes as the number of files. Check the locations of these files. You can use tools like HSTK (Hammerspace ToolKit - a BASH plugin) and the command hs usage volume <share path> to see the distribution of files across the volumes. You can look at the placement of individual files using hs eval -e instances.volume <filename>.

Data Placement Using Objectives

Objectives are Hammerspace rules that control the behavior of files. They control creation and placement of instances on storage volumes and other advanced functionality. There is extensive documentation on objectives in the Hammerspace Administration Guide, and in the How to Configure Hammerspace Objectives [Support article may require login to view.].

Objectives can be set on the whole share, or directories and files within a share, giving the flexibility and granularity to control files for many use cases.

Default Behavior

Simply by creating a share in Hammerspace, you get a set of 8 default objectives that control sane behavior of files. For example, they prevent instances from filling up one volume when there are other appropriate volumes for the data in question. Effectively, they balance the usage of volumes.

If you create a set of files on a share with default objectives, you should see the files spread across all volumes with available space. These objectives will respect capacity thresholds for volumes. If a volume is at or above the low threshold, data will be placed in other volumes. If a volume is at or above the high threshold, data will be evacuated to other volumes.

Assuming a share called share1, we can test this behavior as follows:

# mkdir /mnt/share1

# mount -o vers=4.2,nconnect=8 <anvil_IP>:/share1 /mnt/share1

# mkdir /mnt/share1/dir1

# cd /mnt/share1/dir1

# for f in `seq 1 100`; do dd if=/dev/urandom of=f$f count=10 bs=1024k; done

10+0 records in

10+0 records out

10485760 bytes (10 MB, 10 MiB) copied, 0.0790123 s, 133 MB/s

<snip>

# hs usage volume

SUMS_TABLE{

        |KEY = STORAGE_VOLUME('AZ1:node1::/mnt/hsvol0'),

        |VALUE = 9;

        |KEY = STORAGE_VOLUME('AZ1:node1::/mnt/hsvol1'),

        |VALUE = 9;

        |KEY = STORAGE_VOLUME('AZ2:node2::/mnt/hsvol1'),

        |VALUE = 9;

        |KEY = STORAGE_VOLUME('AZ2:node2::/mnt/hsvol0'),

        |VALUE = 9;

        |KEY = STORAGE_VOLUME('AZ3:node3::/mnt/hsvol0'),

        |VALUE = 8;

        |KEY = STORAGE_VOLUME('AZ3:node3::/mnt/hsvol1'),

        |VALUE = 8;

        |KEY = STORAGE_VOLUME('AZ4:node4::/mnt/hsvol0'),

        |VALUE = 8;

        |KEY = STORAGE_VOLUME('AZ4:node4::/mnt/hsvol1'),

        |VALUE = 8;

        |KEY = STORAGE_VOLUME('AZ5:node5::/mnt/hsvol0'),

        |VALUE = 8;

        |KEY = STORAGE_VOLUME('AZ5:node5::/mnt/hsvol1'),

        |VALUE = 8;

        |KEY = STORAGE_VOLUME('AZ6:node6::/mnt/hsvol1'),

        |VALUE = 8;

        |KEY = STORAGE_VOLUME('AZ6:node6::/mnt/hsvol0'),

        |VALUE = 8}

The hs command above is from the HSTK. You can get this from GitHub, and it is extensively described in Hammerscript and Hammerspace Toolkit (HSTK) [Support article may require login to view.].

Simple Objective to Make Multiple Instances

There are a few different ways to create and apply objectives to create multiple instances of files on different storage volumes. You could create "place-on" objectives for each volume, but that quickly becomes impractical since you would have to apply different objectives to each file. A simpler way to use an objective to make multiple instances and have them balanced across volumes, nodes or even AZs is to use a durability or availability objective, specified as a number of "9s" such as 99.999%, also referred to as "five nines". Some of these objectives are created by default, but you can create more with different durability or availability requirements. The main difference between durability and availability is that with durability, the system or volume might go offline, but the data is intact and will be there when the volume comes back online. With availability, the volume is expected to stay online for the specified percentage of time.

The way this works is when a durability or availability objective is applied, the system looks for a storage volume that by itself meets or exceeds the requirement. If there isn’t one, the system determines a set of volumes that together add up to the requirement and places an instance of the file on each volume in the set. It turns out that the probability math works out that you can add up the number of nines from each volume and get the same effective reliability of a single volume with that number of nines. When there are many volumes and multiple files are created, the system determines a different set of volumes for each file.

It goes even further. If you configure availability zones using the "AZx:" prefix for volume names as discussed earlier, the system will place instances such that no two instances of the same file are placed in volumes in the same availability zone.

The default volume durability is 99.9% and availability is 99%. These values can be changed, and for most storage platforms, should be set to values determined based on the storage system and the vendor documentation and guidance. For Tier 0 and LSS, the defaults are reasonable.

The following table lists default durability or availability objectives as included with a Hammerspace installation, and the number of instances you should expect with the default volume durability and availability:

Objective Number of Instances

availability-1-nine

1

availability-3-nines

2

availability-5-nines

3

durability-1-nine

1

durability-3-nines

1

durability-5-nines

2

durability-10-nines

4

To get 3 instances using durability, we would need to create an objective with 2 times the volume durability plus 1, which would be 7 nines.

anvil1> objective-create --name durability-7-nines --durability 7 --description 'Make 3 instances when vol dur = 99.9'

ID: f27e26e4-6c3d-40a4-907d-9fb154f35d2b

Name: durability-7-nines

Internal ID: 8

Priority: MEDIUM

Durability: 99.99999%

Description: Make 3 instances when vol dur = 99.9

Assuming we create a directory called /csm3, we can apply the durability-7-nines to that directory:

anvil1> share-objective-add --name test4 --objective durability-7-nines --path /csm3

Share name: test4

Share internal ID: 2

Path: /csm3

Locally applied objectives:

[Name: durability-7-nines, Applicability: TRUE]

All applied objectives:

[Name: delegate-on-open, Applicability: true]

[Name: durability-3-nines, Applicability: DATA_ORIGIN_LOCAL AND IS_DURABLE]

[Name: durability-7-nines, Applicability: TRUE]

[Name: keep-online, Applicability: IS_BEING_CREATED OR HAS_KEEP_ON_THIS_SITE]

[Name: layout-get-on-open, Applicability: IS_BEING_CREATED OR HAS_ONLINE_INSTANCE]

[Name: optimize-for-capacity, Applicability: true]

[Name: availability-1-nine, Applicability: DATA_ORIGIN_LOCAL?ALWAYS]

[Name: durability-1-nine, Applicability: DATA_ORIGIN_LOCAL?ALWAYS]

[Name: keep-online, Applicability: IS_BEING_CREATED OR HAS_ONLINE_INSTANCE AND IS_RECENTLY_USED?ALWAYS]

Active objectives:

[Name: durability-7-nines]

[Name: optimize-for-capacity]

[Name: delegate-on-open]

You can also set objectives on files and directories using HSTK from a client:

# cd csm3

# hs objective add durability-7-nines .

Then you can run the same loop to create files in the /csm3 directory and look at the file locations and distributions.

# cd csm3

# for f in `seq 1 100`; do dd if=/dev/urandom of=f$f count=10 bs=1024k; done

10485760 bytes (10 MB, 10 MiB) copied, 5.95156 s, 1.8 MB/s

10+0 records in

10+0 records out

<snip>

# hs eval -e instances.volume f3

INSTANCES_TABLE{

        |VOLUME = STORAGE_VOLUME('AZ4:node4::/mnt/hsvol1');

        |VOLUME = STORAGE_VOLUME('AZ5:node5::/mnt/hsvol1');

        |VOLUME = STORAGE_VOLUME('AZ6:node6::/mnt/hsvol0')}

[root@peterlearmonth-23360-07102025-client6 csm3]# hs eval -e instances.volume f55

INSTANCES_TABLE{

        |VOLUME = STORAGE_VOLUME('AZ1:node1::/mnt/hsvol1');

        |VOLUME = STORAGE_VOLUME('AZ2:node2::/mnt/hsvol1');

        |VOLUME = STORAGE_VOLUME('AZ6:node6::/mnt/hsvol0')}

[root@peterlearmonth-23360-07102025-client6 csm3]# hs usage volume

SUMS_TABLE{

        |KEY = STORAGE_VOLUME('AZ1:node1::/mnt/hsvol0'),

        |VALUE = 40;

        |KEY = STORAGE_VOLUME('AZ1:node1::/mnt/hsvol1'),

        |VALUE = 40;

        |KEY = STORAGE_VOLUME('AZ2:node2::/mnt/hsvol1'),

        |VALUE = 33;

        |KEY = STORAGE_VOLUME('AZ2:node2::/mnt/hsvol0'),

        |VALUE = 16;

        |KEY = STORAGE_VOLUME('AZ3:node3::/mnt/hsvol0'),

        |VALUE = 22;

        |KEY = STORAGE_VOLUME('AZ3:node3::/mnt/hsvol1'),

        |VALUE = 22;

        |KEY = STORAGE_VOLUME('AZ4:node4::/mnt/hsvol0'),

        |VALUE = 22;

        |KEY = STORAGE_VOLUME('AZ4:node4::/mnt/hsvol1'),

        |VALUE = 22;

        |KEY = STORAGE_VOLUME('AZ5:node5::/mnt/hsvol0'),

        |VALUE = 21;

        |KEY = STORAGE_VOLUME('AZ5:node5::/mnt/hsvol1'),

        |VALUE = 21;

        |KEY = STORAGE_VOLUME('AZ6:node6::/mnt/hsvol1'),

        |VALUE = 20;

        |KEY = STORAGE_VOLUME('AZ6:node6::/mnt/hsvol0'),

        |VALUE = 21}

In the current version of Hammerspace, distribution may not be exactly even, as shown above. This is fixed in the next major release. However, each file has instances in 3 different availability zones.

When you set an objective that requires more than one instance of files, and your Linux client has a recent enough kernel, up to 3 instances (with the current Hammerspace product) will be written by the client. This is referred to as Client Side Mirroring, as defined in RFC 8435. If you set an objective that requires more than 3 instances, the first 3 will be written by the client using CSM, and the rest will be created by a mover (DI).

Keep Instances Where You Want Them

When you have another tier of storage that has higher availability and durability than your Tier 0 or LSS nodes, the system will attempt to satisfy availability and durability objectives by placing fewer instances on a volume or volumes with higher availability and durability. With many Tier 0 use cases, the data should remain on Tier 0 volumes for some period of time. To keep data on specific volumes or a set of volumes, we can use a confine-to objective with a volume group.

First, create the desired volume group. Volume groups can consist of any combination of storage systems, volumes, or even other volume groups. It may make sense to have a volume group for each AZ, and a volume group that includes all AZ volume groups.

anvil1> volume-group-create --name AZ1 --expressions 'node:AZ1:node101,node:AZ1:node102,node:AZ1:node103,node:AZ1:node104,node:AZ1:node105,node:AZ1:node106'

anvil1> volume-group-create --name AZ2 --expressions 'node:AZ2:node201,node:AZ2:node202,node:AZ2:node203,node:AZ2:node204,node:AZ2:node205,node:AZ2:node206'

anvil1> volume-group-create --name AZ3 --expressions 'node:AZ3:node301,node:AZ3:node302,node:AZ3:node303,node:AZ3:node304,node:AZ3:node305,node:AZ3:node306'

anvil1> volume-group-create --name AZ4 --expressions 'node:AZ4:node401,node:AZ4:node402,node:AZ4:node403,node:AZ4:node404,node:AZ4:node405,node:AZ4:node406'

anvil1> volume-group-create --name AZ5 --expressions 'node:AZ5:node501,node:AZ5:node502,node:AZ5:node503,node:AZ5:node504,node:AZ5:node505,node:AZ5:node506'

anvil1> volume-group-create --name AZ6 --expressions 'node:AZ6:node601,node:AZ6:node602,node:AZ6:node603,node:AZ6:node604,node:AZ6:node605,node:AZ6:node606'

anvil1> volume-group-create --name AZ_all --expressions 'volume-group:AZ1,volume-group:AZ2,volume-group:AZ3,volume-group:AZ4,volume-group:AZ5,volume-group:AZ6'

When you create a volume group, the system automatically creates a set of objectives for that volume group including place-on, exclude-from, and confine-to. Now you can apply the confine-to for AZ_all to the data that needs to stay on the Tier 0 volumes. You can even include an expression to define an "applicability" such as how long the data should stay on Tier 0.

anvil1> share-objective-add --name test4 --path /csm3 --objective confine-to-AZ_all --applicability 'LAST_USE_AGE<1*HOURS?TRUE'

Configure Tier 0 Nodes to Write Local

Future versions of Hammerspace are planned to have additional functionality to simplify node affinity for use cases like checkpointing. Currently, it is possible to use node objectives to direct I/O for a given directory to storage volumes on that node.

Verify that the job manager can instruct Tier 0 nodes to write checkpoints to a specific directory within the share.

Create per Tier 0 node directories. From a client with the necessary permissions, use the mkdir command:

# mkdir /mnt/checkpoints/node001

Add a place-on-node objective to the directory:

anvil1> share-objective-add --name checkpoints --objective place-on-node001 --path /node001

To protect checkpoints after the initial fast write to local storage, you need an objective to place instances of the checkpoint on other storage, either other Tier 0 nodes, on one or more Linux Storage Servers, or other robust storage such as a NAS or cloud. You could require multiple instances by specifying a higher availability or durability than one node provides, which results in multiple instances of files. Basically, take the number of nines specified in the objective and divide by the number of nines Tier 0 volumes provide and that gives the number of instances needed. If there is a more robust storage system (another tier), a place-on for that system would create an instance of each file in that system. With either of these, a brief time condition is needed to ensure the system doesn’t try to immediately write first instances to non-local storage.

anvil1> hscli objective-create --name place-on-BigStore1 --place-on-list node:BigStore1

ID: d170c879-5c24-45d5-8de4-ba94a829470e

Name: place-on-BigStore1

Internal ID: 5

Priority: MEDIUM

Place on:

[node: BigStore1]

anvil1> hscli objective-create --name tier0_to_tier1 --applied-objective "d170c879-5c24-45d5-8de4-ba94a829470e,MODIFY_AGE>5*MINUTES"

ID: 50b0db52-85f6-42d3-ab8e-ef08bd49ea26

Name: tier0_to_tier1

Internal ID: 6

Applied objectives:

[ID: 1, Objective ID: d170c879-5c24-45d5-8de4-ba94a829470e, Objective name: place-on-BigStore1, Applicability: MODIFY_AGE>5*MINUTES]