Appendix 1 - Installing Data Movers on Linux Nodes

There are two components that are collectively referred to as data movers:

Mover, sometimes referred to as NFS mover, and internally referred to as Data Instantiator or DI. This mover moves or copies instances of files between local storage volumes via NFS. This could be between DSX nodes, to/from a Linux storage server or other third part storage system.
Cloud Mover or CM. CM moves or copies files to and from object storage (local or remote) and cloud storage using S3, HTTPS and proprietary cloud protocols. CM also manages chunking, hash values, deduplication, and encryption for data placed in object storage.

There are three ways to implement the Hammerspace data mover technology:

Deploy DSX nodes, which include data movers as part of their included functionality.
Deploy the data mover RPMs on a supported Linux distribution.
Deploy the data movers as containers

While it is supported to install DI and CM on GPU nodes, customers may choose to install on separate servers to avoid additional resource consumption on the CPUs of GPU nodes.

Supported Distributions / Platforms

DI as an RPM is available for el8 and el9 platforms.

Other x86 based Linux distributions must use container based deployments.

Dependencies

Firewall ports 9095 and 9096 open for inbound from all Anvil IPs.
epel-release
platform-python
lttng-tools
lttng-ust
jemalloc
Babeltrace

Download and installation instructions for the above packages are below.

Installation, Setup, and Configuration of the DI Using RPM

Install Extra Packages for Enterprise Linux and other dependencies:

# dnf install -y epel-release

# dnf install -y platform-python lttng-tools lttng-ust jemalloc

Download and install babeltrace:

# wget https://dl.rockylinux.org/pub/rocky/9/devel/x86_64/os/Packages/b/babeltrace-1.5.8-10.el9.x86_64.rpm

# dnf install -y ./babeltrace-1.5.8-10.el9.x86_64.rpm

Get the Hammerspace Data Services Components Kit

Download the Hammerspace Data Services Components appropriate to your version of Hammerspace, and the CPU architecture and Linux distribution of the servers that will run the DI. If the version you need is not listed, contact your Hammerspace tech team.

For some Hammerspace releases, all of the components are included in a single tarball. Download and extract tarball, then extract and install the DI. File names will match the version of the components.

# wget trans.doit.hammerspace.com/download/tier0/components/el9-components-5.1.41-452.tar.gz

# gunzip el9-components-5.1.41-452.tar.gz

# tar xvf el9-components-5.1.41-452.tar */pd-di-*

el9-components/pd-di-5.1.41-452.el9.x86_64.rpm

# dnf install -y el9-components/pd-di-5.1.41-452.el9.x86_64.rpm

Add the management IP address of the Hammerspace cluster to /etc/hosts. Be sure to use the correct management floating IP for your Hammerspace cluster.

# echo "10.1.2.3 data-cluster" >>/etc/hosts

Open up the firewall ports Anvil uses to talk to the DI:

# firewall-cmd --zone=public --add-port=9095/tcp --permanent; firewall-cmd --zone=public --add-port=9096/tcp --permanent

# firewall-cmd --reload

Check firewall settings with firewall-cmd --list-all. You should see "ports: 9095/tcp 9096/tcp".

# firewall-cmd --list-all

public (active)

target: default

icmp-block-inversion: no

interfaces: ens192

sources:

services: cockpit dhcpv6-client mountd nfs rpc-bind ssh

ports: 9095/tcp 9096/tcp

protocols:

forward: yes

masquerade: no

forward-ports:

source-ports:

icmp-blocks:

rich rules:

Disable SELinux.

# vi /etc/selinux/config

# Set SELINUX to disabled and reboot.

SELINUX=disabled

Enable Linux Trace Toolkit: next generation (LTTNG) services:

# systemctl enable --now lttng-sessiond.service

# systemctl enable --now pd-di-lttng-recorder.service

Install Python PIP and other dependencies needed by the script to add the DI to the Hammerspace cluster:

# dnf install -y pip

# pip3 install urllib3

# pip3 install requests

Download and run the add_node.py script to add the DI to the cluster:

# ./add_node.py -i 10.4.5.6 -l 20 -d -u admin -n di-6

In the above command, the following options are used:

-i IP address of the DI node being added

-l Netmask prefix length of the DI node being added

-d A DI is being added (as opposed to a cloud mover)

-u Administrative user name on the Hammerspace cluster

-n Node name of the DI as you wish it to appear in Hammerspace

You can also specify admin user password with the -p option, but if not, the script will prompt.

Enable and start the DI service:

systemctl enable --now pd-di.service

You can verify the DI is properly configured and added to Hammerspace by logging in to the Hammerspace CLI as admin (or equivalent user) and running the node-list command specifying the DI node name.

admin@anvil> node-list --name my_di1

Name: my_di1

Type: External Mover

Internal ID: 1073744505

ID: bae33620-0281-11ef-8cab-5254009a4b8c

HW state: OK

Node state: MANAGED

Management IP: 10.4.5.6/20

S3 auth signing type: Default

Created: 2024-06-05 15:45:23 UTC

Modified: 2024-06-05 15:45:33 UTC

System services:

Object type: Data Mover

ID: 254e163c-1767-4b02-87b4-2f9c9c688b26

Internal ID: 536873594

Admin state: Up

Oper state: Up

Port: 9095

Troubleshooting

The following node-list command shows the status of the DI from the Anvil perspective. In this example, there is a problem with the firewall on the node running the DI.

admin@anvil> node-list --name di-173

Name: di-173

Type: External Mover

Internal ID: 1073741834

ID: 594662a3-e045-507b-a609-0d2cb6f16beb

HW state: OK

Node state: MANAGED

Management IP: 10.200.100.173/64

S3 auth signing type: Default

Created: 2025-06-03 23:59:31 UTC

Modified: 2025-06-05 05:29:37 UTC

System services:

Object type: Data Mover

ID: 553b23ef-0709-567b-80f0-2be3fe68a852

Internal ID: 536870923

Admin state: Up

Oper state: Down

Oper state reason: Failed port tests: [10.200.100.173:9095(Connection refused), 10.200.100.173:9096(Connection refused)]

Port: 9095

The above error can also be caused by the pd-di service not installed, correctly configured, or started. You might also see:

Oper state reason: Failed port tests: [10.200.101.3:9095(No route to host), 10.200.101.3:9096(No route to host)]

When you fix firewall and pd-di service issues, it may take a few seconds for Anvil to recognize the change.