External Movers (DI and Cloud Mover Containers)

External movers — the Data Instantiator (pd-di) and Cloud Mover containers — move data between Hammerspace and external storage. To participate in a TLS-required cluster, an external mover must present its own PKI material.

External movers are handled differently from Anvil and DSX nodes. The cluster does not issue them certificates, does not monitor their PKI health, and does not restart them when TLS is enabled on the cluster. Each of those is an administrator responsibility. :leveloffset: +1

Supplying PKI Material to External Movers

The cluster issues certificates only to Anvil and DSX nodes. Nothing is distributed to a mover container.

Supply the mover’s certificate and key yourself. The certificate must be signed by a certificate authority the cluster trusts, uploaded to the cluster as a trusted CA, and must carry the mover’s IP addresses as subject alternative names.

External movers receive no PKI health checking and raise no PKI_* events, so a mover with missing or invalid PKI material is not reported by cluster monitoring.

The Cloud Mover checks that the certificate presented by each Hammerspace node or storage server it connects to includes the IP address it connects to as a subject alternative name. Certificates that Hammerspace issues to Anvil and DSX nodes include their IP addresses. For third-party storage, make sure the storage server’s certificate does too.

Creating the tlshd Configuration File

pd-di and the Cloud Mover do not use tlshd. They perform their own TLS handshakes and read /etc/tlshd.conf only to locate the certificate, key, and trust store. Create the file whether or not tlshd is installed in the container. tlshd matters only if something in the container performs kernel NFS mounts.

Create /etc/tlshd.conf with the following two sections — one for the container acting as a TLS client, one for it acting as a TLS server — and point each at the mover’s PKI files:

[authenticate.client]
x509.truststore = /etc/pki/tls/certs/ca-bundle.crt
x509.certificate = /etc/pki/hs-tls/nfs-pki/cert.pem
x509.private_key = /etc/pki/hs-tls/nfs-pki/key.pem

[authenticate.server]
x509.truststore = /etc/pki/tls/certs/ca-bundle.crt
x509.certificate = /etc/pki/hs-tls/nfs-pki/cert.pem
x509.private_key = /etc/pki/hs-tls/nfs-pki/key.pem

Set all three keys in both sections. pd-di does not report a key that is missing or misspelled. It treats the key as not configured and still logs that it is TLS-capable, but its PKI material is incomplete, and TLS connections that need the missing file fail.

The private key must be an unencrypted PEM file; pd-di cannot read an encrypted key. Restrict access to the key file, for example with mode 600, as Hammerspace does on its own nodes.

Adding the Required Trust Anchors

The container must trust the Hammerspace root CA, and the certificate authority of any third-party storage it moves data to or from.

Copy /etc/pki/hs-tls/cacerts/hs-root-cacert.pem from any Anvil node, together with the CA certificates of any third-party storage involved, into /etc/pki/ca-trust/source/anchors/ inside the container.
Run update-ca-trust extract.

Pointing x509.truststore at a PEM file is not sufficient. The Cloud Mover ignores x509.truststore and reads the Java trust store that update-ca-trust extract generates.

Restarting an External Mover

pd-di reads /etc/tlshd.conf and checks the PKI files only at startup. If they are missing or invalid when it starts, the mover remains TLS-incapable until it is restarted. It logs DI is TLS-incapable (PKI material unavailable) and refuses mobilities to TLS-required storage, which DME then reschedules to another DI.

Where possible, put the PKI files in place and restart the mover container before enabling TLS on the cluster.

On DSX nodes the cluster restarts pd-di when TLS is enabled. Nothing does this for an external mover, so the restart is manual.

Restart pd-di again after renewing its certificate or adding a trusted CA, because its outbound TLS context is loaded once at startup.