Remounting an Enterprise Linux 10.1 (or Later) Client

These instructions apply to Red Hat Enterprise Linux 10.1, Rocky Linux 10.1, and compatible distributions.

On Enterprise Linux 10.1 (kernel 6.12.0-124), a mount that uses xprtsec=mtls succeeds and directory listings work, but every file read and write hangs. The client’s pNFS data connections to the DSX nodes (port 3049) do not use TLS, and the DSX nodes refuse them. Until you run a kernel that fixes this, disable the client’s pNFS flexible-files layout driver before you remount, as described under Prerequisites on this page. File I/O then goes through the TLS connection to the Anvil.

Prerequisites

Verify the following before remounting:

The nfs-utils and ktls-utils packages are installed.

The tlshd service is enabled and running:

sudo systemctl enable --now tlshd.service

The Hammerspace cluster CA certificate is trusted by the client. This is the issuer of every Anvil and DSX certificate (Administration TLS Cluster Root CA Download Certificate); the client needs it whichever CA issued the client’s own certificate:
```
sudo trust anchor /path/to/hammerspace-ca.crt
```

The client has a valid X.509 certificate and private key, and tlshd is configured to use them. If the client certificate was issued by an intermediate CA, put the intermediate CA certificate after the client certificate in the x509.certificate file, or add the intermediate CA to the Hammerspace trust store with cert-add --trust-intermediate-ca. The tlshd configuration file at /etc/tlshd.conf must reference the correct certificate paths:

[authenticate.client]
x509.truststore= /etc/pki/tls/certs/ca-bundle.crt
x509.certificate= /etc/pki/hs-tls/nfs-pki/cert.pem
x509.private_key= /etc/pki/hs-tls/nfs-pki/key.pem

In ktls-utils 1.3.0 and later, the default configuration file is /etc/tlshd/config. If /etc/tlshd.conf exists, tlshd reads it instead. Edit whichever file your distribution uses, and restart tlshd afterward.

On kernel 6.12.0-124 (Enterprise Linux 10.1), the pNFS flexible-files layout driver is disabled, so that file I/O uses the TLS connection to the Anvil instead of unencrypted connections to the DSX nodes. Create a file named /etc/modprobe.d/hammerspace-no-pnfs.conf that contains these two lines, then reboot the client:
```
blacklist nfs_layout_flexfiles
install nfs_layout_flexfiles /bin/false
```
After you remount, grep pnfs= /proc/self/mountstats shows pnfs=not configured for each Hammerspace mount. Reboot rather than unloading the module: a client that already hung on file I/O keeps the module in use until it restarts.

Procedure

If the share is currently mounted, stop any processes using it, then unmount:
```
sudo umount /mnt/your_mount_point
```
If the unmount fails because the mount point is busy, use fuser -m /mnt/your_mount_point to identify and stop the active processes before retrying.
Remount the share with the xprtsec=mtls option to require mutual TLS authentication:
```
sudo mount -o xprtsec=mtls <cluster_ip>:/<export_path> /mnt/your_mount_point
```
To ensure the share remounts automatically with TLS after a system reboot, add or update the entry in /etc/fstab:
```
<cluster_ip>:/<export_path>  /mnt/your_mount_point  nfs  xprtsec=mtls,_netdev  0  0
```
- xprtsec=mtls — Requires mutual TLS authentication for the NFS connection.
- _netdev — Ensures the network is available before the system attempts to mount the share at boot time.

Verify the TLS Connection

After remounting, verify that the connection is using TLS:

Check the connection status. Connections to the Anvil use port 2049. With a pNFS mount (NFSv4.2), data connections go directly to DSX nodes on port 3049. Check both:
```
sudo ss -tni | grep -E -A1 ':(2049|3049)\b'
```
A connection that uses TLS shows tcp-ulp-tls in its details, for example tcp-ulp-tls version: 1.3 cipher: aes-gcm-256. Run the command as root: on some kernels the TLS details are omitted for other users.

Data connections to port 3049 open when files are read or written, so run the check during file I/O. On kernel 6.12, ss shows the TLS details to non-root users as well. If a port 3049 connection does not show tcp-ulp-tls, or file I/O hangs, see Troubleshoot TLS Issues. A client whose pNFS layout driver is disabled (see Prerequisites on this page) has no port 3049 connections: all traffic uses the TLS connection on port 2049.
Check the NFS mount statistics. Confirm that xprtsec=mtls appears in the reported mount options:
```
nfsstat -m
```