Search the docs

NFS

Hammerspace provides data access over NFS to a broad set of client operating systems, including Linux, Windows, UNIX, and macOS.

NFSv4.2 has only been tested with Linux clients. More information about which distributions are tested is available in the Hammerspace 5.2 Hardware and Software Compatibility List.

NFSv4 Client Unique-Id Requirement

The NFSv4 protocol (applies to both 4.1 and 4.2) requires each client to have a unique ID. By default, the Linux NFS client uses an ID string that contains the local node name. Therefore, if all node names are unique, no further action is required. Failure to use unique IDs on nodes will result in unreliable NFSv4 operations.

If node names are not unique, or if clients were cloned from the same image, assign each client its own NFSv4 ID:

  1. Write a unique ID to a file in /etc/modprobe.d/. The following command generates a new UUID for this client:

    # echo "options nfs nfs4_unique_id=$(uuidgen)" > /etc/modprobe.d/nfs4-unique-id.conf
    Every client needs a different value. Run the command on each client rather than copying the file between clients, and do not reuse the file from a cloned image.

  2. Load the new ID. Either reboot the client, or unmount all NFS file systems, reload the nfs kernel module, and remount. The following command assumes that all current NFS mounts are defined in /etc/fstab:

    # umount -a -t nfs; modprobe -r nfs; mount -a -t nfs

  3. Optional: confirm that the client is using the new ID:

    # cat /sys/module/nfs/parameters/nfs4_unique_id

For more details, refer to https://www.kernel.org/doc/Documentation/filesystems/nfs/nfs.txt.

NFS Version-Specific Mount Examples

If you are not familiar with NFS mount options, read the NFS man page for the client operating system. Here are a few examples that will work for most Linux clients:

NFSv4.2 example:

# mount -t nfs -o vers=4.2 <DSX DATA IP>:<EXPORT PATH> <MOUNT POINT>

NFSv4.1 example:

# mount -t nfs -o vers=4.1,nolock <DSX DATA IP>:<EXPORT PATH> <MOUNT POINT>

NFSv3 example:

# mount -t nfs -o vers=3,nolock <DSX DATA IP>:<EXPORT PATH> <MOUNT POINT>

Supported NFS Protocols

NFS version Status

Version 3

Supported

Version 4.1

Supported
[IMPORTANT] When NFS 4.1 support is enabled, NFS 4.2 clients must mount the shares from the Anvil and not the DSX nodes.

Version 4.2

  • Supported by qualified clients. See the Hammerspace Hardware and Software Compatibility List for specifics. If a client tries to access and it is not on the qualified list, it will transparently be re-directed to NFSv3 protocol version, and the mount will succeed.

  • NFS 4.2 clients must have network access to all storage volumes where data may be stored.

  • For RHEL 7.x & CentOS 7.x clients, make sure the client flexfiles timeout is adjusted due to an incorrect default value.

  • Add the following single line to /etc/modprobe.d/hs_flex.conf and reboot the client:

    options nfs_layout_flexfiles dataserver_retrans=0 dataserver_timeo=600

  • Mount kernel check bypass option:

    For NFS clients that cannot mount due to the kernel output not matching the approved list, there is a way to bypass the strict kernel check. It is recommended to only use this option when the kernel is 5.15 or later. Note that the bypass option is only supported when mounting against the Anvil IP. Example:

    mount -t nfs -o vers=4.2,port=20492 <ANVIL IP>:<EXPORT PATH> <MOUNT POINT>

NFSv4.1 ACL Passthrough on Portal Mounts

On an NFS 4.1 client mounted against a Hammerspace portal, you can read and write the full NFSv4 ACL of a file or directory using the standard nfs4_getfacl and nfs4_setfacl commands (from the nfs4-acl-tools package).

nfs4_setfacl --dacl -a A::4001:RWX /mnt/share41/test
nfs4_getfacl --dacl /mnt/share41/test

These commands default to the older NFSv4.0 ACL attribute. To work with the full NFSv4.1 ACL on a portal mount, pass --dacl (discretionary ACL, for access) or --sacl (system ACL, for auditing).

Either consistently use the --dacl/--sacl flags or consistently never use them. Mixing the two modes produces inconsistent results, because the default operates on the NFSv4.0 ACL attribute while --dacl/--sacl operate on the separate NFSv4.1 attributes. (The getrichacl and setrichacl tools default to NFSv4.1 and do not need the equivalent flags.)

Enterprise Linux 8 clients (for example, RHEL 8 and CentOS 8) lack the separate NFSv4.1 ACL extended attributes (system.nfs4_dacl and system.nfs4_sacl); on these clients, --dacl/--sacl fail with Failed to instantiate ACL, and you can manage only NFSv4.0 ACLs. Use a client distribution qualified for Hammerspace 5.3 (see the Hardware and Software Compatibility List) for full NFSv4.1 ACL support.

Enabling NFS LOCALIO on Linux Clients

You can enable the LOCALIO auxiliary RPC protocol on RHEL 10 or on any other Linux kernel that has LOCALIO support enabled. To do this, add a file to /etc/modprobe.d/, for example /etc/modprobe.d/localio.conf, that contains the following line:

Enabling LOCALIO in localio.conf
options nfs localio_enabled=Y

Alternatively, add nfs.localio_enabled=Y to the kernel command line. For example, on a system that uses grubby:

Command:

# grubby --args=nfs.localio_enabled=Y --update-kernel=ALL

Whenever a client connects over NFSv4.2 to protod, and protod does not have a cached entry for it, protod will initiate an EXCHANGE_ID request to a potential server on the client. If the client accepts the request and replies, it will send back the so_major_id in the reply.

The following example shows the server owners that protod has recorded:

Example server owner list

Command:

$ hs-server-owner-list

Expected output:

Volume ID    Client ID    Owner ID
337          6            "dsx1.example.com"
395          9            "dsx2.example.com"
421          11           "dsx3.example.com"
477          18           "dsx5.example.com"
482          138          "dsx4.example.com"
502          20           "dsx6.example.com"

Each of the following clients would be considered to have local I/O to dsx6.example.com:

Example showing clients with LOCALIO enabled

Command:

$ hs-client-owner-list | grep dsx6

Expected output:

165          "Linux NFSv4.2 21d1ba73-b292-48b8-bfbf-e37ead97eb8d/dsx6.example.com"
140          "Linux NFSv4.2 cc8011c0-ab65-4df1-8576-8ab97291ea01/dsx6.example.com"
19           "Linux NFSv4.2 dsx6.example.com"
166          "Linux NFSv4.2 21d1ba73-b292-48b8-bfbf-e37ead97eb8d/dsx6.example.com"
139          "Linux NFSv4.2 cc8011c0-ab65-4df1-8576-8ab97291ea01/dsx6.example.com"
20           "Linux NFSv4.2 dsx6.example.com"
Not shown in the example above is that 19 and 20 probably correspond to the port 20491 and 20492 connections for case-sensitive and case-insensitive mounts.

Then, for each client that connects, protod scans the list of server owners, and if the so_major_id is a substring at the end of the co_ownerid, the client is considered to have localized storage.

The following sections describe how to set up the NFSv4 server and client identifiers on a Tier 0 architecture so that protod can discover the co-location of the client and data server.

To configure the NFSv4 client owner on Linux, refer to the following article: https://docs.kernel.org/filesystems/nfs/client-identifier.html.

Based on the Linux kernel documentation and implementation details, the Linux NFSv4 server likely constructs the server owner name (also called "server scope") using the system’s hostname by default.