Search the docs

Connecting Storage to a DSX Using NVMe-oF

Hammerspace supports NVMe over Fabric (NVMe-oF), enabling NVMe drives to be added to a DSX node. This extends the storage capacity of the DSX node by providing a managed, reboot-persistent way to mount and incorporate networked NVMe block devices from external arrays into the cluster’s usable volume pool.

  • Configuration and management:

    • Configuration is performed on a per-DSX node basis. Multiple NVMe-oF enclosures (targets) can be added to a single DSX node.

    • The Admin CLI command nvmeof-config handles all NVMe-oF operations, including adding enclosures, connecting subsystems, and changing each enclosure’s connection settings.

    • The system uses NVMe Qualified Names (NQNs) for identification. Each DSX node has a host NQN that identifies it to the storage array, and each subsystem (which contains namespaces/drives) has its own NQN. The host NQN is an identifier, not a credential. Enter the DSX host NQN into the storage array’s host access configuration so that the array presents only the intended namespaces to that DSX.

    • The configuration workflow involves discovering targets and then connecting to discovered namespaces to expose them as block devices.

  • Logical volume creation: Once the external storage devices (namespaces) are connected and discovered by the DSX, administrators can create logical volumes on them (i.e., format them with a file system) and then add them to the Hammerspace cluster as regular volumes.

  • Multipathing and redundancy:

    • The system supports multipathing, which is automatically enabled if multiple IP address targets are provided for a single NVMe enclosure. Provide every target address when you add the enclosure; a target is identified by its address, and a target added to an enclosure after its subsystems are connected does not become a path to them.

    • Multipathing enhances redundancy (if one IP network fails, the system can use the other) and helps distribute load across multiple paths.

    • Configurable parameters include the I/O policy for multipathing. The supported policies are queue-depth (the default), round-robin, and numa.

  • Configuration persistence and boot ordering: Connections to NVMe-oF namespaces persist across reboots. On each boot, Hammerspace reconnects the configured subsystems after the network is up and before file systems are mounted, so the block devices are present when the mounts occur. No additional configuration is required.

  • Health monitoring: A DSX raises two NVMe-oF events to System Manager, both per path. NVMe-oF path degraded is raised at Warning severity when a path has been stuck trying to reconnect. NVMe-oF path lost is raised at Warning severity while the subsystem still has at least one working path, and at Critical severity once the subsystem has no working paths left. Both clear automatically when the path recovers, and both are debounced, so a brief interruption does not raise an event. By default, reconnection attempts are unlimited. For the event names, what each severity means, and the matching SNMP traps, see NVMe-oF Path Health and Recovery.

  • Use the nvmeof-config command to view configured enclosures and the state of their connections. When given no subcommand, it defaults to listing.

  • When connecting a subsystem within an enclosure, all of the namespaces within that subsystem are connected.

  • The same is true for disconnecting from a subsystem; all namespaces of that subsystem are also disconnected from the DSX. However, even when a subsystem is disconnected, the namespaces remain visible in Hammerspace inventory until the Hammerspace administrator reconciles the inventory.

To remove disconnected namespaces from the inventory, refresh the DSX node:

Command:

node-refresh --name <dsx-node-name> --reconcile-components

This section includes the following subsections: