Search the docs

Upgrading a Cluster That Uses NVMe-oF Storage

If any DSX node in the cluster uses NVMe-oF storage, follow this guidance in addition to the software update procedure in Upgrading Hammerspace in the Hammerspace Installation and Licensing Guide.

Before You Upgrade

  1. Make sure that every NVMe-oF path is healthy. If the cluster runs 5.2.14 or later, list the NVMe-oF path events that have not been cleared. In the Management GUI, open Events, select Uncleared events only, and look in the Action column for NVMe-oF path degraded and NVMe-oF path lost. From the Admin CLI, run event-list --type NVMEOF_PATH_DEGRADED --uncleared, and then the same command with --type NVMEOF_PATH_LOST. Do not start the upgrade while either event is active; restore the path first. Earlier builds do not raise these events, so ask Hammerspace Support to check path health before you upgrade.

    Do not use the State shown by nvmeof-config --list --full for this check. It shows CONNECTED for a path that is down. See NVMe-oF Path Health and Recovery.

  2. Upgrade to 5.2.14P3 or later. These builds include fixes for clusters that use NVMe-oF storage. Upgrade directly to the build you choose; there is no benefit in stopping at intermediate builds.

  3. Plan to have out-of-band access to every node during the upgrade, for example through the server’s BMC or through vCenter for a virtual machine. A DSX node that uses NVMe-oF storage can stop responding while it restarts at the end of the upgrade, and then it needs a reset from outside the operating system.

  4. Do not change the NVMe-oF configuration while the upgrade is running. That includes adding or removing enclosures, connecting or disconnecting subsystems, and changing connection settings. Some nodes run the new release and some the old one until the upgrade finishes.

After You Upgrade

  1. Check that every volume on NVMe-oF storage is online, and that no NVMEOF_PATH_DEGRADED or NVMEOF_PATH_LOST event is active for a path you expect to work.

  2. If a volume on NVMe-oF storage is missing or offline, contact Hammerspace Support.

    Never re-create a missing volume to recover it. Without Force Initialize (logical-volume-create --force in the Admin CLI), the operation fails because the device already holds a file system; with it, the operation erases the data on the device. Contact Hammerspace Support instead.
  1. Check the controller loss timeout of each enclosure. Enclosures keep the value they had before the upgrade. See Controller Loss Timeout.