Appendix 3 - Complex NUMA Systems

Some server systems have NUMA architectures that require careful configuration and device usage. Dual-socket AMD-based systems are the main example.

This AMD document describes the processor and supporting component architecture, along with guidance on BIOS settings for various workloads. One of the key parameters is NUMA Nodes per Socket (NPS), which divides the resources associated with a processor into NUMA domains. In practice, for GPU nodes this is often dictated by Nvidia or other GPU vendors, commonly setting NPS = 4. On a dual socket system, this sets up 8 NUMA domains.

The following simplified diagram illustrates the two processor sockets, the External Global Memory Interconnects (xGMI) that connect the NUMA domains, and a set of I/O devices hanging from them.

The result is that I/O between devices on different NUMA domains, especially between sockets, can incur a significant latency penalty. This has been measured at customer sites, with up to 50% impact on throughput between devices on different NUMA domains.

There are several commands that can be used to determine the NUMA architecture of a system and attached or installed devices. Some commands yield different levels of detail or even what devices are included, depending on the platform. Commands like dmidecode, lspci (with -vvv or -tv options), and lstopo (if installed) can yield system architecture. The files under /sys/class are dynamically generated and are a consistent way to determine the NUMA configuration we need. The following grep commands show all the devices and their NUMA node in one concise output.

Note that depending on how NVME devices enumerate, the NUMA nodes may not match the above output. The lower numbered half of the NUMA nodes are associated with the first CPU socket, NUMA 1 in the diagram above, and in AMD’s documentation, while the higher numbered half of the NUMA nodes are associated with the second CPU socket and NUMA 2.