Node Software Stack: Drivers, CUDA/ROCm, NCCL & Firmware

Above the rack and below the scheduler sits a layer that almost no slide deck shows and that many multi-week training failures trace back to: the node software stack. It is the column of software that turns a powered, networked GPU server into a node a distributed job can use. That column holds the kernel driver that talks to the silicon, the CUDA or ROCm runtime the framework links against, the collective-communication library (NCCL on NVIDIA, RCCL on AMD) that carries every all-reduce, and the firmware beneath all of it (GPU VBIOS, NIC, BMC, NVSwitch, CX/Thor, PSU). These are not independent knobs. They are a single coupled organism with a compatibility matrix, and the matrix is unforgiving at scale. A synchronous training job runs at the speed of its slowest, most-skewed node, and an unsupported or unvalidated library and firmware combination can fail, hang, corrupt output, or silently regress collective bandwidth across every participating GPU.

Most operators treat this layer as plumbing until it costs them a run. This chapter covers the vertical stack and driver lifecycle, the injection model (GPU Operator vs golden image), the container-runtime layer, the AMD ROCm path as a real but taxed second source, NCCL/RCCL tuning, stack pools that each run one tested tuple, and firmware update mechanics under attestation. The trade is the same at every layer: a tested, attested tuple buys goodput, separate pools keep room to move, and choosing wrong shows up not as a config error but as a stalled $50M run.

Read the node stack bottom to top and the coupling shows. At the base is firmware: GPU VBIOS/InfoROM, NVSwitch/UALink-switch firmware, NIC firmware (NVIDIA ConnectX/BlueField, Broadcom Thor, AMD Pollara/Pensando), BMC, BIOS/UEFI, PSU and CDU controllers. Each has its own version, its own update path and its own dependency on the layers above. On top sits the kernel driver (the NVIDIA data-center driver branch, or AMDGPU/amdkfd for ROCm), which must match the kernel ABI and the GPU's firmware. The driver exposes a user-space runtime: the CUDA driver API and toolkit, or the ROCm runtime (HIP, rocBLAS, MIOpen). As a dated example, DGX OS 8.0.0 lists CUDA 13.3 Update 1 and NCCL 2.31.2 (31 August 2026). As of October 2026 newer releases exist, including CUDA 13.4 and NCCL 2.32. Above the runtime are the math and communication libraries (cuDNN, cuBLAS, NCCL on NVIDIA; MIOpen, rocBLAS, RCCL on AMD) and finally the framework (PyTorch, JAX, vLLM) the user writes against.

Two facts catch golden-image builders on current NVIDIA hardware. On Grace Hopper and Blackwell the kernel driver must be NVIDIA's open GPU kernel modules. NVIDIA does not support its proprietary modules on those platforms (NVIDIA, July 2024), so build every B200, GB200 and GB300 image on the open flavor. And on a GB200 or GB300 NVL72 rack the tuple reaches past the node. The IMEX service, which lets GPUs in the NVLink domain address each other's memory, is installed per driver branch and stops at startup if the loaded driver is incompatible. The switch trays run their own NVOS image, whose NMX-Controller carries the NVLink fabric manager and subnet manager. Pin and test the switch-tray software and firmware with the compute-tray driver, not as a separate estate; rack-scale placement over IMEX is in Chapter 10.2.

The constraints run both ways. A given CUDA toolkit requires a minimum driver. A given NCCL build is compiled against a CUDA version. A given firmware revision is validated against a driver branch, and the framework pins the cuDNN and NCCL it was built and tested with. NVIDIA's CUDA minor-version compatibility and forward compatibility loosen this slightly, and they are two different mechanisms. Minor-version compatibility lets a qualifying application built with a newer toolkit in the same major family run against that family's documented minimum driver, subject to feature and PTX limits and with no compat package involved. Forward compatibility is a separate user-mode cuda-compat package carrying its own driver and hardware matrix. Both are escape hatches for a validated tuple, not a license to let the fleet drift. A production cluster does the opposite: it admits only a tested tuple at each scheduling boundary (the rule below), and treats undeclared drift as a defect. The DGX OS 8.0.0 example above uses driver 595.91.07 and lists ARM64 DGX support as a future release, so do not transfer its tuple to a GB200 image or mix it with an arbitrary R580 patch.

NVIDIA ships data-center drivers on two cadences, and choosing between them is the first decision. Production Branches release about twice yearly, and each receives fixes for up to one year. Long-Term Support Branches (LTSB) get a three-year lifecycle that prioritizes stability over features. As of October 2026 the R580 branch is the current LTSB, with support running to June 2028 and CUDA 13.x support subject to the specific driver and runtime compatibility limits. A fleet whose hardware and required features fit the LTSB can pin a tested patch and upgrade on a deliberate cadence. Mixed fleets carry one more constraint: CUDA 13.0 dropped library and offline-compilation support for Maxwell, Pascal and Volta, so a pool that keeps V100s stays on a CUDA 12.x toolchain while newer pools move on. Its driver is capped as well: NVIDIA's deprecation schedule makes R580 the final branch for those three architectures, so a V100 pool can go no further than the R580 LTSB, whose support ends in June 2028. A fleet chasing the newest accelerator generation or a kernel feature may have to ride a Production Branch and accept the shorter support window. The cost of choosing wrong is lopsided: ride too new a branch and you inherit regressions on hardware you cannot afford to debug under a live run; ride too old a branch and you cannot enable the next density generation when the ramp arrives.

The second decision is how the driver lands on the node. Two models dominate. They give installation and drift control to different owners, and both need the same tuple tests and recovery plan.

The GPU Operator (NVIDIA's implementation) is the Kubernetes-native answer. It deploys and manages the containerized driver, the container toolkit, the Kubernetes device plugin, DCGM/DCGM-exporter, the MIG manager and node-feature discovery as one coordinated set, and it rolls driver upgrades through the fleet by cordoning and draining a node before swapping its stack. It is a candidate tool for a cloud-native, inference-heavy or heterogeneous fleet where you want to roll a driver across thousands of nodes without re-imaging. The cost is more moving parts in the critical path and a dependency on the container runtime being wired correctly. The NVIDIA Container Toolkit must inject /dev/nvidia*, libraries and the right capabilities into the container, and the AMD side uses equivalent CDI-based device injection.

For change-controlled Slurm/HPC training fleets the alternative is a golden OS image with the driver and toolkit baked in. You attest it at boot, and every node in a pool runs the same signed, byte-identical image. Another pool may run a different image, and the scheduler keeps a job from mixing the two unless that combination has been tested. Many large operators run a hybrid: a host-baked driver for a repeatable host artifact, and Operator-managed plugins, DCGM and MIG for the layers that benefit from declarative lifecycle. AMD offers its own GPU Operator and device metrics exporter with a separate support matrix, and DCGM and NVIDIA MIG management do not administer AMD devices. The provisioning machinery that produces and promotes those images is the subject of Chapter 10.5.

For most of the last decade the node stack was a single-vendor question because the alternative was not credible at production scale. That has changed. ROCm 10.1.0, released 2026-10-05, is current. Its notes warn that RCCL all-reduce with the Low Latency protocol might return incorrect results on MI350 Series GPUs (workaround: NCCL_PROTO=LL128,Simple). The same release stops building ROCm SMI in the standard build, so move health checks and exporters to amd-smi before adopting it. ROCm 7.14 (July 2026), after the 7.9–7.13 technology previews, was the first production release built on TheRock; AMD then jumped straight to 10.0 (26 August 2026), so there is no 8.x or 9.x. The legacy 7.2 stream still installs to /opt/rocm-7.2 while the Core SDK installs to /opt/rocm/core-10.0, so pin which stream your image carries. AMD also committed at Advancing AI 2026 to a rigid six-week feature-release cadence. The ROCm half of a dual-vendor golden stack therefore rotates on a faster clock than CUDA 13.x, an operational cost to budget and not just a velocity win.

The strategic prize is real: a credible second source breaks single-vendor allocation pain and lands a lower acquisition price per accelerator, plus negotiating leverage that is usually worth more than the sticker discount. Do not carry a fixed percentage into a business case. The gap moves with SKU, OEM bundle, volume, support and networking content, and it can close or reverse once you normalize by delivered tokens instead of by part. What keeps it from being a free lunch is the ROCm tax, the engineering cost of the remaining ecosystem gaps.

Two things make ROCm viable today. First, RCCL implements NCCL's host collective API and follows NCCL releases at a lag: ROCm 10.1.0 (October 2026) ships RCCL 2.30.7, while NCCL is at 2.32.3. Frameworks written for NCCL can switch to RCCL with little change, and the tuning ideas carry over: algorithm and protocol choice, rail awareness, the LL and LL128 protocols. NCCL's newer device-API and NVLink-SHARP features have no direct RCCL equivalent. Second, the framework story has matured: upstream PyTorch and JAX run on ROCm, and inference engines such as vLLM support AMD targets.

Where the tax still bites is in the long tail: custom CUDA kernels and Triton paths that assume NVIDIA intrinsics, profiling and debugging tools less mature than Nsight/CUPTI, library coverage gaps (a kernel tuned in cuDNN but not yet in MIOpen), and the realized-MFU gap that drives TCO. Published benchmarks no longer show one uniform gap. It depends on the workload, the serving engine and the date. SemiAnalysis's InferenceX runs show how fast it moves: a single vLLM/AITER fix raised MI355X peak throughput on Kimi K2.5 by 7.7x in April 2026, and that still left it at about two-thirds of B200 single-node vLLM. On DeepSeek V4 Pro, SemiAnalysis (25 August 2026) found MI355X on SGLang level with B200 on vLLM for performance per dollar, until vLLM optimizations after 21 August moved B200 ahead. Signal65, in a study AMD commissioned (July 2026), found MI355X ahead on GPT-OSS-120B (1.30x, though it compared ATOM with TensorRT-LLM) and on Qwen3-Next-80B (1.27x, vLLM on both). On Kimi-K2.6 it had B200 roughly 17% ahead when concurrency was low to moderate. InferenceX shows B200 ahead on GPT-OSS-120B. Treat SemiAnalysis and Signal65 alike as starting points for a matched test of your own, not as a verdict. Disaggregated serving still favors NVIDIA. That is a crossover on some workloads, not a platform inversion.

The node-stack consequence is concrete: a dual-vendor fleet means two golden stacks, two firmware estates, two sets of NCCL/RCCL tuning tables and two on-call runbooks. You buy second-source leverage and pay in operational surface area. The deeper hardware and TCO treatment of this choice lives in Chapter 7.3, and the lock-in economics of CUDA vs ROCm vs XLA vs Neuron in Chapter 7.9.

The collective library is the most performance-critical and most tuning-sensitive component in the node stack. At scale, exposed communication (the part of each step the fabric fails to hide behind compute) increasingly sets job completion time, and NCCL/RCCL is what drives the fabric. Its job is to run the collectives that define distributed training (all-reduce, all-gather, reduce-scatter, all-to-all) at the highest possible bus bandwidth (busbw): algorithm bandwidth scaled by the collective's traffic factor, so it can be compared directly with the hardware's peak link bandwidth.

Operators quote one concrete starting point. Together AI's published cluster-testing rule of thumb (2024) is an all_reduce_perf busbw of about 92% of the fabric's theoretical maximum, roughly 370 GB/s on a 400 GB/s fabric, holding from two nodes up to the full cluster. It is not a portable constant, and it carries only with its conditions: command, ranks and GPUs per node, collective, message-size range, NCCL version, node count, rail use, and how reported busbw maps to the node's line rate. A GB300 NVL72 compute tray, for example, carries four 800 Gb/s ConnectX-8 NICs for its four GPUs (NVIDIA NVL72 reference architecture, 2026), and no published measurement establishes that the same ratio holds there. Recompute the byte target against the aggregate line rate of the node you actually bought, and set the acceptance bar from measurement on that node. Falling short of the bar you set is not a benign inefficiency. It is a tax amortized across the entire run, because every GPU waits on every collective.

NCCL auto-tunes well out of the box, and for many fleets the right answer is to ship defaults and leave the knobs alone. On a specific topology, though, the last stretch of busbw is recovered only by tuning the library to the fabric you built: algorithm selection (ring vs tree vs the newer adaptive trees, CollNet/PAT), protocol selection (Simple/LL/LL128), rail awareness so traffic stays on the rail-optimized topology, channel and buffer sizing, and, where the fabric supports it, SHARP in-network reduction. NCCL 2.27+ composes NVLink-SHARP and IB-SHARP, cutting the GPU SM count a reduction consumes from ~16 to ≤6 (NVIDIA's figure, for AllGather and ReduceScatter). The 2.28+ line adds a device-side communication API and Multimem multicast over NVLink-SHARP within an NVL72 domain. Ship defaults first, measure busbw and step time against your own baseline, and tune only where the gap justifies the work. The in-network-compute mechanics live in Chapter 8.6, and the topology and oversubscription decisions that NCCL tuning must match in Chapter 8.5. Defaults or tuned, the settings you run belong in the pinned golden stack, because a tuning skew is a silent bandwidth skew across nodes.

Everything above converges on one operating model: stack pools, each running one tested tuple. A tuple names the driver, CUDA/ROCm runtime, cuDNN/MIOpen, NCCL/RCCL, container toolkit and firmware revisions. Immutable digests and configuration attestation prove which artifacts each node in the pool runs. Whether those versions work together is a separate question, answered by vendor matrices and your own test of the complete tuple.

Candidate tuples move through lab, canary and ring stages as units, because upgrading one component alone can regress bandwidth or reliability in combination. A team can run a different tuple in an isolated pool beside production, which keeps room to move without forcing the whole fleet onto one byte-identical image.

Firmware is the part of the node stack operators most often defer: it is tedious, it requires reboots, and it rarely breaks loudly. The GB200 NVL72 ramp showed how much rack reliability rests on the layers under the application. The FT reported in May 2025 (via DCD) that partners lost months to overheating, coolant leaks, software bugs and chip-to-chip connectivity problems before volume shipments began in 2025. SemiAnalysis's August 2025 benchmarks blamed early-rack downtime on the copper NVLink backplane and on diagnostics too immature to find faults quickly. A node carrying stale NIC, NVSwitch or VBIOS firmware contributes gray failures: link flaps, lane dropouts, intermittent throttling. These do not crash the node, but they drag synchronous collectives and add to the every-few-hours interruption rate Meta logged on its Llama 3 405B run across up to 16,384 GPUs (Chapter 10.1), the rate that destroys multi-week run economics. Stale firmware is therefore a slow leak in reliability, not a benign backlog.

The modern firmware estate is managed out-of-band and as code. The OCP GPU Firmware Update specification standardizes the mechanics: Redfish for the management API, PLDM-over-MCTP for the update transport, and secure out-of-band update so firmware can be staged and applied without a host agent. At fleet scale, firmware updates follow the same ring discipline as the golden stack: rolling/canary rollout, drift detection against a pinned firmware baseline, a dependency matrix that sequences firmware against driver and CUDA so no node is stranded between incompatible layers, and rollback when a canary regresses.

Two constraints make this harder than ordinary patching. Every update is a scheduling event, because a node must drain its job before it can reboot. And firmware is a high-value attack surface, so updates must be signed and attested against a hardware root of trust (Caliptra/DICE, measured boot), the subject of Chapter 11.4. The day-2 firmware and software lifecycle, covering estate operations and change management, is deepened in Chapter 14.8. The refresh and disposition end of the hardware estate is Chapter 14.9.

One layer is easy to overlook because it is invisible when it works: the container runtime that exposes GPUs to a workload. On NVIDIA, the NVIDIA Container Toolkit hooks the OCI runtime to inject the GPU devices (/dev/nvidia*), the driver user-space libraries and the right capabilities into the container at start. The containerized framework therefore links against the host's pinned driver rather than a stale library baked into the image. The industry is migrating this device-exposure model to the vendor-neutral Container Device Interface (CDI), which both NVIDIA and AMD implement, decoupling device injection from any single runtime. For the golden stack the consequence is subtle but important: even in a containerized fleet, the driver is a host property and the container inherits it. A host-baked driver plus Operator-managed user space is therefore a coherent posture, not a contradiction.

Treat the Container Toolkit and GPU Operator as tuple members: pin their versions and patch them the way you patch the driver. CVE-2025-23266 (CVSS 9.0, July 2025) let a crafted container image gain root on the host, because the toolkit's createContainer OCI hook runs with host privilege. The fixed releases are Container Toolkit 1.17.8 and GPU Operator 25.3.2. Multi-tenant GPU nodes need the patch before they run customer images (Chapter 11.6).

Kubernetes has also changed how jobs request GPUs. Under Dynamic Resource Allocation (DRA), a job requests GPUs by their attributes, such as MIG geometry, NVLink domain or memory, rather than by count. DRA went stable in Kubernetes v1.34, released in August 2025. NVIDIA's GPU DRA driver, donated to the Kubernetes community in 2026, and AMD GPU Operator 1.5 both support it. That pushes more of the node stack's partitioning and topology into the scheduling layer covered in Chapter 10.2 and Chapter 10.3.

The node-stack failures that recur are all variations on one mistake: treating a coupled organism as a set of independent knobs.

• Admitting an untested stack tuple. A driver, runtime, collective-library or firmware combination nobody tested can hang or slow every rank in the job. Gate admission on the tested tuple and artifact attestation, not on one matching version string or a wiki page.
• Upgrading one component on a live training fleet. A 'safe' point-upgrade to a driver or NCCL passes in isolation, regresses in combination, and is then mixed into production runs. Promote the stack atomically through rings, never component by component.
• Deferring firmware until something breaks. Stale NIC/NVSwitch/VBIOS firmware contributes gray failures (link flaps, throttling, lane dropouts) that never crash a node but drag every synchronous collective and inflate the interruption rate. Firmware is a scheduled, attested, ring-rolled estate, not an emergency activity.
• Going dual-vendor for the discount without budgeting the surface area. A second vendor adds golden stacks, firmware matrices and support responsibilities. Any acquisition-price saving must come from matched quotes, and the operational cost needs its own estimate. Qualify the workload and price that work before using a line-item discount to justify the second path. → Chapter 7.3.

Promote the complete tuple only when application and recovery tests pass on the intended topology. Holding a release delays new features, while promoting a library on support status alone can stall every rank. Keep the incumbent pool available until rollback of the candidate has been exercised.