Node Software Stack: Drivers, CUDA/ROCm, NCCL & Firmware

Above the rack and below the scheduler sits a layer that almost no slide deck shows and almost every multi-week training failure traces back to: the node software stack. It is the vertical column of software that turns a powered, networked GPU server into a node a distributed job can actually use — the kernel driver that talks to the silicon, the CUDA or ROCm runtime the framework links against, the collective-communication library (NCCL on NVIDIA, RCCL on AMD) that carries every all-reduce, and the firmware estate (GPU VBIOS, NIC, BMC, NVSwitch, CX/Thor, PSU) that sits beneath all of it. These are not independent knobs. They are a single coupled organism with a compatibility matrix, and the matrix is unforgiving at scale: a synchronous training job runs at the speed of its slowest, most-skewed node, and an unsupported or unvalidated library and firmware combination can fail, hang, corrupt output, or silently regress collective bandwidth across every participating GPU.

Most operators treat this layer as plumbing until it costs them a run. We trace the vertical stack and the driver lifecycle; the injection model (GPU Operator vs golden image) and the container-runtime layer beneath it; the AMD ROCm path as a real but taxed second source; NCCL/RCCL tuning as the difference between paper bandwidth and realized bus bandwidth; and the discipline that holds it all together — tested-compatible, attested stack pools at each scheduling boundary, and firmware update mechanics under attestation. Across every layer the trade is the same: qualified compatibility and attestation buy goodput while approved pools preserve controlled agility, and choosing wrong shows up not as a config error but as a stalled $50M run.

Read the node stack bottom-to-top and the coupling becomes obvious. At the base is firmware — GPU VBIOS/InfoROM, NVSwitch/UALink-switch firmware, NIC firmware (ConnectX/BlueField/Thor on the NVIDIA side; Pollara/Pensando on AMD), BMC, BIOS/UEFI, PSU and CDU controllers — each with its own version, its own update path, and its own dependency on the layers above. On top of firmware sits the kernel driver (the NVIDIA data-center driver branch, or AMDGPU/amdkfd for ROCm), which must match the kernel ABI and the GPU's firmware. The driver exposes a user-space runtime: the CUDA driver API and toolkit (CUDA 13.3 as of mid-2026), or the ROCm runtime (HIP, rocBLAS, MIOpen). Above that are the math and communication libraries — cuDNN, cuBLAS, NCCL on NVIDIA; MIOpen, rocBLAS, RCCL on AMD — and finally the framework (PyTorch, JAX, vLLM) the user actually writes against.

The reason this matters is that the constraints run both ways. A given CUDA toolkit requires a minimum driver; a given NCCL build is compiled against a CUDA version; a given firmware revision is validated against a driver branch; and the framework pins a cuDNN/NCCL it was built and tested with. NVIDIA's CUDA forward-compatibility and minor-version compatibility features loosen this slightly — a newer CUDA toolkit can run on an older driver within the same major branch via the compat package — but those are escape hatches, not a license to let the fleet drift. The operative posture for a production cluster is the opposite: admit only a tested-compatible, attested tuple at each scheduling boundary; approved pools may carry different qualified tuples, while undeclared drift is a defect.

NVIDIA ships data-center drivers in two cadences, and choosing between them is the first decision. Production Branches rotate roughly annually with about a year of full support; Long-Term Support Branches (LTSB) get a three-year lifecycle prioritizing stability over features. As of mid-2026 the R580 branch is the current LTSB, with support running to roughly August 2028, paired against CUDA 13.x. A fleet that values determinism over feature velocity pins to an LTSB and upgrades on a deliberate cadence; a fleet chasing the newest accelerator generation or a kernel feature may have to ride a Production Branch and accept the shorter support window. The consequence of choosing wrong is asymmetric: ride too-new a branch and you inherit regressions on hardware you cannot afford to debug under a live run; ride too-old a branch and you cannot enable the next density generation when the ramp arrives.

The second decision is how the driver lands on the node. Two models dominate, and they trade agility against bare-metal determinism.

The GPU Operator (NVIDIA's, and AMD's equivalent) is the Kubernetes-native answer: it deploys and lifecycles the containerized driver, the container toolkit, the Kubernetes device plugin, DCGM/DCGM-exporter, the MIG manager, and node-feature discovery as a coordinated set, and it performs rolling driver upgrades by cordoning and draining a node before swapping its stack. That is exactly the right tool for a cloud-native, inference-heavy, or heterogeneous fleet where you want to roll a driver across thousands of nodes without re-imaging. The cost is more moving parts in the critical path and a dependency on the container runtime being correctly wired (the NVIDIA Container Toolkit injecting /dev/nvidia*, libraries, and the right capabilities into the container; the equivalent CDI-based device injection on the AMD side). For change-controlled Slurm/HPC training fleets the opposite instinct usually wins: bake the driver and toolkit into a golden OS image, attest it at boot, and treat each node's signed golden-image artifact as byte-identical within its approved pool; another approved pool may carry a different qualified artifact, and the scheduler enforces the tested compatibility boundary. Many large operators run a hybrid: host-baked driver for determinism, Operator-managed plugins/DCGM/MIG for the layers that benefit from declarative lifecycle. The provisioning machinery that produces and promotes those images is the subject of Chapter 10.5.

For most of the last decade the node stack was a single-vendor question because the alternative was not credible at production scale. That has changed. ROCm 7.x (7.0 shipped September 2025 with MI350X/MI355X support; the current production line is 7.14, the first production release built on the TheRock infrastructure, 2026-07-15 — 7.9–7.13 were tech previews, which is why the version number jumped) is the first AMD software stack that a serious operator can stand up a multi-thousand-GPU training or inference fleet on without heroics. AMD also committed at Advancing AI 2026 to a rigid six-week feature-release cadence — meaning the ROCm half of a dual-vendor golden stack now rotates on a faster clock than CUDA 13.x, an operational cost to budget, not just a velocity win. The strategic prize is real: a credible second source breaks single-vendor allocation pain and lands roughly 15-30% lower hardware cost per unit of compute, plus negotiating leverage that is worth more than the sticker discount. What keeps it from being a free lunch is the ROCm tax — the engineering cost of the remaining ecosystem gaps.

Two things make ROCm viable today. First, RCCL maintains NCCL-API parity: it implements the same collective API surface NCCL exposes, so frameworks that call NCCL can call RCCL with minimal change, and the tuning concepts (algorithm/protocol selection, rail awareness, LL/LL128 protocols) carry over. Second, the framework story has matured — upstream PyTorch and JAX run on ROCm, and inference engines like vLLM support AMD targets. Where the tax still bites is in the long tail: custom CUDA kernels and Triton paths that assume NVIDIA intrinsics, profiling and debugging tooling that is less mature than Nsight/CUPTI, library coverage gaps (a kernel that exists tuned in cuDNN but not yet in MIOpen), and the realized-MFU gap that drives TCO. Independent benchmarking has shown the gap is now workload-and-engine-specific rather than uniform: with upstream AITER kernels, aggregated vLLM on MI355X has beaten B200 vLLM on at least one production MoE workload (Kimi K2.5; SemiAnalysis, Aug 2026), while Signal65's July 2026 evaluation has MI355X ahead on GPT-OSS-120B and Qwen3-Next-80B but B200 ~17% ahead on Kimi-K2.6 — and disaggregated serving still favors NVIDIA. A crossover on some workloads, not a platform inversion. The node-stack consequence is concrete: a dual-vendor fleet means two golden stacks, two firmware estates, two sets of NCCL/RCCL tuning tables, and two on-call runbooks. You buy second-source leverage and pay in operational surface area. The deeper hardware and TCO treatment of this choice lives in Chapter 7.3; the lock-in economics of CUDA vs ROCm vs XLA vs Neuron in Chapter 7.9.

The collective library is the most performance-critical and most tuning-sensitive component in the node stack, because at training scale above a few hundred GPUs the fabric — not the GPU — sets job completion time, and NCCL/RCCL is what drives the fabric. Its job is to execute the collectives that define distributed training (all-reduce, all-gather, reduce-scatter, all-to-all) at the highest possible bus bandwidth (busbw) — the effective throughput a collective achieves relative to the link's theoretical ceiling. The acceptance bar operators gate handoff on is concrete: NCCL all_reduce_perf should reach roughly 92% of the node's aggregate theoretical fabric bandwidth scaling from two nodes to the full cluster — about 370 GB/s busbw against an eight-rail NDR node's 400 GB/s aggregate line rate (8 × 400 Gb/s = 8 × 50 GB/s); one link is 50 GB/s. That pair of numbers is an eight-rail NDR observation, not a portable constant: a GB300 NVL72 compute tray carries four 800 Gb/s ConnectX-8 NICs for its four GPUs (NVIDIA NVL72 reference architecture, 2026), and no published measurement establishes that the same ratio holds there. Recompute the byte target against the aggregate line rate of the node you actually bought, and establish the acceptance bar from measurement on that node. Falling short of the bar you set is not a benign inefficiency; it is a tax amortized across the entire run, because every GPU waits on every collective.

NCCL auto-tunes well out of the box, and for many fleets the right answer is to ship defaults and not touch the knobs. But the last 10-20% of busbw on a specific topology is recovered only by tuning the library to the fabric you actually built: algorithm selection (ring vs tree vs the newer adaptive trees, CollNet/PAT), protocol selection (Simple/LL/LL128), rail awareness so traffic stays on the rail-optimized topology, channel and buffer sizing, and — where the fabric supports it — SHARP in-network reduction. NCCL 2.27+ composes NVLink-SHARP and IB-SHARP, cutting the GPU SM count consumed by a reduction from ~16 to ≤6 and halving the data on the wire; the 2.28+ line adds a device-side communication API and Multimem multicast over NVLink-SHARP within an NVL72 domain. The in-network-compute mechanics live in Chapter 8.6, and the topology/oversubscription decisions that NCCL tuning must match in Chapter 8.5. The trade-off is concrete: ship defaults and accept ~80-90% of achievable busbw, or invest in per-topology tuning and recover the rest. Either way the tuning tables become part of the pinned golden stack, because a tuning skew is a silent bandwidth skew across nodes.

Everything above converges on a controlled operating model: approved stack pools governed by tested compatibility and attestation. A tuple names the driver, CUDA/ROCm runtime, cuDNN/MIOpen, NCCL/RCCL, container toolkit and firmware revisions. Exact artifact identity is enforced within each signed image or container pool by immutable digests and configuration attestation. Supported interoperability is a separate property established by vendor matrices and project qualification of the complete tuple. A fleet may run several approved pools; the scheduler prevents a job or service from spanning tuple combinations that were not qualified for that boundary.

Candidate tuples are promoted through lab, canary and ring stages as units because an isolated component upgrade can regress bandwidth or reliability in combination. A team may use a different approved tuple in an isolated pool, and two pools may serve the fleet concurrently. Crossing them inside one synchronous job requires representative collective, framework and workload validation of that exact combination. This preserves controlled agility without claiming that all participating artifacts must be byte-identical or that every version difference deterministically hangs.

Firmware is the part of the node stack operators most often defer — it is tedious, it requires reboots, and it rarely breaks loudly. The GB200 ramp showed what that deferral costs at scale: integration reliability hinged on firmware, with NVLink copper-backplane issues and NVL36×2 cross-rack signal-integrity problems fixed partly through firmware revisions. A node carrying stale NIC, NVSwitch, or VBIOS firmware is a node that contributes gray failures — link flaps, lane dropouts, intermittent throttling — that do not crash the node but do drag synchronous collectives and accumulate into the every-few-hours interruption rate that destroys multi-week run economics. Stale firmware is therefore a slow leak in reliability, not a benign backlog.

The modern firmware estate is managed out-of-band and as code. The OCP GPU Firmware Update specification standardizes the mechanics — Redfish for the management API, PLDM-over-MCTP for the update transport, and secure out-of-band update so firmware can be staged and applied without a host agent. At fleet scale, firmware updates follow the same ring discipline as the golden stack: rolling/canary rollout, drift detection against a pinned firmware baseline, a dependency matrix that sequences firmware against driver/CUDA so you never strand a node between incompatible layers, and rollback when a canary regresses. Two constraints make this harder than ordinary patching: every update is a scheduling event (a node must drain a job before it can reboot), and firmware is a high-value attack surface — so updates must be signed and attested against a hardware root of trust (Caliptra/DICE, measured boot), the subject of Chapter 11.4. The day-2 firmware-estate operations, change-management, and refresh mechanics are deepened in Chapter 14.9.

One layer is easy to overlook because it is invisible when it works: the container runtime that actually exposes GPUs to a workload. On NVIDIA, the NVIDIA Container Toolkit hooks the OCI runtime to inject the GPU devices (/dev/nvidia*), the driver user-space libraries, and the right capabilities into the container at start — so the containerized framework links against the host's pinned driver rather than a stale library baked into the image. The industry is migrating this device-exposure model to the vendor-neutral Container Device Interface (CDI), which both NVIDIA and AMD implement, decoupling device injection from any single runtime. The consequence for the golden stack is subtle but important: even in a containerized fleet, the driver is a host property and the container inherits it, which is why a host-baked driver plus Operator-managed user-space is a coherent posture rather than a contradiction. Kubernetes is also moving the resource-request model itself — from counting whole GPUs toward Dynamic Resource Allocation (DRA), which lets a job claim attributes (MIG geometry, NVLink topology, memory) rather than a count — which pushes more of the node-stack's capabilities (partitioning, topology) into the scheduling layer treated in Chapter 10.2 and Chapter 10.3.

The node-stack failures that recur are all variations on one mistake — treating a coupled organism as a set of independent knobs:

• Admitting an unvalidated stack tuple. An unsupported driver, runtime, collective-library or firmware combination may error, hang, corrupt output or regress bus bandwidth. Prevent it with admission on the tested compatibility policy and artifact attestation, not equality of one version string or a wiki page.
• Upgrading one component on a live training fleet. A 'safe' point-upgrade to a driver or NCCL that passes in isolation but regresses in combination, now mixed into production runs. Promote the stack atomically through rings, never component-by-component.
• Deferring firmware until something breaks. Stale NIC/NVSwitch/VBIOS firmware contributes gray failures — link flaps, throttling, lane dropouts — that never crash a node but drag every synchronous collective and inflate the interruption rate. Firmware is a scheduled, attested, ring-rolled estate, not an emergency activity.
• Going dual-vendor for the discount without budgeting the surface area. A second vendor doubles golden stacks, firmware matrices, and on-call rotations. The 15-30% hardware saving is real, but so is the operational cost — go dual-vendor for leverage and scale, not for a line-item discount on a small fleet. → Chapter 7.3.