Firmware & Software Lifecycle Management at Fleet Scale

A modern AI cluster is a synchronized estate of tens of thousands of accelerators, and its collective performance is gated by the weakest, oldest, most-drifted node in the job. Firmware and software are the only parts of that estate you change deliberately and often. A GPU stays a GPU for its operating life, but its BMC firmware, GPU VBIOS, NVSwitch microcode, PSU and BBU firmware, kernel driver, CUDA toolkit and NCCL build all move on independent cadences: some quarterly, some weekly, some hot-patched mid-incident. Each change is a chance to fix a silent-data-corruption bug or close a critical CVE. It is equally a chance to brick a node, cut collective bandwidth by a few percent across the whole fabric, or introduce a version skew that makes a 100,000-GPU job refuse to start. This chapter is about making those changes thousands of times a year without losing goodput.

The costs are asymmetric and quantified. A synchronized training job restarts from its last checkpoint the moment any node fails or falls out of version. SemiAnalysis reported that a 512-H100 cluster at a top-tier operator typically has a hardware MTBF of about seven days (October 2024). Planned change costs restarts too. In Meta's 54-day snapshot of Llama 3 405B pre-training on up to 16,384 H100s, 47 of the 466 job interruptions were planned (automated maintenance such as firmware upgrades, or operator configuration and dataset changes), and 419 were unexpected (2024). A botched firmware push manufactures failures on top of all that. An inference fleet earning revenue against an SLO cannot take the whole hall down for a maintenance window. The firmware estate is also an attack surface now: 2025 brought real, CVSS-9-class vulnerabilities in the GPU software supply chain that demanded a fleet-wide patch on a schedule the operator did not choose. The chapter covers the firmware estate and its update mechanics, fleet orchestration (rolling, canary, drift detection), the dependency matrix that makes version skew the silent killer, the supply-chain-integrity gate, and the rollback discipline that decides whether a bad bit costs one node or one cluster. Chapter 14.12 owns the change-management procedures that govern all of it; this chapter is the firmware-and-software view into that framework.

Strategists tend to picture "the GPU" as one thing running one driver. An operations team actually manages a stack of a dozen-plus independently versioned firmware images per node, and most of them are invisible until one is wrong. On a GB300 NVL72 rack, and on the GB200 NVL72 installed base beside it, the firmware bill includes the host BMC, the GPU VBIOS/InfoROM and on-die microcode, the NVSwitch tray firmware, the ConnectX/BlueField NIC firmware, optics/transceiver firmware, PSU and power-shelf firmware, BBU/capacitor-module firmware, CDU and rack-controller firmware on the cooling side, and the Grace CPU UEFI/BIOS. The software stack sits above that: the GPU kernel driver, CUDA, cuDNN, NCCL, the container toolkit and the orchestrator agents. It moves faster than firmware and is covered as a node-stack subject in Chapter 10.4.

One decision governs everything downstream: treat this as one atomically pinned fleet image or as independently floating layers. Pinning everything to a single validated bundle gives you deterministic, reproducible nodes and a trivial answer to "what is running where". The cost is slower patching, and you cannot hot-fix one layer without re-validating the whole bundle. Floating the layers lets you push a security driver in hours without re-qualifying firmware. The cost there is a combinatorial drift surface that turns a 100,000-GPU fleet into thousands of unique version tuples, any of which can harbor the skew that kills a job. Most mature operators land on a hybrid: firmware pinned to validated bundles on a slow cadence, the software stack floated within a tested compatibility window, and a hard rule that no node joins a synchronized job unless its full version tuple belongs to the job’s qualified compatibility set, with documented exceptions and regression gates.

The industry has standardized firmware updates on out-of-band management. As of 2025-2026 the reference is the OCP GPU Firmware Update Specification (v1.0, now v1.1), which layers a fleet orchestrator on Redfish UpdateService for transport and PLDM-for-Firmware-Update over MCTP (DMTF DSP0267) for the device-level protocol. The orchestrator copies a signed firmware bundle to the host or accelerator BMC via Redfish. The BMC, acting as PLDM Update Agent, discovers the update-capable firmware devices behind it over MCTP and stages the new image to each. Then it activates across a reset. This convergence is why one fleet tool can flash GPUs, NICs and switches from different suppliers through one interface, instead of a zoo of vendor-specific flashing utilities run by hand. The NVIDIA-stack instance is Mission Control 2.3.1 (documentation current as of October 2026). It provides autonomous hardware and job recovery on DGX B200/B300 and DGX/OEM GB200/GB300 NVL72 systems, firmware upgrades for compute trays, switches and NVOS through its autonomous-hardware-recovery component, Grafana dashboards (beta on B300) and air-gapped deployment. It is vendor-specific, not a substitute for the interoperable OCP/Redfish path.

Your maintenance-window math depends on how each component separates image copy and apply from activation, and on which phases disrupt service. The variables are each component's measured copy/apply duration, whether that work is non-disruptive, its arm semantics, its activation/reset duration, and its rollback-bank design. The forks are real money. Copy → Arm → Activate is the current OCP reference behavior. The specification defines copy, verification and apply behavior intended to avoid service impact on conforming devices. Check what your devices actually implement, and which phases they support, before spending production time on staging. Explicit arming separates that work from activation in a maintenance window, which shrinks the outage to activation/reset, retraining and health/SLO readmission checks. A/B (dual-bank) firmware writes the new image to the inactive slot and flips a pointer, which separates image transfer from bank selection. Activation and rollback may still need a reset, retraining and health checks. Not every device in the rack has dual banks, and the ones that do not (often a PSU, a BBU controller or an optic) become the long pole and the one-way-door risk in your patch plan.

The table is a risk gradient. At the top, software gives you freedom: in-band, fast, reversible, low blast radius. At the bottom, power and fabric firmware is where the discipline lives: out-of-band, slow, sometimes one-way, and able to take down a 72-GPU NVLink domain or trip a rack. Operators underestimate the PSU/BBU row. Power-path firmware rarely has an A/B slot, and a failed flash on a power shelf is a truck roll, not a re-pull. Power-path firmware also sets grid behavior now. OCP's Interface Specification Between Hyperscalers and Utility Grid for Disturbance Performance (v1.0.0, effective 17 September 2026) is an industry specification, not a regulation. It names PSU firmware that taps the supply's stored energy as a likely way for computing load to ride through voltage sags. It also warns that feasibility differs by PSU maker and model, and that the fix is not always firmware alone. Put ride-through settings in the validated power-shelf tuple, and test them again after every PSU or power-shelf firmware change. Chapter 4.10 covers the ride-through envelope, Chapter 13.6 the commissioning test and Chapter 14.14 the live-campus retest. Order the updates by the OEM dependency matrix and by the rollback or recovery path you have actually tested, and hold each layer behind a health check before the next one moves. Updating the most reversible layer first limits the blast radius, but when a new driver requires newer device firmware, the firmware has to go first.

At fleet scale the question is never "how do I flash a node" but "how do I move a version across 100,000 GPUs without flashing all of them at once". The default is canary then rolling. Validate on a tiny, deliberately chosen canary set, ideally spanning every hardware revision and supplier in the fleet because firmware bugs are often SKU-specific. Gate on an automated health and goodput check, then expand in waves with a gate on each wave. The orchestration plane that does this runs break-fix workflows and health checks, drains a node from the scheduler before it is touched, and reintegrates it after validation. That is the autonomous-recovery and fleet-control subject of Chapter 10.7, and the telemetry that feeds the health gate is Chapter 10.6. Firmware lifecycle is a first-class consumer of both.

Operators who build this loop themselves now have two open-source NVIDIA components to start from, both Apache-2.0. Rack Management Service (October 2026) runs rack-level firmware and switch work as asynchronous jobs, built on its embedded nvfwupd library. As of 19 October 2026 its repository carries only a release-candidate tag and no published release. Cluster Readiness Engine (open source, v0.7.0, October 2026) is a Kubernetes controller. It runs multi-node burn-in workloads and lists the nodes that fail. It never cordons, taints or modifies a node, so quarantine, rollback and acceptance stay with you.

Cordon-and-drain versus in-place sets your update cadence. For a training fleet, you drain the canary and each later wave out of the scheduler so no synchronized job is running on a node mid-flash. The cost is reduced effective capacity during the campaign, but a synchronized job cannot tolerate a node vanishing, so this is non-negotiable. For an inference fleet you exploit loose coupling: drain a node's request traffic, let in-flight requests finish, flash, health-check, and return it to rotation, with spare headroom absorbing the temporary dip. The N+1-style margin you carry for hardware failures now also funds your patch velocity. Getting this wrong has a direct price: flash a node that still holds a synchronized training rank and you have killed a job and forced a checkpoint restart (Chapter 9.4).

Trace. Maximum simultaneous outage = 8 − 6 = 2 nodes. Required waves = ceil(8/2) = 4. Window = 4 min + 4 × (4 + 3) min = 32 min, which exceeds the 30-minute allowance by 2 min. Get a longer approved window or defer; a bigger wave would breach the six-node demand floor. The two-node allowance is maintenance capacity: if it is also your failure reserve, one node failure during a wave takes the service below demand. If the service must keep one spare node through every wave, only one node can be offline at a time, which means 8 waves and 4 + 8 × 7 = 60 min.

Flip. With the other figures fixed, activation/reset must take at most (30 − 4)/4 − 3 = 3.5 min per wave, or the window must be at least 32 min. Equality leaves no contingency, so the approved plan needs a reserve sized from measured variation. Stop expansion on a canary regression and use the tested recovery path. Chapter 14.14 covers the retests when a change alters behavior, and Chapter 14.9 the hardware refresh and disposition decision.

Drift is the gap between what you believe is deployed and what is actually flashed, and it accumulates relentlessly: a node that failed a flash and silently reverted, a hot-fix applied by hand during an incident and never recorded, an RMA replacement that arrived with factory firmware, a wave that the orchestrator marked complete but that timed out on three nodes. At fleet scale it never stops accumulating, so drift is the steady state you actively fight. Only continuous drift detection keeps the "what is running where" answer trustworthy. On a schedule, the orchestrator pulls each node's firmware inventory from Redfish, reads driver, CUDA, NCCL and container-toolkit versions off the running OS, and compares the combined tuple with the intended manifest. A trustworthy inventory is the precondition for every safe rollout.

The dependency matrix is the second silent killer. Driver, CUDA, NCCL and GPU firmware are not independently choosable. NVIDIA publishes driver/CUDA compatibility documentation, and its data-center driver branches decide how long a tuple can stay put. A Production Branch gets bug and security fixes for up to one year, a Long Term Support Branch for three years (NVIDIA driver lifecycle, checked October 2026). Choose a branch that stays supported for as long as each bundle runs in production. If it reaches end of life first, the next security fix arrives only on a newer branch, and you change branches mid-cycle without planning to. Within one major CUDA release, minor-version compatibility lets a newer toolkit run on that driver. Crossing to a newer major release on an older driver takes NVIDIA's forward-compatibility package. That driver must sit on a branch NVIDIA's compatibility table still lists (New Feature Branches and end-of-life branches do not qualify), and the GPU must be a data-center part, a select NGC Server Ready RTX card or a Jetson board. AMD's ROCm has its own compatibility matrix. NVIDIA's XID error catalog belongs to the health gate (Chapter 10.6), not to the tuple.

The failure modes are real, not theoretical: a driver below the toolkit's minimum supported version, an unsupported GPU/OS combination in the qualified tuple, a NCCL build that assumes a fabric-firmware feature the NVSwitch tray does not yet have, a CUDA forward-compat shim that papers over a kernel-driver gap until it does not. You cannot update one cell of the matrix in isolation; you validate a tuple and promote it as a unit. Teams that skip this learn it the expensive way, when a security-driver push that was "obviously safe" silently regresses collective bandwidth across the fabric because it shifted the validated NCCL pairing.

A specification version printed on a datasheet does not prove the feature is implemented. A NIC, switch, driver or collective library can cite UEC 1.1, October 12, 2026 and still be missing the pieces of that release your fabric depends on. Pin the exact tuple you validated and prove it during acceptance (Chapter 13.7). Open Rack Wide likewise spans base, canister and Meta design documents, each with its own status in Chapter 0.4; check which one a new rack follows rather than treating “ORW” as one interface.

Every firmware image you push is privileged code that runs below the OS, often before it, on hardware holding model weights worth more than the building. That makes the firmware pipeline a first-class attack surface, and 2025 made the point concrete. CVE-2025-23266 ('NVIDIAScape'), a CVSS-9.0 container escape in the NVIDIA Container Toolkit, was systemic across managed GPU services because, as Wiz notes, many managed AI and GPU services at all the major clouds are built on that toolkit. The flaw reached Toolkit 1.17.7 and GPU Operator 25.3.1 and every release before them. Operators had to coordinate a fleet-wide patch on a timetable none of them picked. In 2026 the toolkit needed another urgent fix, for CVE-2026-24260 (CVSS 8.5; Chapter 10.3 covers it). The lesson generalizes: the GPU software-and-firmware supply chain is now a place where one upstream defect becomes ten thousand operators' incident at once.

The defensive stack has standardized faster than most operators realize. Measured/secure boot anchored in a silicon root of trust (vendor BMC RoTs, and the open Caliptra RoT where the specific part carries it; as of October 2026 no shipping data-center part is confirmed to carry it, so check per part) separates measurement for attestation from signature-based execution policy. Neither proves the approved firmware is defect-free; Chapter 11.4 covers the hardware-security depth. Signed firmware bundles with Reference Integrity Manifests (RIM) and vendor SBOMs let you verify provenance and contents before flashing, the supply-chain-provenance subject of Chapter 11.3. The OCP S.A.F.E. program, with eleven listed independent Security Review Providers, checked 16 October 2026, gives operators a portable, third-party firmware-security audit (a JSON Short Form Report with firmware hashes and outstanding findings), so they are not re-auditing every vendor image themselves.

The operational decision is where you set the gate: a hard rule that no firmware reaches the staging path without the required signature/provenance evidence, a matching RIM where supported, and the security review required by the procurement policy. A S.A.F.E. report is one scoped review mechanism. That gate makes supply-chain integrity a prerequisite for staging. The cost is slower onboarding of new vendor releases; the consequence of skipping it is flashing unverified privileged code to the entire estate.

Capacity out of service is revenue not earned, with roughly $12–13B/GW/yr in SemiAnalysis’s contested revenue scenario. To price an outage, use your own workload’s marginal value and demand. Three levers cut the goodput cost of keeping a fleet current. First, exploit the workload's own tolerance. A synchronized training fleet must be drained, but an inference fleet's loose coupling lets you patch rolling-in-place behind spare headroom, so patch velocity is partly free if you sized redundancy for failures anyway. Second, move supported image copy/staging outside the outage window, using impactless staging and A/B slots wherever the hardware supports them. Where non-disruptive staging is supported, that removes the OCP v1.1 example’s roughly 15-minute image-copy phase from the outage. The unavailable interval still includes activation/reset, retraining and health/SLO readmission. Third, batch firmware change into already-scheduled drains. When a node is down for predictive or preventive maintenance of its power and cooling plant (Chapter 14.5) or for a hardware swap, flash it then instead of spending a second window.

Operations is also becoming software-defined. The same Redfish/SMI control plane that flashes firmware also caps power and clocks, smooths transients and re-routes around failures, so a firmware change and an operational policy change flow through one programmable interface. That is powerful and dangerous in equal measure: an operator can change the behavior of 100,000 GPUs with one API call, which is exactly why the governance and human-error controls below stand between you and a self-inflicted fleet-wide incident. Human and organizational failures can defeat those controls, and a programmable fleet multiplies the reach of a single mistake (Chapter 14.11).

Rollback bounds the blast radius of a bad decision. Ask the governing question for every firmware push before the push, not after the failure: can I revert this, how fast, and to what granularity? The answer depends on the hardware (A/B dual-bank or single-bank), the protocol (staged activation lets you abort before the flip) and the plan (did you validate that the prior version is still flashable, keep the old bundle, and stop the rollout the instant the canary's health gate tripped). A fleet with disciplined rollback turns a bad vendor image into a contained canary incident. A fleet without it turns the same image into a multi-day re-flash or RMA campaign across every node the rolling update reached before anyone noticed.

Three rules separate disciplined operators from the rest. Never push a firmware change without a tested rollback or replacement-and-recovery path. For one-way devices (many PSU/BBU controllers, some VBIOS slots), the revert plan is a hardware swap, and the canary must be wide enough and soaked long enough to justify the irreversibility. Gate every wave on automated health and goodput, and make the gate able to stop the rollout itself. A human watching a dashboard is too slow to keep a bad bit from reaching 10,000 nodes, so the gate must halt expansion on a delta without waiting for a pager. Record every change against the manifest in real time, including hand-applied incident hot-fixes, because the drift you do not record is the drift that ambushes a job three weeks later. These rules are the firmware instance of the rollback-and-change discipline that the procedures framework formalizes as MOPs, the CAB/MOC process and human-error trapping in Chapter 14.12.