GPU Node Burn-In, Diagnostics & Stress Validation

Every earlier chapter in Part 13 commissioned the building: power chain energized, cooling loop proven, fabric link-tested, and the integrated system shown to transfer to backup and recover from faults (Chapter 13.6). This chapter commissions what the building exists to run: the GPU nodes. It is also the gate most often skipped under schedule pressure, because the nodes boot, the smoke test passes, and putting revenue-earning work on them looks one decision away. A GPU fleet follows a bathtub failure curve. A population of marginal dies, cold solder joints, mis-seated HBM stacks and borderline cold-plate contacts will fail early and predictably. Burn-in forces that infant-mortality population to fail on your acceptance clock, in a controlled window, before any of it touches a synchronous training job that restarts from checkpoint when a single GPU dies.

The consequences are easy to see because they are denominated in goodput. Skip the soak and the infant-mortality failures move into the first real workload. There, one bad GPU at hour 40 of a synchronous run means a checkpoint restart across the whole job, a straggler that drags the slowest-link collective, or worse, a silent corruption that poisons gradients for days before anyone notices. The chapter covers node bring-up and inventory, the DCGM diagnostic ladder, soak-duration selection, HBM and ECC validation, thermal screening under load, silent-data-corruption (SDC) hunting, straggler detection and power-behavior validation, each with the downstream cost of its acceptance decision. It qualifies the individual node; Chapter 13.9 takes the qualified fleet to cluster scale.

Burn-in opens with an inventory reconciliation against ground truth, before any stress test runs. At the scale of a modern AI hall, a non-trivial fraction of nodes arrive with something quietly wrong: a GPU running an older VBIOS than its siblings, an HBM stack that enumerates at the wrong capacity, a PCIe link that trained at x8 instead of x16, an NVLink that came up degraded, a NIC in the wrong slot, or a firmware revision that does not match the approved baseline. None of these stops the node booting, and all of them silently destroy performance or reliability once it joins a tightly coupled job. The inventory pass enumerates every accelerator, memory stack, link, firmware version and driver version, and reconciles them against the approved design baseline, not the purchase order.

Inventory is also where you choose between homogeneity as an acceptance criterion and tolerated drift, and each has a cost. A synchronous training job moves at the speed of its slowest node, so mixed firmware, mixed clock states or one node with a degraded NVLink slows the whole job, every rack and every replica. Treating any deviation from the golden baseline (firmware, VBIOS, driver, link width, power limit, even GPU clock offset) as a defect to fix before acceptance costs remediation time now. Tolerating the drift costs a tax on every step of every collective for the life of the cluster. Out-of-band firmware management (Redfish / PLDM-over-MCTP, per the OCP GPU firmware spec) makes baseline enforcement tractable at fleet scale; Chapter 8.7 commissioned the management fabric that carries it.

NVIDIA's Data Center GPU Manager (DCGM) supplies the de facto diagnostic ladder for Blackwell- and Hopper-class fleets, and its four run levels are the working vocabulary of GPU acceptance. The tension is depth against throughput. The deepest level is too slow to run on every node continuously and too important to skip before acceptance, so the program runs each level at its own gate.

• -r 1 (seconds): deployment and software sanity. Does the GPU enumerate, is the driver healthy, are NVML, persistence mode and the basic plumbing correct? This is the triage check you run constantly, including as a day-2 health gate before scheduling a job onto a node.
• -r 2 (NVIDIA Hopper estimates: under 2.5 min for four GPUs; under 10.5 min for eight): adds quick integration and PCIe/NVLink bandwidth sanity. Measure its installed runtime and coverage before deciding whether the pre-job slot can carry it; don't assume a two-minute allowance.
• -r 3 (NVIDIA Hopper estimates: under 10 min for four GPUs; under 35 min for eight): hardware stress, with targeted compute, memory-bandwidth and power-stress plugins that actually load the silicon. Put its compute and power scope in the project diagnostic matrix; acceptance also needs every other contracted test, not just this level number.
• -r 4 (NVIDIA Hopper estimates: under 45 min for four GPUs; under 2.25 hr for eight; release-dependent): the deepest pass. It adds the memtest plugin (memtest86-style HBM pattern testing), the Pulse (Input EDPp) test, which swings power draw hard enough to show whether a node's power supplies cope with sharp current spikes, and, from DCGM's supported installed diagnostic package, the Extended Utility Diagnostics (EUD) profile. The run-level label does not establish that EUD ran. Select the applicable profile at soak entry and exit, then check the emitted test manifest: a missing plugin or restricted package is missing coverage.

The DCGM ladder is diagnostic, not durational. It searches for defects in the patterns and conditions it exercises, and it certifies neither complete present health nor survival through the next week. The soak answers that second question. Long-running synthetic stress (GPU-Burn, dense GEMM loops, NCCL collective floods) holds the silicon at full thermal and electrical load for hours to days, and DCGM runs at the soak boundaries and after every fault.

DCGM's level 1–4 diagnostics run inside one host; its NVBandwidth and NCCL plugins do not test across hosts. In an NVL72-class rack the NVLink domain spans 18 compute trays, so a clean per-tray result does not accept the domain. Log the DCGM version and the IMEX domain state beside each node's results. Chapter 13.7 holds the multi-node NVLink gate (DCGM's multi-node diagnostics, which DCGM 4.7 extended with nvloom and IMEX health checks). Chapter 13.9 holds workload-based fleet certification, including NVIDIA's open-source Cluster Readiness Engine (v0.7.0, October 2026), which scripts level-4 DCGM runs alongside NCCL sweeps. It reports failing nodes but never cordons them, so quarantine stays the operator's call.

AMD Instinct fleets use a parallel stack: AGFHC (AMD's field health check), the ROCm Validation Suite and TransferBench, driven by AMD's Cluster Validation Suite and described in its Customer Acceptance Guide. AMD hands out AGFHC only with its authorization, so put access to it, or vendor-run execution, in the purchase contract before the burn-in plan depends on it. Chapter 13.9 compares the equivalent tooling for NVIDIA, AMD, TPU and Trainium.

The run-level estimates above come from NVIDIA's Hopper table, so measure the installed runtime on your own hardware. NVIDIA's EUD page describes supported profiles inside levels 3 and 4, while the diagnostics matrix does not list EUD. Resolve the installed release, package and device support with the OEM. Measured runtimes also set how many nodes you can test in parallel inside the accepted power and cooling ramp. Keep command and configuration hashes, per-device test names, execution status, skip reasons, durations, errors and raw artifacts. The level-3 NCCL test in DCGM depends on DCGM_NCCL_TESTS_BIN_PATH pointing at the NCCL test binaries; if the path is unset, DCGM skips the test without listing it, so check that it appears as executed in the manifest. A summary PASS with a required memory or interconnect test absent keeps the node on hold. Link each node to Chapter 13.7's port map and fault record: a clean node test cannot release a fabric path that is still on hold.

Soak duration is a project acceptance decision, not an industry constant. Start with the OEM procedures and the contract, write down the workload, thermal and power cycles and the fleet size, then extend or stop based on the measured defect-discovery curve and a statistical stopping rule agreed in advance. Published reference points range from roughly 12 hours (AMD's AGFHC recipes) to SemiAnalysis's recommended minimum of 3–4 weeks of factory burn-in. None of them is a standard or a floor, so the rows below weigh what each window catches against what it costs.

The decision metric is marginal defects found per added test hour, against deferred revenue and residual failure cost. Plot discoveries by failure mode and node-hours, with confidence bounds suited to the fleet size, and keep going while the contract or the agreed stopping rule calls for more test data. A checkpoint-tolerant inference pool and a 50,000-GPU synchronous training pool can rationally run different campaigns, but neither workload label selects a duration by itself. Write the final duration, the pass/re-soak boundary and the vendor-return trigger into the signed test plan and the OEM or support contract.

Where no plan exists yet, the guide suggests a starting campaign built from those reference points. Use factory rack-level burn-in where the integrator offers it. Then run about 2–3 days on site of full-power dense GEMM or HPL plus NCCL collectives across the whole scale-up and scale-out domain, at the production coolant setpoint, with a deep DCGM pass at entry and exit. Extend the soak while new hard failures are still turning up across the cohort in its last 24 hours. A node that falls outside the performance band is a remediation case: pull it, fix it and retest it, rather than lengthening the soak for every node. Stop when the last 24 hours pass clean, every node left in the cohort is inside the band and the contract's rule is satisfied. A new GPU generation or cold-plate design arrives with no failure history, and that is the case for the extended soak of roughly a week in the table above.

High-bandwidth memory earns its own acceptance gate, and the Llama 3 405B data sizes the exposure. Meta's 54-day snapshot of Llama 3 405B training on up to 16,384 H100s logged 466 interruptions, 419 of them unexpected. HBM3 faults caused 72 of the 419 (17.2%). Another 148 were classed as faulty GPUs, the largest single category. The two together account for 220 of the 419. The unspecified GPU failures are not attributed to memory, so the numbers justify dedicated HBM screening without ranking accelerator failure modes universally. The DCGM memtest plugin (-r 4) walks the HBM with memtest86-style patterns to surface hard stuck bits and addressing faults. The more important signal during soak, though, is the ECC error-rate trajectory.

ECC silently corrects single-bit errors, so a node accumulating a high but correctable error rate can look healthy to a smoke test and still need the OEM's investigation and disposition. A criterion of zero uncorrectable errors is necessary but trivial. The one that does the work is a ceiling on correctable-error rate and on row-remapping events over the soak window. A rising correctable rate or remap activity does not by itself predict an inevitable uncorrectable failure. Modern GPUs expose row remapping, which spares out bad memory rows. Hold a node that exhausts or rapidly consumes its spare rows during burn-in out of the production pool until the OEM's diagnostics establish whether HBM repair or RMA is required. If an uncorrectable double-bit error hits a synchronous run, it can show up as an XID, a crashed rank and a checkpoint restart.

NVIDIA publishes the hard edges of that decision in its GPU memory error management guide (updated September 2026). A row-remapping failure meets its RMA criteria once NVIDIA's field diagnostic confirms it. The failure flag sets when a bank that already holds eight remapped uncorrectable-error rows needs another, when an already remapped row fails again, or after 512 uncorrectable-error remaps in total. On Blackwell, the third remap in a bank first triggers HBM channel repair if a spare channel is left; a successful repair returns the GPU to service. Those RMA criteria are framed around uncorrectable-error remaps. NVIDIA gives no ceiling for remapping driven by correctable errors, so setting that limit falls to your contract. DCGM's memory health watch flags a GPU on any volatile double-bit error or any row-remap failure.

During the soak, the XID log is the second thing you act on. NVIDIA's Xid catalog gives each code an immediate action; the burn-in disposition is what your plan adds on top. A starting triage, not an acceptance threshold:

• XID 63 (row remapping recorded): no immediate action; count it toward the node's remap trajectory.
• XID 64 (remapping failure): reset the GPU, then raise it with support (the OEM, if the system came through one). NVIDIA's field diagnostic has to validate the remap failure before RMA applies.
• XID 48 (double-bit ECC error), 94 (contained) and 95 (uncontained memory error): 94 restarts the application and 95 resets the GPU. A 48 needs a GPU reset (drain and reset when it arrives with a 63 or 64), and a 48 on its own calls for NVIDIA's field diagnostic. A repeat on one GPU during burn-in sends the node to quarantine.
• XID 92 (high single-bit ECC rate): no immediate action, but investigate with support; in a soak it is a trajectory flag.
• XID 79 (GPU has fallen off the bus): restart the node and escalate to support; a repeat during burn-in sends the node to quarantine.
• XID 74 (NVLink error on A100 and H100): follow NVIDIA's NVLink error workflow. It uses the register bits in the message to tell an error caused by some other Xid apart from a fault in the link hardware at either end. A GPU reset or node reboot clears it. Escalate to the hardware vendor if it returns after that, and quarantine the node if that happens during burn-in. Blackwell GPUs do not report a 74. Their NVLink faults arrive as Xids 144–150, which NVIDIA decodes in a separate table in the same catalog; give those the same burn-in handling.

In an air-cooled world, thermal screening was a property of the chip. Across the liquid-cooled density ramp from 120 kW racks toward announced 600 kW roadmap systems, it becomes a property of the chip, cold plate, loop and rack position together, so the challenge is attribution. A node that throttles under load might have a marginal GPU, a poorly seated cold plate, an under-flowing quick-disconnect, a partly blocked manifold, or simply a rack position at the warm end of the CDU's loop. The limits are rack-specific: TCS supply stays at the product's supported setpoint, with flow on the CDU schedule (Chapter 13.5 works the per-rack flow for the HPE GB200 NVL72's 115 kW liquid duty). ASHRAE W-classes describe what the facility water (FWS) can supply, and running a rack away from its rated supply point can throttle GPUs. So the thermal acceptance criterion is the contracted performance floor inside the supported temperature, clock and power-mode limits throughout the soak, with thermal throttle reasons distinguished from normal power management. A single throttling node forces a structured attribution before you can RMA the GPU.

Node burn-in and cooling commissioning interlock here. Chapter 5.11 sets the local limits; Chapter 13.5 accepts the cooling loop, CDU and coincident heat duty, and Chapter 13.6 integrates them with power. Liquid banks exercise the loop before real silicon confirms the die and contact path. A GPU under dense GEMM produces a spatially and temporally spiky heat flux that a resistive bank cannot reproduce. That is why a node can pass cooling IST and still throttle under real compute: burn-in is the first time the cooling system meets a true thermal load. When a node throttles, follow the OEM service procedure and compare flow, inlet temperature, power limits and contact condition before replacing the approved GPU/cold-plate assembly and re-soaking. A paired swap alone cannot identify which part failed. Mislabel a cold-plate problem as a GPU defect and you ship a good GPU back to the vendor and re-seat the same bad plate under its replacement.

Hard failures are merciful: they crash the rank, throw an XID and force a restart you can see. Silent data corruption is the adversary. A GPU computes the wrong answer and returns it with no error signal, poisoning gradients or inference outputs while every dashboard stays green. Meta’s 22 July 2025 reliability discussion reports about one SDC fault per thousand devices, with no exposure period. It contrasts this with soft-error-related bit flips and does not count the fraction of machines harboring a defect. Turning a fault-per-device statement into an event clock takes a defined device population, exposure window, detection method and event definition. Size screening from the accepted fleet’s own observations and measured test coverage; a separately reported training campaign does not supply the missing denominators.

How much commissioning compute to spend on silent corruption, versus deferring it to the day-2 fleet-scanner regime, has no free answer, because no burn-in catches every marginal die. Commissioning-time SDC hunting runs deterministic, self-checking workloads across the fleet and flags any node whose arithmetic diverges: known-answer GEMMs, redundant computation with bit-exact comparison, and NCCL collectives verified against a golden reference. It is expensive (compute spent producing no model) and incomplete (SDC is often data-pattern- and temperature-dependent, so a node clean at commissioning can corrupt later). Treat it as a coarse cull that removes the grossly defective dies, and hand the residual, marginal, condition-dependent population explicitly to a continuous day-2 detection program, in the lineage of Meta's Fleetscanner / Ripple / Hardware Sentinel out-of-band and in-band scanning. That hand-off carries a commitment: what you do not catch at acceptance, you must catch in production, or you have chosen to ship silent corruption. Chapter 14.3 develops the taxonomy of hard, transient and silent failures.

A straggler is the most insidious node in a tightly coupled cluster: it works, computes correct answers and throws no errors, but it is slow. In a synchronous collective the slowest participant sets the pace for every other GPU, so one node running 10% slow can slow the entire synchronous job, every rack and every replica, by up to 10% on every step, indefinitely, with no alarm. Stragglers come from the drift the inventory pass was meant to catch (a lower power limit, a degraded NVLink running at reduced lane count, a node thermally throttling at the warm end of a loop) and from defects that appear only under collective load. DCGM's health watches give starting limits for the link counters. On Blackwell and later, a single NVLink link-recovery event trips the watch; earlier generations trip on any NVLink CRC, replay or recovery error. Separate limits flag more than 100 NVLink CRC errors per second and more than 8 PCIe replays within a minute. Commissioning is the cheapest time to find stragglers, because in production a straggler hides inside aggregate throughput metrics and can persist for weeks.

The technique is an offline node-sweep qualification: run identical, fixed workloads (single-GPU compute, then pairwise and ring NCCL collectives) across every node and every link, and flag any node or link whose completion time deviates from the fleet median beyond a tight threshold. The threshold is the contested setting. Too loose and you ship slow nodes that quietly erode goodput; too tight and you RMA nodes for noise and stall the ramp. The published research lineage (Guard and related straggler-detection work) treats this as one problem at two timescales: an offline sweep at commissioning to qualify the fleet, and online straggler detection as a day-2 SLO-burn signal. The acceptance criterion is a per-node performance band, typically a maximum allowed deviation from the fleet median on a reference collective. An illustrative starting band is 5% below the fleet median, the same figure Chapter 13.9 uses for rails; tighten it once you know the run-to-run spread of known-good nodes. A node outside the band is remediated (re-seat, re-flash, re-cable) and re-tested, or rejected. Chapter 13.9 develops the NCCL bandwidth acceptance gates this feeds into at cluster scale, and the fabric link-health baseline it builds on came from Chapter 13.7.

The last acceptance dimension reaches back into the power chain, and it is uniquely an AI-cluster problem. GPU clusters do not draw smooth power. A synchronous training job produces violent synchronized load swings every few hundred milliseconds, as thousands of GPUs move together between compute (full draw) and communication (collective idle). At node scale, burn-in validates that an individual node's power draw, transient behavior and power-capping response match the rated operating range: it holds its rated power limit under sustained load, responds correctly to power-capping commands, and stays within the budget the rack PDU and busbar were sized for. DCGM's level-4 Pulse test already covers the node-level transient: it induces sharp current spikes and fails a node whose power supply cannot cope. Pair it with power-cap response tests. A node that over-draws or ignores a power cap can trip a breaker or push a rack past its provisioned budget, a real risk in a power-bound facility where the rack is sized close to its limit.

Node-level power validation establishes the per-node transient signature, but the aggregate synchronized-swing behavior appears only at cluster scale, when many nodes phase-lock. That is where node burn-in hands back to facility commissioning. Chapter 13.6 covers the gap between load banks and real dynamic load: air-rejecting resistive banks held flat during IST cannot reproduce the reactive, fast-transient, phase-correlated swing of a real collective, so node burn-in (real GPUs, real workloads, real transients) is the first hardware to produce the genuine electrical signature. The staged load ramp of Chapter 13.10 and the proxy training run of Chapter 13.9 are the installed-workload validations of how the BBU/BESS/in-shelf-capacitance mitigation stack behaves under the load dynamics the cluster will actually impose. Accept a node whose power behavior is out of range and you have seeded a future breaker trip or brownout-induced throttle into the production fleet.

Burn-in is done when it produces a signed acceptance artifact that the next phase and the eventual operator inherit, not when the soak timer expires. The package bridges "the nodes survived a week" and "the fleet is qualified to schedule", and it is what separates acceptance from a soak that was merely run. It records: the inventory reconciliation against baseline (every node at approved firmware, driver and link state); the DCGM and OEM results, with executed and skipped test manifests, at soak entry and exit; the DCGM release and IMEX domain state logged for each node; the HBM/ECC trajectory with the correctable-error ceiling, any remapping events and every Xid with its disposition; the thermal record proving the contracted performance floor and supported operating range under load; the SDC-hunt results and the explicit hand-off to the day-2 detection regime; the straggler node-sweep with the per-node performance band; the power-behavior range; and the disposition of every node that failed: what it failed, what was remediated, what was RMA'd, and the re-soak result for anything repaired.

This artifact is the per-node counterpart of the commissioning documentation discipline of Chapter 13.2. It feeds two downstream gates. The first is the cluster-scale benchmarking and reference training run of Chapter 13.9, which assumes a fleet of qualified nodes and will expose any straggler or SDC the node-level cull missed. The second is the day-2 reliability program seeded at handover in Chapter 13.10, which inherits the burn-in baselines as the reference for measuring fleet drift. A node that was never burned in has no baseline, so the day-2 program has nothing to detect degradation against, and the first time anyone learns the node is marginal is when it corrupts a training run.

Report the original cohort denominator and first-pass yield separately from repaired-node yield. Repaired nodes start a new qualification exposure after the intervention, and failed and censored hours should be kept by failure mode rather than pooled into a cleaner-looking final fleet. Under an explicitly justified constant-rate Poisson model with complete detection, zero failures over exposure E give a one-sided upper rate bound −ln(α)/E at confidence 1−α (NIST). That bound describes the node-hour population and conditions tested. Infant mortality, common causes and unseen SDC coverage do not inherit it. Chapter 12.5 develops the reliability model.

Release only the cohort whose required diagnostics actually executed, whose repaired members have fresh results, and whose remaining node and port holds are excluded from scheduling. Extending a soak without fixing its missing test coverage costs GPU-hours and leaves the same defect invisible; releasing early makes the first customer job pay for that blind spot.