Component Failure Modes, Failure Rates & Fleet Reliability Data

This is the canonical home for one uncomfortable fact: an AI training cluster is the least reliable large machine humans operate at scale, and it is supposed to be. A traditional enterprise data center measures uptime in nines and counts annual outages on one hand. A 16,384-GPU training cluster experiences an unplanned interruption roughly every three hours — Meta's published Llama 3 405B run logged 419 of them over 54 days — and a well-run fleet still delivers over 90% effective training time through automation, not through preventing the failures (Meta, Llama 3 Herd, 2024). At this density and scale, the per-component physics guarantees components fail; the reliability problem is measuring the rate precisely enough to size spares, model availability, and detect the failures that hide.

This chapter establishes three things every other reliability chapter in the guide depends on. First, the failure taxonomy — hard, transient, and silent — which is the canonical fault vocabulary cross-referenced from the GPU operations view in Chapter 10.7, the redundancy view in Chapter 12.1, and the IST failure-demonstration view in Chapter 13.6. Second, the empirical fleet failure-rate data — the actual published numbers from at-scale operators, with their vintages and caveats, that you plug into an availability model rather than inventing. Third, the AFR modeling and burn-in discipline that turns raw component failure rates into a spares forecast and an acceptance gate. Every AFR in this chapter feeds the cluster availability and goodput roll-up in Chapter 12.5; the consolidated FMEA catalog these modes populate lives in Appendix F.

Every fault in a GPU fleet falls into one of three classes, and each demands completely different detection and recovery machinery; confusing them is a common operational error. The taxonomy organizes everything downstream: what you instrument, how fast you must react, and whether the failure is even visible at all.

Hard failures are the easy ones, paradoxically, because they announce themselves. A GPU throws an uncorrectable XID and falls off the PCIe bus, an optical transceiver goes dark, a CDU trips on low flow, a power supply faults. The component is unambiguously dead or unreachable; the job crashes or the node drops out; the telemetry screams. XID 79 ('GPU has fallen off the bus') is an archetypal hard-failure signal, but this guide does not assign it a portable fleet prevalence without a named equipment population, exposure window and study. These are expensive in lost goodput but cheap to detect: the recovery path is fail-fast, drain, restart from checkpoint, swap the FRU. Hard failures are where mature operators are already good, because the signal is loud.

Transient failures are the hard middle case. A correctable ECC error storm on HBM, a link that flaps rather than goes down, a thermal excursion that throttles a GPU for ninety seconds, an XID that self-clears on reset. The component is not dead — it works again after a power-cycle or a few minutes — but it degraded the run while it misbehaved, and it will very likely do so again. Transients are the raw material of lemon nodes: hardware that passes every point-in-time health check yet fails repeatedly under load. The decision a transient forces is not 'is it broken' but 'is it broken often enough to eject' — and getting that threshold wrong either keeps a repeat-offender poisoning runs or ejects healthy capacity. Crucially, Meta found that single-bit ECC error trends predict hard GPU failure 48–72 hours in advance with 89–96% accuracy (Meta fleet research, 2025) — the transient is frequently the early-warning tremor before the hard quake, which is why instrumenting transients is the highest-leverage detection investment most fleets under-fund.

Silent failures are the dangerous class, because by definition nothing screams. Silent data corruption (SDC) is a computational error — a multiply that returns the wrong product, a memory read that flips a bit undetected by ECC — that produces no fault, no XID, no log line. The hardware reports success. The math is wrong. In training, an SDC silently corrupts gradients and weights; the loss curve drifts or diverges days later, and you cannot tell whether it is a bad hyperparameter, a data bug, or a single faulty multiplier in one of a hundred thousand chips. The historical SDC rate was roughly 1 per million devices — a cosmic-ray-grade rarity nobody budgeted for. At current process nodes and scale it has risen to roughly 1 per 1,000 silicon devices, which Meta attributes to fundamental silicon manufacturing variation, not particle effects (Meta, How Meta keeps its AI hardware reliable, July 2025). On a 100,000-device fleet, 1-in-1,000 is not an edge case — it is a near-certainty on any multi-day run, and the only defense is a deliberate detection program.

Fleet-level AFR is an aggregate that hides a strongly skewed distribution: a handful of components dominate the failure budget, and knowing which ones lets you target spares, burn-in, and detection where they pay off. The Meta Llama 3 root-cause breakdown is a useful public job-interruption dataset, but neither its counts nor its category mix is a portable equipment failure-rate distribution.

The GPU and its HBM dominate. In the Llama 3 paper's Table 5, 148 interruptions were attributed to faulty GPUs and 72 to HBM3 among 419 unplanned interruptions on one 16,384-H100, 54-day run. These are event counts, not unique failed devices or component exposure; the paper's printed percentages are internally inconsistent, so use the counts and preserve the named job boundary. This is not surprising once you see the physics: the accelerator package is the densest, hottest, highest-current component in the rack, and HBM stacks are the most thermally and mechanically stressed memory ever shipped at volume. HBM is also temperature-sensitive in a way that compounds the density-ramp: HBM error rates roughly double per ~5 °C above ~75 °C junction (commissioning/thermal guidance, 2025) — so a cooling system that runs warm does not just risk throttling, it directly inflates your memory failure rate. Every kilowatt you add per rack in the Chapter 1.2 density ramp pushes this term up unless coolant temperature holds.

Network and optics are the persistent long tail. Switches and cables accounted for 8.4% of Llama 3 interruptions, and link-flaps are as damaging as hard-down links because they corrupt collectives without obviously failing. At 800G XDR and the optics densities of a rail-optimized fabric, transceiver and cable failures scale with link count — a 100k-GPU cluster has millions of optical links, and even an excellent per-link AFR multiplies into a steady drip of fabric faults. The fabric is the failure domain that grows fastest as you scale out.

Infrastructure failures are rarer but higher-impact. Power and cooling faults are far less frequent than GPU faults per-event, but a single CDU trip or a PDU fault can take down an entire rack or pod at once — converting one component failure into dozens of simultaneous node losses. The Uptime data is stark: power is implicated in roughly 45% of impactful data-center outages (mostly UPS), and Uptime Institute's 2025 human-error survey findings use narrower denominators: 58% of human-error outages involved failure to follow an established procedure, while roughly 85% involved either that behavior or a flawed procedure. Neither figure is a share of all serious outages. The lesson for AI fleets is that the GPU dominates frequency while infrastructure dominates blast radius — and your FMEA in Appendix F must weight both.

Job interruption cadence belongs to a named source population and event definition, not to GPU count alone. Meta measured 419 unplanned interruptions over 54 days while training Llama 3 405B on 16,384 H100s — about one every three hours for that run, including hardware and software causes. A separate SemiAnalysis reference reports roughly seven days MTBF for one 512-H100 cluster at a top-tier operator. These are useful anchors, but they are not points on one arithmetic curve: fleet maturity, topology, job membership, correlated faults, software, detection policy, and the interruption denominator differ.

The stable scale lesson is directional: a larger synchronous job participates in more failure domains, so recovery and GOODPUT matter more as the job grows. Fit the effective job-level failure distribution from fleet telemetry, then compute checkpoint cadence from that measured distribution and the actual save/restart cost; do not infer a universal two-minute interval or a minute-level failure forecast from accelerator count. The checkpoint math is canonical in Chapter 9.4; operational tuning is in Chapter 14.4.

Because SDC by definition leaves no log line, detecting it is an active program, not a passive alarm — and the state of the art is a layered defense, with each layer trading coverage against the GPU-hours it steals from production. Meta's published stack is the reference architecture the rest of the industry is converging on.

Fleetscanner is the offline sweep: dedicated silicon test patterns scheduled across the fleet so the entire estate is covered every 45–60 days. It is the most thorough layer (over three years it reached ~93% coverage for a major defect family, with ~23% unique coverage no other method caught) but the most expensive, because the GPU under test is not earning revenue. Ripple co-locates with live workloads, slipping millisecond-to-second test bursts into the gaps between real work, so it achieves fleet-wide coverage in days rather than weeks at near-zero opportunity cost — at lower per-pass depth. Hardware Sentinel is the newest layer: it watches application exceptions in kernel space and infers core-level SDC without allocating any test time at all, raising effective coverage roughly 1.74x over Fleetscanner and 1.92x over Ripple (ASPLOS 2025). The architectural lesson is that no single method suffices — you layer a deep-but-slow sweep, a fast-but-shallow in-workload probe, and a zero-cost inference layer, and the union catches what any one misses.

For training specifically, the framework-level defenses matter as much as the fleet-level ones: redundant computation on a sample of operations, gradient/activation checksums, and divergence monitors that flag when a replica's numerics drift from its peers. These catch SDC at the moment it corrupts the math rather than 45 days later in a sweep. The decision here mirrors the funding fork above — every layer you add costs GPU-hours or engineering, and the right depth is set by how catastrophic a silently-corrupted run is for your business. A frontier lab burning months of compute on one run buys all the layers; a batch-inference shop running idempotent, re-runnable jobs may rationally buy none.

Component failure rates are not constant over life — they follow the classic bathtub: a high infant-mortality phase early, a low flat useful-life phase, and a rising wear-out phase late. Burn-in is the deliberate decision to pull infant mortality forward into a controlled acceptance window so it lands on a test harness instead of a customer's training run. It is a direct schedule-versus-reliability trade, settled at go-live.

The campaign is selected from the signed OEM/project/contract plan: declared stress profile and cycles, fleet exposure, measured discoveries by failure mode and node-hours, pass/re-soak dispositions, and a pre-agreed statistical stopping rule. Together AI and ClusterMAX described 72–168 hours in 2025 practitioner guidance; that is an example, not a universal bound. DCGM diagnostics can be components of the evidence package, but no run level alone is a universal acceptance gate. A separate October 2024 SemiAnalysis playbook recommends at least 3–4 weeks of factory high-temperature burn-in before deployment; it does not report a universal early-failure settling period. Remove the unsupported ~98% claim unless the tested population, protocol, warranty interval, and source locator are supplied.

Burn-in does not end at acceptance — it transitions into a steady-state cadence. The day-2 discipline is a weekly deep node-health pass (dcgmi diag -r 3 plus NCCL on idle GPUs) and the continuous straggler/lemon detection that watches for the repeat-offenders burn-in could not catch. A node that runs ~15% below a golden-reference benchmark is auto-flagged for quarantine, and the lemon-ejection decision — proven to cut 512+-GPU job failure rates from ~14% to ~4% and lift completion ~30% (Meta lemon-node studies, 2024) — is where transient-failure data becomes reclaimed goodput. Burn-in front-loads the cost of infant mortality; lemon ejection back-stops the transients that slip through. Both are detection programs paid for in GPU-hours, and both are justified by the same arithmetic: at scale, the goodput you reclaim dwarfs the capacity you spend finding it.