Component Failure Modes, Failure Rates & Fleet Reliability Data

This chapter is the canonical home for one uncomfortable fact: a large AI training cluster must expect component interruptions during a run and recover from them. A traditional enterprise data center measures uptime in nines and counts annual outages on one hand. Meta's Llama 3 405B pre-training on up to 16,384 H100s logged 419 unexpected interruptions (466 in total) over a 54-day snapshot, roughly one every three hours. Even so, a well-run fleet delivers over 90% effective training time, through automation rather than by preventing the failures (Meta, Llama 3 Herd, 2024). At this density and scale, component physics guarantees failures. The reliability problem is measuring the rate precisely enough to size spares, model availability, and detect the failures that hide.

Three things here support every other reliability chapter in the guide. First, the failure taxonomy (hard, transient and silent), the fault vocabulary reused in the GPU operations view of Chapter 10.7, the redundancy view of Chapter 12.1, and the IST failure-demonstration view of Chapter 13.6. Second, the empirical fleet failure-rate data: published numbers from at-scale operators, with their vintages and caveats, to plug into an availability model instead of inventing your own. Third, the AFR modeling and burn-in discipline that turns raw component failure rates into a spares forecast and an acceptance gate. Every AFR here feeds the cluster availability and goodput roll-up in Chapter 12.5. The consolidated FMEA catalog these modes populate lives in Appendix F.

Every fault in a GPU fleet falls into one of three classes, and each needs completely different detection and recovery machinery. Confusing them is a common operational error. The taxonomy decides what you instrument, how fast you must react, and whether the failure is visible at all.

Hard failures are the easy ones, paradoxically, because they announce themselves. A GPU throws an uncorrectable XID and falls off the PCIe bus, an optical transceiver goes dark, a CDU trips on low flow, a power supply faults. The component is unambiguously dead or unreachable, the job crashes or the node drops out, and the telemetry screams. XID 79 ('GPU has fallen off the bus') is an archetypal hard-failure signal, but no published fleet prevalence for it transfers to another fleet, so the guide gives none. Hard failures are expensive in lost goodput but cheap to detect. Recovery is fail-fast, drain, restart from checkpoint, swap the FRU. Mature operators are already good at this, because the signal is loud.

Transient failures are the hard middle case. Examples are a correctable ECC error storm on HBM, a link that flaps instead of going down, a thermal excursion that throttles a GPU for ninety seconds, and an XID that self-clears on reset. The component is not dead. It works again after a power-cycle or a few minutes, but it degraded the run while it misbehaved and will very likely do so again. Transients are the raw material of lemon nodes: hardware that passes every point-in-time health check yet fails repeatedly under load. The question a transient forces is not 'is it broken' but 'is it broken often enough to eject'. Get the threshold wrong and you either keep a repeat offender poisoning runs or eject healthy capacity. Correctable-ECC trends are one degradation signal; combine them with XIDs, row-remap state, thermals and load-correlated recurrence before you decide to eject hardware.

Silent failures are the dangerous class, because by definition nothing screams. Silent data corruption (SDC) is a computational error that produces no fault, no XID and no log line: a multiply that returns the wrong product, or a memory read that flips a bit undetected by ECC. The hardware reports success. The math is wrong. In training, an SDC silently corrupts gradients and weights, and the loss curve drifts or diverges days later. You cannot tell whether the cause is a bad hyperparameter, a data bug, or one faulty multiplier in one of a hundred thousand chips. Meta’s 22 July 2025 reliability discussion puts SDC at about one fault per thousand devices, and contrasts it with soft-error bit flips, a different error class. Meta gives no exposure period, so on a 100,000-device fleet the figure cannot tell you how many machines are defective or the odds of corruption during a multi-day run. Build a deliberate detection program, know which instructions, data patterns and operating conditions it covers, and budget both the GPU-hours for screening and the trusted progress lost before detection.

Fleet-level AFR is an aggregate that hides a strongly skewed distribution. A handful of components dominate the failure budget, and knowing which ones lets you aim spares, burn-in and detection where they pay off. The Meta Llama 3 root-cause breakdown is a useful public job-interruption dataset, but neither its counts nor its category mix is a portable equipment failure-rate distribution.

The GPU and its HBM dominate. In the Llama 3 paper's Table 5, 148 interruptions were attributed to faulty GPUs and 72 to HBM3, among 419 unexpected interruptions in a 54-day snapshot of one run on up to 16,384 H100s. These are event counts, not unique failed devices or component exposure. The paper prints 30.1% for the faulty-GPU row although 148/419 is 35.3%, so use the counts. The physics explains the share: the accelerator package is the densest, hottest, highest-current component in the rack, and HBM stacks are the most thermally and mechanically stressed memory ever shipped at volume. Trend HBM health jointly with junction temperature, workload, ECC events and remap history. As rack power rises, preserve thermal margin through the coolant-temperature, flow and cold-plate limits you choose in Chapter 1.2.

Network and optics are the persistent long tail. Switches and cables accounted for 8.4% of Llama 3 interruptions. Link flaps do as much damage as hard-down links, because they corrupt collectives without obviously failing. At 800G XDR and the optics densities of a rail-optimized fabric, transceiver and cable failures scale with link count. At 100,000 GPUs the back-end fabric needs hundreds of thousands of links, about one per GPU at each switching tier (more where each GPU's NIC is split across several planes), and most of them are optical. Even an excellent per-module AFR turns that many links into a steady drip of fabric faults. The fabric is the failure domain that grows fastest as you scale out.

Infrastructure failures are rarer but higher-impact. Power and cooling faults are far less frequent than GPU faults, event for event. But a single CDU trip or PDU fault can take down an entire rack or pod at once, turning one component failure into dozens of simultaneous node losses. Uptime Institute's survey behind its 2026 Annual Outage Analysis is stark: asked to name the primary cause of their most recent impactful outage in 2025, 45% of respondents said power, down from 54% in 2024. When operators trace a power outage to a cause, the UPS is the one they name most often. Uptime's 2025 human-error findings use a narrower denominator. Of human-error outages, 58% involved failure to follow an established procedure, and roughly 85% involved either that behavior or a flawed procedure. Neither figure is a share of all serious outages. For AI fleets, the GPU dominates frequency while infrastructure dominates blast radius, and your FMEA in Appendix F must weight both.

The first-order law comes from Meta's research clusters. A synchronous job stops when any node it holds fails, so if node failures are independent, the job fails at the sum of its nodes' rates: MTTF ≈ 1 / (nodes in the job × failure rate per node). Meta measured 6.50 job failures per 1,000 allocated node-days on RSC-1 and 2.34 on RSC-2, two A100 research clusters, counting node failures and health-check-attributed failures on jobs larger than 128 GPUs. At RSC-1's rate a 16,384-GPU job, 2,048 eight-GPU nodes, fails about 13 times a day: a projected MTTF of about 1.8 hours. Meta reports that this rate-based projection matches observed MTTFs for jobs larger than 32 GPUs; on RSC-1 it closely tracks jobs of 4 to 512 eight-GPU servers. Smaller jobs scatter around it, mostly because experimental use produces correlated node failures. Across its job sizes, observed MTTF falls by roughly two orders of magnitude, from about 47.7 days for an 8-GPU job to about 7.9 hours for a 1,024-GPU job (Meta, Revisiting Reliability in Large-Scale ML Research Clusters, arXiv 2410.21680, 2024–25).

The formula is a first-order estimate, and it breaks in predictable places. Correlated faults violate independence: a CDU or PDU trip takes a whole rack, a fabric event hits many links at once, and a bad firmware rollout lands everywhere together. Software failures depend on code, configuration and job behavior, not on node count alone. Llama 3 traced 12.9% of its interruptions to software. Detection policy decides what counts as a failure at all. So the published anchors sit on different curves. Meta's Llama 3 405B run logged 419 unexpected interruptions over 54 days on up to 16,384 H100s, about one every three hours, against a 1.8-hour projection for the same GPU count from a different cluster, generation and event definition. SemiAnalysis (October 2024) reports that a top-tier operator typically sees about seven days MTBF on a 512-H100 cluster, a third population again. Fleet maturity, topology, job membership, software and the interruption denominator all differ, so none of them is a point you can read your own fleet off.

The stable lesson about scale is directional: a larger synchronous job participates in more failure domains, so recovery and goodput (the share of wall-clock time that advances the job) matter more as the job grows. Fit the effective job-level failure distribution from fleet telemetry, then compute checkpoint cadence from that measured distribution and the actual save/restart cost. The checkpoint math is canonical in Chapter 9.4; operational tuning is in Chapter 14.4.

SDC leaves no log line by definition, so detecting it is an active program, not a passive alarm. The state of the art is a layered defense, and each layer trades coverage against the GPU-hours it takes from production. Meta's published CPU-fleet stack shows the tradeoff; it does not establish GPU-fleet coverage.

Fleetscanner is the offline sweep: dedicated CPU test patterns scheduled across Meta’s fleet on a reported 45–60-day CPU sweep cadence. It is the most thorough layer (over three years it reached ~93% coverage for a major defect family, with ~23% unique coverage no other method caught) and the most expensive, because the CPU under test cannot run its production workload. Ripple co-locates with live workloads and slips millisecond-to-second test bursts into the gaps between real work. It reaches fleet-wide coverage in days rather than weeks at near-zero opportunity cost, but with less depth per pass. Hardware Sentinel is the newest layer. It watches application exceptions in kernel space and infers core-level SDC without allocating any test time at all. In the studied population (ASPLOS 2025) it reported effective CPU defect coverage roughly 1.74x over Fleetscanner and 1.92x over Ripple. The lesson is that no single method suffices. Layer a deep-but-slow sweep, a fast-but-shallow in-workload probe and an application-exception inference layer, with their monitoring and validation costs, and the union catches what any one misses.

For training specifically, framework-level defenses matter as much as fleet-level ones: redundant computation on a sample of operations, gradient and activation checksums, and divergence monitors that flag when a replica's numerics drift from its peers. On the paths they cover, these can catch SDC while the computation runs; they will not catch every wrong result. The decision mirrors the funding fork above: every layer you add costs GPU-hours or engineering, and the right depth depends on how catastrophic a silently corrupted run is for your business. A frontier lab burning months of compute on one run has a strong reason to fund layered detection. A batch-inference shop can buy less only if output checks and replay contain silent errors before they escape.

Component failure rates are not constant over life. They follow the classic bathtub: a high infant-mortality phase early, a low flat useful-life phase, and a rising wear-out phase late. Burn-in is the decision to pull infant mortality forward into a controlled acceptance window, so it lands on a test harness instead of a customer's training run. It is a direct schedule-versus-reliability trade, settled at go-live.

Published reference points run from hours to weeks, and the table below sets them side by side. AMD's Kubernetes reference architecture for Instinct publishes two AGFHC burn-in recipes, and its health-check design holds every new node for a 24-hour burn-in; AGFHC is not public, and AMD must authorize access. SemiAnalysis's October 2024 playbook pairs weeks of high-temperature burn-in at the integration factory with days of on-site acceptance. Its ClusterMAX 3.0 ratings (September 2026) open each rented-cluster evaluation with an 8-hour GPU-and-network burn-in, a test for rating clouds rather than an operator's acceptance soak. Whatever the length, load every node to full power with dense GEMM or HPL, drive NCCL collectives over every scale-up and scale-out link with coolant at its production setpoint, and bracket the campaign with a deep DCGM pass at entry and exit. That DCGM pass feeds the acceptance decision; it does not make it alone. Chapter 13.8 sets the duration and the stopping rule from the OEM procedure, the contract and the measured defect-discovery curve. Unless a campaign publishes its tested population, protocol and warranty interval, treat it as an acceptance gate, not a quantified reliability guarantee.

Burn-in does not end at acceptance. It becomes a steady-state cadence. One possible day-2 schedule pairs a weekly deep node-health pass (dcgmi diag -r 3 plus NCCL on drained GPUs) with continuous straggler and lemon detection, which watches for the repeat offenders burn-in could not catch. The actual cadence follows supported diagnostics, fleet evidence and the maintenance budget. A site may investigate a node running ~15% below a golden-reference benchmark, as an illustrative trigger, after matching workload, power cap, temperature and software. Automatic quarantine needs a validated threshold and a false-positive budget.

The lemon-ejection decision is where transient-failure data becomes reclaimed goodput. In one Meta RSC deployment it cut failures of 512+-GPU jobs from ~14% to ~4%, and Meta estimates it can raise large-job completion by over 30% (arXiv 2410.21680, 2024). Burn-in front-loads the cost of infant mortality; lemon ejection back-stops the transients that slip through. Both are detection programs paid for in GPU-hours, and both need the same accounting: compare reclaimed goodput with diagnostic GPU-hours, false quarantines and engineering cost in the useful-output accounting of Chapter 14.1.