Component Failure Modes, Failure Rates & Fleet Reliability Data

This is the canonical home for one uncomfortable fact: an AI training cluster is the least reliable large machine humans operate at scale, and it is supposed to be. A traditional enterprise data center measures uptime in nines and counts annual outages on one hand. A 16,384-GPU training cluster experiences an unplanned interruption roughly every three hours — Meta's published Llama 3 405B run logged 419 of them over 54 days — and a well-run fleet still delivers over 90% effective training time through automation, not through preventing the failures (Meta, Llama 3 Herd, 2024). At this density and scale, the per-component physics guarantees components fail; the reliability problem is measuring the rate precisely enough to size spares, model availability, and detect the failures that hide.

This chapter establishes three things every other reliability chapter in the guide depends on. First, the failure taxonomy — hard, transient, and silent — which is the canonical fault vocabulary cross-referenced from the GPU operations view in Chapter 10.7, the redundancy view in Chapter 12.1, and the IST failure-demonstration view in Chapter 13.6. Second, the empirical fleet failure-rate data — the actual published numbers from at-scale operators, with their vintages and caveats, that you plug into an availability model rather than inventing. Third, the AFR modeling and burn-in discipline that turns raw component failure rates into a spares forecast and an acceptance gate. Every AFR in this chapter feeds the cluster availability and goodput roll-up in Chapter 12.5; the consolidated FMEA catalog these modes populate lives in Appendix F.

Every fault in a GPU fleet falls into one of three classes, and each demands completely different detection and recovery machinery; confusing them is a common operational error. The taxonomy organizes everything downstream: what you instrument, how fast you must react, and whether the failure is even visible at all.

Hard failures are the easy ones, paradoxically, because they announce themselves. A GPU throws an uncorrectable XID and falls off the PCIe bus, an optical transceiver goes dark, a CDU trips on low flow, a power supply faults. The component is unambiguously dead or unreachable; the job crashes or the node drops out; the telemetry screams. The XID 79 'GPU has fallen off the bus' event — which affects roughly 3.2% of H100s in their first year (Last9 / Introl fleet analyses, 2025) — is the archetypal hard failure. These are expensive in lost goodput but cheap to detect: the recovery path is fail-fast, drain, restart from checkpoint, swap the FRU. Hard failures are where mature operators are already good, because the signal is loud.

Transient failures are the treacherous middle. A correctable ECC error storm on HBM, a link that flaps rather than goes down, a thermal excursion that throttles a GPU for ninety seconds, an XID that self-clears on reset. The component is not dead — it works again after a power-cycle or a few minutes — but it degraded the run while it misbehaved, and it will very likely do so again. Transients are the raw material of lemon nodes: hardware that passes every point-in-time health check yet fails repeatedly under load. The decision a transient forces is not 'is it broken' but 'is it broken often enough to eject' — and getting that threshold wrong either keeps a repeat-offender poisoning runs or ejects healthy capacity. Crucially, Meta found that single-bit ECC error trends predict hard GPU failure 48–72 hours in advance with 89–96% accuracy (Meta fleet research, 2025) — the transient is frequently the early-warning tremor before the hard quake, which is why instrumenting transients is the highest-leverage detection investment most fleets under-fund.

Silent failures are the ones that should keep you up at night, because by definition nothing screams. Silent data corruption (SDC) is a computational error — a multiply that returns the wrong product, a memory read that flips a bit undetected by ECC — that produces no fault, no XID, no log line. The hardware reports success. The math is wrong. In training, an SDC silently corrupts gradients and weights; the loss curve drifts or diverges days later, and you cannot tell whether it is a bad hyperparameter, a data bug, or a single faulty multiplier in one of a hundred thousand chips. The historical SDC rate was roughly 1 per million devices — a cosmic-ray-grade rarity nobody budgeted for. At current process nodes and scale it has risen to roughly 1 per 1,000 silicon devices, which Meta attributes to fundamental silicon manufacturing variation, not particle effects (Meta, How Meta keeps its AI hardware reliable, July 2025). On a 100,000-device fleet, 1-in-1,000 is not an edge case — it is a near-certainty on any multi-day run, and the only defense is a deliberate detection program.

Fleet-level AFR is an aggregate that hides a strongly skewed distribution: a handful of components dominate the failure budget, and knowing which ones lets you target spares, burn-in, and detection where they pay off. The Meta Llama 3 root-cause breakdown is the most-cited public dataset, and its shape generalizes across operators even if your absolute numbers differ.

The GPU and its HBM dominate. In the Llama 3 run, faulty GPUs caused 30.1% of interruptions and HBM3 a further 17.2% — together nearly 59% of all failures were GPU/HBM-related, and roughly 78% were confirmed hardware (Meta, 2024). This is not surprising once you see the physics: the accelerator package is the densest, hottest, highest-current component in the rack, and HBM stacks are the most thermally and mechanically stressed memory ever shipped at volume. HBM is also temperature-sensitive in a way that compounds the density-ramp: HBM error rates roughly double per ~5 °C above ~75 °C junction (commissioning/thermal guidance, 2025) — so a cooling system that runs warm does not just risk throttling, it directly inflates your memory failure rate. Every kilowatt you add per rack in the Chapter 1.2 density ramp pushes this term up unless coolant temperature holds.

Network and optics are the persistent long tail. Switches and cables accounted for 8.4% of Llama 3 interruptions, and link-flaps are as damaging as hard-down links because they corrupt collectives without obviously failing. At 800G XDR and the optics densities of a rail-optimized fabric, transceiver and cable failures scale with link count — a 100k-GPU cluster has millions of optical links, and even an excellent per-link AFR multiplies into a steady drip of fabric faults. The fabric is the failure domain that grows fastest as you scale out.

Infrastructure failures are rarer but higher-impact. Power and cooling faults are far less frequent than GPU faults per-event, but a single CDU trip or a PDU fault can take down an entire rack or pod at once — converting one component failure into dozens of simultaneous node losses. The Uptime data is stark: power is implicated in roughly 45% of impactful data-center outages (mostly UPS), and human error is behind 70–80% of all outages, 85% of those traced to process failures (Uptime Institute, 2025). The lesson for AI fleets is that the GPU dominates frequency while infrastructure dominates blast radius — and your FMEA in Appendix F must weight both.

Here is the result that reframes every reliability conversation in AI infrastructure, and the one most newcomers get wrong: mean time between failures is not a property of your hardware — it is a property of your scale. If a single 8-GPU node has a mean time to failure of ~47.7 days, then a synchronous job that depends on all GPUs staying up sees a cluster MTBF that drops roughly in proportion to GPU count. The observed numbers are stark (these are empirical anchors that include software-induced failures, not a clean arithmetic chain — they do not divide exactly from the single-node base rate):

• 8 GPUs: ~47.7 days between failures.
• 1,024 GPUs: ~7.9 hours.
• 16,384 GPUs: ~1.8 hours (consistent with Llama 3's observed ~3-hour cadence including software).
• 131,072 GPUs: ~0.23 hours — roughly one failure every 14 minutes (SemiAnalysis / Meta-derived scaling, 2024–25).

This is the GOODPUT thread made concrete. At 100k+ GPUs, the cluster is never fully healthy — something is always failing — so the design target shifts from 'maximize uptime' to 'maximize the fraction of GPU-hours doing useful work despite a continuous failure stream.' It is also why the checkpoint interval must shrink as you scale: a 100k-GPU run needs roughly 2-minute checkpoint intervals to hit 0.9 ETTR, where a 16k-GPU run is comfortable at 5 minutes (the optimal-interval math is canonical in Chapter 9.4; operational tuning in Chapter 14.4). The scale law is the bridge between component AFR and cluster availability: the same per-GPU failure rate is a non-event at 8 GPUs and an existential design constraint at 100k.

Because SDC by definition leaves no log line, detecting it is an active program, not a passive alarm — and the state of the art is a layered defense, with each layer trading coverage against the GPU-hours it steals from production. Meta's published stack is the reference architecture the rest of the industry is converging on.

Fleetscanner is the offline sweep: dedicated silicon test patterns scheduled across the fleet so the entire estate is covered every 45–60 days. It is the most thorough layer (over three years it reached ~93% coverage for a major defect family, with ~23% unique coverage no other method caught) but the most expensive, because the GPU under test is not earning revenue. Ripple co-locates with live workloads, slipping millisecond-to-second test bursts into the gaps between real work, so it achieves fleet-wide coverage in days rather than weeks at near-zero opportunity cost — at lower per-pass depth. Hardware Sentinel is the newest layer and the cleverest: it watches application exceptions in kernel space and infers core-level SDC without allocating any test time at all, raising effective coverage roughly 1.74x over Fleetscanner and 1.92x over Ripple (ASPLOS 2025). The architectural lesson is that no single method suffices — you layer a deep-but-slow sweep, a fast-but-shallow in-workload probe, and a zero-cost inference layer, and the union catches what any one misses.

For training specifically, the framework-level defenses matter as much as the fleet-level ones: redundant computation on a sample of operations, gradient/activation checksums, and divergence monitors that flag when a replica's numerics drift from its peers. These catch SDC at the moment it corrupts the math rather than 45 days later in a sweep. The decision here mirrors the funding fork above — every layer you add costs GPU-hours or engineering, and the right depth is set by how catastrophic a silently-corrupted run is for your business. A frontier lab burning months of compute on one run buys all the layers; a batch-inference shop running idempotent, re-runnable jobs may rationally buy none.

Component failure rates are not constant over life — they follow the classic bathtub: a high infant-mortality phase early, a low flat useful-life phase, and a rising wear-out phase late. Burn-in is the deliberate decision to pull infant mortality forward into a controlled acceptance window so it lands on a test harness instead of a customer's training run. This is the central fork at go-live, and it is a direct schedule-versus-reliability trade.

The mechanics: 72–168 hours of sustained, thermally-stressful load — GPU-Burn, NCCL all-reduce loops, HBM stress patterns, FIO storage soak — at or near max TDP, with DCGM diagnostics (the dcgmi diag -r 4 deep level is the standard acceptance gate) bracketing the run. Vendors claim burn-in removes on the order of 98% of infant-mortality failures within the warranty period, and a fresh cluster's MTBF during its first 3–4 weeks is dramatically worse than its mature ~7-days-per-512-GPU steady state — which is precisely the infant-mortality phase you are trying to flush before acceptance. The acceptance criteria, levels, and tooling are the subject of the commissioning chapters (Chapter 13.6 covers Level 5 IST and failure-mode demonstration); here the point is the economic fork.

Burn-in does not end at acceptance — it transitions into a steady-state cadence. The day-2 discipline is a weekly deep node-health pass (dcgmi diag -r 3 plus NCCL on idle GPUs) and the continuous straggler/lemon detection that watches for the repeat-offenders burn-in could not catch. A node that runs ~15% below a golden-reference benchmark is auto-flagged for quarantine, and the lemon-ejection decision — proven to cut 512+-GPU job failure rates from ~14% to ~4% and lift completion ~30% (Meta lemon-node studies, 2024) — is where transient-failure data becomes reclaimed goodput. Burn-in front-loads the cost of infant mortality; lemon ejection back-stops the transients that slip through. Both are detection programs paid for in GPU-hours, and both are justified by the same arithmetic: at scale, the goodput you reclaim dwarfs the capacity you spend finding it.