Fleet Reliability, Fault Tolerance & Autonomous Recovery

A larger synchronous job spans more component and service failure domains, so interruptions become a first-order goodput constraint. Meta reported about 47.7 days MTTF for an 8-GPU job and 7.9 hours for a 1,024-GPU job in its research clusters. Those observations show the direction of scale. They do not justify a universal cadence for a six-figure-GPU job; that would need fleet telemetry and explicit assumptions about independence and common-mode failures, job membership, software and how an event is defined. Here MTTF and MTBF both mean the job-level interval between interruptions, not the reliability of any single component. Fleet reliability is therefore an engineering discipline of its own, not a facility-uptime line item.

This chapter covers the system that turns that failure rate from a goodput catastrophe into a manageable tax. Three parts must be designed together: the detection-to-recovery loop (how fast you notice, isolate and restart), the fault-tolerance strategy (hot spares, elastic or redundant training, algorithmic resilience) and the checkpoint/restore substrate that decides how much work a failure erases. A reliability dollar can go to facility nines, recovery speed or spare capacity, and the three buy very different amounts of goodput. The supporting math lives elsewhere: checkpoint intervals in Chapter 9.4, the failure taxonomy and fleet AFR data in Chapter 14.3, and the availability-versus-goodput rethink in Chapter 12.2.

Two facts combine to make the modern training-reliability problem. First, synchronous coupling makes every node a single point of failure: in a data-, tensor- or pipeline-parallel run the job moves at the speed of its slowest rank, and a dead rank halts all of them. Second, the number of failure domains grows with job size. Under an independent, identical-rate model the aggregate hazard grows with membership, but real fleets also carry correlated hardware, shared-service, network, software and detection effects. The input you need is the measured effective job-level interruption distribution, not a universal hourly cadence.

Several public anchors are useful, but they are not directly interchangeable. Meta's research-cluster study measured 6.50 failures per thousand node-days on its A100 RSC-1 cluster over 11 months (2.34 on RSC-2). The paper's model takes a job's MTTF as one over the rate multiplied by the job's node count, giving about 1.8 hours at 16,384 GPUs. The authors report that the model fits the MTTF they measured on jobs larger than 32 GPUs. The rate counts only node failures and failures a critical health check flagged, so size against your own measured job-level interval, which also catches software and user-program faults. In a 54-day snapshot of Meta's Llama 3 405B run, up to 16,384 H100s logged 419 unplanned interruptions, about one every three hours. Table 5 attributes 148 events to faulty GPUs and 72 to HBM3 errors. The team still achieved over 90% effective training time in that Llama 3 campaign through aggressive automation, with significant manual intervention needed only three times. SemiAnalysis's October 2024 operator playbook reports about 7 days as the typical MTBF for a 512-H100 cluster at a top-tier operator, and separately recommends at least 3–4 weeks of factory high-temperature burn-in before deployment. The first is not a per-GPU rate, and the second is not an observed universal settling time. Alibaba's Unicron paper (December 2023) reports a 43.4% failure rate among the top 5% most resource-intensive large jobs in its production study, and about 73% of failures are fixable by restarting the job.

Meta's newest public figure is a target, not an observation: about two failures per 8,000 GPUs per day. Trade press reported it from SEMICON Taiwan (September 2026, as hardware failures) and from Meta's OCP Global Summit keynote (October 2026, as training interruptions); Meta has published no text. Scale that target linearly and a synchronous job on 100,000 GPUs still stops about 25 times a day, close to once an hour. At scale, failure is the steady state to engineer for.

A facility-topology classification does not measure training output. Uptime Tier outcomes describe topology under defined conditions, while a 100k-GPU synchronous job loses far more useful time to internal hardware faults that 2N power and cooling do nothing to prevent. The metric that governs a training cluster's return is goodput: useful work delivered within the job's quality and service limits. As a time fraction (ETTR, effective-training-time ratio), it is the share of wall-clock GPU-time that advances the job. Chapter 12.2 is the canonical home of this pivot; here it is the lens for scoring every recovery decision.

Every interruption runs through the same five-stage loop, and the time spent in each stage is what you control. Detect the fault; drain the affected node or rack out of the job; diagnose the root cause; remediate (reboot, reseat, RMA or replace); and restart the job from the last good checkpoint on healthy hardware. Goodput depends on how fast this loop completes and how often it runs. The failure taxonomy the loop must classify (hard faults, transient faults, silent data corruption) is canonical in Chapter 14.3. Here detection is a given input, and the focus is the loop's economics.

The non-obvious lever is that detection latency and restart latency dominate, not repair latency. Repair (an RMA, a reseat) happens asynchronously on a drained node while the job runs on a spare, so it is off the critical path. On the path sit the time to notice the fault and the time to load a checkpoint and re-establish the fabric. A hung collective can stall a job for minutes before a watchdog fires. Alibaba's Unicron paper measured a default recovery path of 68 minutes on a 256-H800 GPT-3 run. Up to 30 of them went to an all-reduce hang that lasted until Megatron's timeout fired. So the reliability spend with the largest return is rarely 'better hardware'. It is faster watchdogs, faster checkpoint loading and a warm spare ready to slot in. Google reports that multi-tier checkpointing cut restore across thousands of nodes from up to 30 minutes with object storage alone to under a minute. Separately, it reports a 6.59% ML-goodput gain on one 35K-chip TPU v5p workload. Both are vendor-reported.

The table is a budget allocator. The three stages on the critical path (detect, drain, restart) are where wall-clock goodput is won or lost. The two off the path, diagnose and remediate, can be slow and asynchronous once the node is safely fenced and a replacement is placed, provided compatible spare capacity keeps the job running meanwhile. That is the structural argument for hot spares: they turn remediate from an on-path stall into a background task. It is also the argument for multi-tier checkpointing, which attacks restart, a potentially expensive on-path stage, by keeping a recent checkpoint in node-local memory or NVMe. That works only if the copy survives the failure domain at hand and holds validated recoverable state. The cadence and tiering math that governs how much a failure erases is canonical in Chapter 9.4.

The loop assumes a fault that stops the job. A fail-slow node stays alive but runs, say, 15% slow from thermal throttling, a degraded link or a failing DIMM. It trips no watchdog and never appears in the interruption log, yet a synchronous job runs every step at its pace. Detect it from per-rank step time and collective latency, and fence it once the slowdown crosses a threshold you have tested against false ejections. Chapter 14.4 counts this straggler loss in its goodput accounting.

At a fleet failing more than once an hour, a human in the loop is a bottleneck: the operator becomes the MTTR. The 2025–2026 answer is an autonomous loop. A fleet control plane detects a drained or unhealthy node, runs automated triage to classify the fault, tries remediation (power-cycle, reflash, re-test) without a ticket, and escalates to a human only when it cannot resolve the fault. NVIDIA's Mission Control (2.3.1, documentation current as of October 2026) packages this as three coupled components: autonomous job recovery, autonomous hardware recovery and the NVIDIA Resiliency Extension (NVRx). They run automated health checks at the tray, rack and system level and execute break-fix workflows that open support tickets only for what cannot auto-resolve. Support covers DGX B200 and B300 and DGX and OEM GB200/GB300 NVL72 systems, and Vera Rubin is not yet listed. Coverage has changed from release to release, so check the matrix for your platform. On Kubernetes, NVIDIA's open-source NVSentinel (December 2025) automates the front of the same loop: health monitoring, cordon, drain, then a GPU reset or node reboot (Chapter 14.4). For operators building their own loop, NVIDIA open-sourced a Cluster Readiness Engine (workload-based pre-job qualification; open-sourced in September 2026, v0.7.0 announced at OCP in October 2026) and, in October 2026, a Rack Management Service (rack-level firmware operations). The engine reports failing nodes and never cordons or modifies them, so quarantine and return-to-service sign-off stay with the operator. Hyperscalers run their own equivalents; Meta's is the automation behind the Llama 3 result above. Neocloud operators differentiate on how mature this loop is.

The decision is not whether to automate detection, since everyone does, but how much authority to give the loop to act. Full autonomy (auto-drain and auto-restart on a confidence threshold) maximizes goodput but can amplify a fault. A mis-classifying triage routine can eject healthy nodes, a restart storm can thrash the scheduler, and a shared firmware bug can be 'remediated' onto every node in turn. The guardrails are the ones that make any control loop safe: rate limits on automated actions, quarantine of repeat offenders (lemon-node ejection), and a circuit breaker that hands control to a human when the action rate or failure rate spikes. An under-automated loop loses goodput to human latency; an over-automated one without guardrails loses it to self-inflicted instability.

Once detection and recovery are fast, the next fork is how the job survives the failure. Three established families trade capacity, framework coupling and recovery speed differently. The choice sets the spare-capacity tax you pay and how deeply reliability is wired into the training stack.

Hot spares plus fast restart is the operationally simplest and most common posture. Hold a pool of idle healthy GPUs, sized to the failures you expect while a node is out for repair and placed in the right fabric domain. On failure, drain the bad node, slot in a spare and restart from the last checkpoint. The cost is the idle capacity and the full restart latency. The virtue is that it is framework-agnostic and easy to reason about. Elastic or redundant training lets the job continue on fewer nodes (shrinking the world size and re-sharding) or run redundant replicas so one failure does not force a full restart. That removes the spare tax and much of the restart stall, but it couples reliability tightly to the training framework: the scheduler, parallelism plan and collective library must all cooperate to reshard live. Algorithmic fault tolerance goes further into the math. Nonuniform or elastic tensor parallelism reduces the goodput amplification a single GPU failure causes, and redundant-computation or erasure-coded schemes let the job tolerate a fault without re-execution. That is the lowest overhead in principle, but the least mature and the most workload-specific.

In 2025–2026 a fourth family reached production: checkpointless recovery from live peers. Clockwork's TorchPass (generally available March 2026, licensed) moves a running job's state onto a spare GPU instead of restarting it. Clockwork's own benchmark used a 64-H200 cluster running Llama-4 Scout with injected faults. Checkpoint-restart had to recompute 869 steps, while TorchPass and the open-source TorchFT lost none. The failure counts were not equal, however: 6 for TorchPass and up to 12 for the other two. A SemiAnalysis tester quoted in Clockwork's release put TorchPass's recovery time on a par with TorchFT's, with higher throughput, on a gpt-oss-120B run. That is a testimonial inside a vendor release, not a SemiAnalysis publication. AWS's HyperPod checkpointless training for SageMaker (December 2025) restores state from peers. On a 256-H100 job training Llama 3 70B, AWS's own test put recovery at 47 s, down from 4 min 52 s. On 2,304 GPUs running an internal model, AWS reports under 2 minutes against 15–30 minutes for conventional recovery. Every option here still needs spare capacity and framework support, and each falls back to a checkpoint when the peer copy is lost. Use them to cut restart latency, not to retire checkpoint discipline. Chapter 14.4 covers the operational side of all four families.

Moving down this table changes operational complexity, reserve capacity and recovery speed, and elastic or redundant training also pays in framework coupling. An operations team can run hot spares on a stock framework. Elastic and redundant training need a reliability-aware training stack, which is why they are most common at labs that own their framework end to end. The right answer depends on three things: how scarce spare capacity is (a power-bound fleet must price every idle reserved GPU), how much a stall costs (a contracted run with a deadline cannot tolerate restart storms), and how much control you have over the training stack. A hybrid can use hot spares first and a tested elastic shrink when they run out. A spare must match the job's GPU memory, software stack (framework, CUDA/NCCL, driver) and fabric domain; free inventory in the wrong domain cannot back the promise. Test simultaneous claims on shared reserve before selling it twice.

The reliability budget has three competing claims: raise MTBF (better hardware, more burn-in), shrink MTTR (faster detection and restart, hot spares) or add facility nines (2N power, redundant cooling). At frontier scale they are not equally productive. To first order, a checkpointable job's goodput is uptime fraction = MTBF / (MTBF + MTTR + lost-work-per-failure). Lost work is modeled from the interval between checkpoints durably committed inside the failure domain you design for, and the RPO boundary is the newest checkpoint recoverable in that domain. This illustrates failure loss, not full goodput accounting. It has no term for the ordinary cost of checkpointing a healthy job, so it tends to 100% in the no-failure limit even where every 100 s of useful training pays 10 s of blocking save, a true 90.9%. Chapter 9.4 has the complete accounting: productive work, ordinary saves, detection and recovery, recomputation. MTBF falls as you scale and is hard to move, since the hardware is what it is. The dominant levers are therefore MTTR (recovery speed) and lost work per failure (checkpoint cadence), and both are software and control-plane investments, not facility ones.

At a given fleet's measured failure distribution, checkpointing and hot spares may beat facility topology on next-dollar goodput, because the facility does not prevent silicon, software or membership failures. That does not make 2N wasteful. It prevents the facility failures it was designed for, and whether that is worth its cost depends on how often they occur, what they interrupt, whether they share common modes with compute failures and what the contract pays for. The Young/Daly checkpoint derivation is in Chapter 9.4; Chapter 12.5 prices those facility states against compute-stack resilience.

Everything above is the training story, where one fault stalls one tightly coupled job. An online-inference fleet inverts almost every assumption, and the reliability strategy inverts with it. At the API, inference requests stay retry-oriented, but the infrastructure has two serving tiers. In independently replicated serving, a node failure drops one replica's in-flight requests. In frontier serving, one replica can span an NVL72 scale-up domain, so a node or fabric failure can remove the whole distributed replica and every request it is serving. There is no training checkpoint to restore: the unit of failure is a request at the API and a replica or domain in the infrastructure. The reliability targets differ accordingly: request success rate and SLO attainment (TTFT/TPOT within budget), not goodput as effective training time. Required maintenance and fault outcomes select the facility topology, not the inference label, and revenue-critical serving often justifies a stronger posture than checkpointable batch work.

The fault-tolerance toolkit shifts with it. For inference you invest in fast health-checking and load-balancer ejection (pull a sick replica out of rotation in seconds). For distributed replicas, add domain-aware health, replica-level draining, and whole-domain spare and placement policy. Add geographic and zonal redundancy so a rack, hall or region failure degrades capacity rather than availability, and graceful degradation (shed or queue low-priority traffic, fall back to a smaller model, preserve the SLO for what remains). These replace checkpoint cadence and hot training spares as the investments that matter. Separate three recovery paths: an engine-process crash, a GPU fault, and a node or domain loss. After a process crash, reloading weights dominates a cold restart. NVIDIA's Dynamo shadow-engine recovery (preview, August 2026) keeps weights resident in a GPU memory service, so a standby engine took over in 7.3 s instead of 283 s in a two-worker B200 test (vendor-measured). It does not cover hardware or node failures, and the promoted engine starts with a cold KV cache, so expect a TTFT bump after cutover. KV-cache recovery cost follows context length and replica design: short prompts are cheap to recompute, while long-context prefill can make recomputation or replication a material capacity and latency decision. Inference recovery therefore couples API retry with replica or domain recovery, and prefix-cache locality (Chapter 9.7) means losing a replica still costs cache warmth and latency. The serving-engine and SLO mechanics this posture protects are in Chapter 10.11, multi-region failover and DR in Chapter 12.3, and the SLA/goodput-contract framing in Chapter 12.4.

The same reliability mistakes recur because each optimizes one number in isolation instead of goodput end to end:

• Choosing topology from checkpointability. Checkpoint-and-resume lowers the consequence of some interruptions. It does not prove that N/N+1 is sufficient or that 2N is waste. Price the facility failure states against checkpointing and spares (Chapter 12.5).
• Optimizing MTBF instead of MTTR. Pouring the reliability budget into hardware screening to lift MTBF a few percent while restart still takes twenty minutes. At a fleet failing every few hours, compare the loss avoided by halving MTTR with the loss avoided by fewer interruptions, counting lost progress and reserve. The larger return depends on the measured clocks and the cost of each intervention, not on a universal preference for software.
• An autonomous loop without lemon tracking or rate limits. Auto-rebooting and re-admitting nodes with no health history and no remediation rate limiter, so a single lemon flaps the job and a correlated firmware fault triggers a fleet-wide remediation storm.
• Treating SDC as someone else's problem. Relying on the hang-and-restart loop to catch corruption it cannot see, and finding days of poisoned training only at the next eval. SDC needs a dedicated detection program (Chapter 14.3), not a louder watchdog.

Buy recovery time from the stage that dominates the measured critical path, and test it with its spare, checkpoint and ownership dependencies in place. Faster storage buys nothing while allocation is blocked, and more spares buy nothing when the checkpoint cannot resume valid work. Price the remedy in Chapter 14.1's goodput accounting, with Chapter 9.4 for checkpoint semantics.