Reliability Engineering for Training (Operational)

A frontier training job is one synchronous computation spread across tens of thousands of accelerators, and it advances at the pace of its slowest healthy participant. Under the default rigid posture, when any participant dies the whole job stops: there is no graceful degradation, no load shedding, no failing a request over to another server. Elastic and peer-recovery schemes, covered later in this chapter, buy back some of that. Meta's Llama 3 405B pre-training, on up to 16,384 H100s, recorded 419 unexpected interruptions (466 in total) in a 54-day snapshot, about one every three hours. That rate, like RSC-2's component failure rates, does not scale to a minute-level forecast for a 131,072-GPU job without explicit assumptions about which hardware the job touches, correlated faults, software and how events are counted. At that scale the question is no longer whether the run will be interrupted but how much wallclock you burn each time, and summed over a multi-week run that decides whether the job holds an effective-training-time ratio near 0.9 or falls well short of it.

This chapter is the operational counterpart to the design-time reliability work in Part 12 and the failure-data catalog in Chapter 14.3. It does not derive checkpoint intervals: the Young/Daly derivation is canonical in Chapter 9.4, useful-output accounting in Chapter 14.1, and service boundaries in Chapter 12.2. It covers the four levers an ops team turns day to day: the checkpoint tiering that sets how fast you recover, the detection and isolation that finds the faulty node and ejects it, the elastic-versus-rigid recovery posture, and the MTTR decomposition that shows which of these deserves the next dollar.

The facility world measures itself in availability nines, and the legacy Tier III/IV mappings are still quoted in every sales deck (Uptime itself has disavowed them). Facility availability bounds continuity, but it does not tell you whether the job's own GPUs made forward progress on the same step. The operational metric is Effective Training Time Ratio (ETTR): productive runtime divided by the job's available wallclock. It uses the same denominator as Chapter 14.1, scheduled or eligible-queue time with restart overhead included; Google application goodput starts at the application boundary and must not silently inherit scheduler queue time. The industry also speaks of goodput, the share of wall-clock time that advances the job, net of badput (restarts, recomputation since the last checkpoint, idle time waiting for a replacement node, slow stragglers). Meta reports that 2+ day RSC-1 runs at 2,048–4,096 GPUs average an ETTR of around 0.9 (0.85–0.9 for the largest, highest-priority jobs) even in a congested shared cluster, on assumed 5-minute checkpoint and restart overheads. Meta's revised model finds that the largest feasible RSC-1 run, about 12,000 GPUs, reaches ETTR near 0.9 only with checkpoint writes of about 10 seconds or a failure rate cut from 6.50 to about 1 per 1,000 node-days (Chapter 12.5).

This matters in 2026 for economic reasons. The cluster is power-bound, and the GPUs depreciate against a contested 2–3-year bear-case economic clock whether or not they do useful work. The SemiAnalysis April 2026 ClusterMAX goodput-expense model prices one large pretraining scenario (5,184 GB300s) at 6.1% (Gold-tier provider) to 20.9% (Silver tier), with a hyperscaler at 10.5%. Each tier assumes its own recovery method, so measure the job's own loss account before pricing it. On the cost basis Chapter 14.1 uses, Epoch AI's ~$8.5B a year of all-in TCO for 1 GW of IT (May 2026), five ETTR points on a 1 GW fleet are worth about $425M a year of cost-equivalent output (0.05 × $8.5B). That is usable compute capacity, not recovered capital or cash. Chapter 14.1 accounts for lost progress; Chapter 1.3/1.8 value recovered capacity against paid demand and contribution.

The optimal checkpoint interval is a solved problem. Young's first-order approximation uses checkpoint cost and MTBF, and Daly's higher-order model adds recovery, restart and downtime terms; the derivation is canonical in Chapter 9.4. What operations controls is an input to that math: the cost of a checkpoint and the cost of a restart. Drive both down and the optimal interval shrinks, the work at risk between checkpoints shrinks, and ETTR rises without buying a single additional GPU. The lever is tiering: keeping checkpoints at several storage levels so the common-case recovery never touches the slowest, most durable tier.

Take the mixed-precision model-and-optimizer layout of 14 bytes per parameter. A 405B-parameter checkpoint is about 5.7 TB before metadata and redundancy, and other state layouts change that payload. Writing it to a persistent parallel file system on every interval is bandwidth-prohibitive and stalls the job, and reloading it across the fabric on every failure is the dominant restart cost. Tiering addresses both. The canonical three-tier scheme keeps the freshest checkpoint in node-local RAM/host memory (potentially a faster reload when the job restarts on the same hardware; measure complete restart latency, not memory access time), a redundant copy on a peer node or adjacent slice (survives a single-node loss without touching durable storage), and a periodic copy on durable object/parallel storage (survives a whole-job or facility event). Google reported a 6.59% goodput uplift on a 35K-chip TPU v5p workload, with checkpoint-save latency under five minutes and restore under one minute across thousands of nodes. AWS separately reported checkpointless peer recovery cutting 15–30 minutes to under two minutes on 2,304 H100 GPUs.

Two refinements sit on top of the tiers. Asynchronous checkpointing snapshots state to host memory in a brief synchronous pause, then drains it to durable storage in the background while training continues. A common design goal is to keep the visible stall to a small fraction of the interval, which is what makes a frequent durable cadence affordable. Sharded or distributed checkpoint formats let each rank write its own shard in parallel, so checkpoint wall-time is set by shard size and per-node bandwidth, up to the storage system's aggregate ingest limit. Without sharding, the checkpoint itself becomes the straggler. The operational fork is how much memory and fabric you will spend on the upper tiers. A job that checkpoints only to durable storage is simple and cheap to run but recovers slowly. A fully tiered job recovers in seconds but reserves host RAM and burns fabric bandwidth on replica writes. Your MTTR decomposition (below) shows where the time is going, and so which answer is right.

A restart is only as good as the decision about where to restart. Resume the job on the same flaky node and it fails again within the hour; the run thrashes, ETTR collapses and on-call burns out. So the second pillar is detection and isolation: catching that a failure occurred, attributing it to the right component, and pulling the bad hardware from the schedulable pool before it poisons the next attempt. The difficult cases are not the hard failures, which are loud and easy (a GPU that falls off the bus), but the silent and gray ones: silent data corruption that produces wrong gradients with no error, and stragglers that are alive but running 20% slow and drag the whole synchronous job down to their pace. Hardware faults still cause most interruptions. Of the 419 unexpected interruptions in Meta's Llama 3 405B log, faulty GPUs caused 148 and GPU HBM3 memory another 72, together over half, while network switches and cables accounted for 35. So GPU-level checks catch the bulk of the interruptions. They miss the stragglers and silent errors, because those never raise one. The failure taxonomy (hard / transient / silent) is canonical in Chapter 14.3; this chapter covers the operational detection program that sits on top of it.

Detection runs at three boundaries. Synchronous health checks at job launch (and after every restart) sweep the allocated nodes for known-bad signatures before training starts. They are cheap, but they only catch what they test for and add latency to every recovery. Passive in-run monitoring watches per-step timing, collective-op latency, ECC counters and thermal/throttle telemetry to flag stragglers and degrading nodes mid-run. It catches the gray failures the launch check misses but risks false positives that eject healthy nodes. Do not treat GPU utilization as a liveness signal. One 2026 field report (Kim et al., arXiv) describes a 16-GPU B300 fine-tuning job that sat in an NCCL hang for about three hours, with utilization pinned at 100%, until the watchdog fired. Some ranks had run out of data before others. Alert on step-progress counters and per-GPU power draw instead, and add cheap pre-run invariant checks, such as equal batch counts per rank. A periodic offline node sweep tests idle nodes against a benchmark to find slow or SDC-prone hardware before it is ever scheduled. It is the most thorough method but consumes capacity that could be training. Mature operators run all three; the question is the weighting.

Isolation feeds automated remediation. Once a node is flagged, the workflow drains the job off it, attempts an automated recovery ladder (reset the GPU, reload the driver, power-cycle the node, reflash firmware), re-tests the node against the health suite, and either returns it to the pool or opens an RMA and pulls a spare. How far to automate is the facility-ops/ML-platform-ops boundary made concrete. The platform owns the detect-flag-eject loop and job-level recovery; facility ops owns the physical swap and the RMA logistics. Where that boundary sits, and how far the autonomous ladder runs before a human is paged, is an organizational decision covered in Chapter 14.11. The autonomous-recovery mechanics overlap with Chapter 10.7.

You do not have to build the first rungs from scratch. NVIDIA's open-source NVSentinel (December 2025) watches DCGM health, kernel logs and cloud-provider maintenance events on Kubernetes, then cordons and drains the node and triggers a GPU reset or reboot. Its default install only monitors until you enable those actions; NVIDIA reports running it across more than 40,000 GPUs (March 2026). Slurm sites hang the same checks on the HealthCheckProgram hook, Kubernetes sites on node-problem-detector, and NVIDIA's Xid catalog gives a recommended resolution for each error code (Chapter 10.6). Each still needs your thresholds and your RMA hand-off.

The default recovery posture is rigid: the job fails and stops entirely, the scheduler allocates a full healthy set of nodes, and the run reloads the last checkpoint and replays. It is simple, robust and the right choice for most jobs, but it pays the full detect-isolate-reschedule-reload cost on every failure. At six-figure GPU counts the reschedule term alone (waiting for the scheduler to find and provision a clean replacement set) can dominate. The two alternatives buy faster recovery by spending capacity.

Elastic training lets the job continue on the surviving nodes at reduced width: drop the failed node's data-parallel replica, reshard and keep stepping, then re-expand when a replacement is qualified. It removes the stop-and-reschedule stall for the common single-node case, at the cost of running slower (and at a slightly different effective batch size) until the node is back. Redundant / spare-pool training keeps hot spares qualified and idle, so a failed node is swapped in seconds instead of waiting on the scheduler or an RMA. Recovery approaches the reload time alone, but the spares are a standing goodput tax: idle GPUs that depreciate without computing. The fork trades recovery speed directly against reserved capacity. It also depends on the parallelism scheme: nonuniform or elastic tensor parallelism reduces the failure-amplification penalty of a node loss, which is what makes degrade-and-continue viable at all.

Everything above serves one number, mean time to recovery, and you cannot improve it without decomposing it. Record four phases on one incident timeline and sum only non-overlapping critical-path durations. Each has its own owner, its own cost to shave and its own scaling behavior:

• Detection latency: wallclock from the failure to the system knowing it failed. It is dominated by how fast health monitoring fires. Gray and silent failures (a slow straggler, an SDC) can take many steps to surface and are the worst offenders. Shave it with passive in-run monitoring and tighter step-time anomaly thresholds.
• Isolation & scheduling: attributing the fault to a component, ejecting it and provisioning a healthy replacement set. This term grows with cluster size (more nodes to schedule around), and hot spares and elastic continuation attack it directly.
• Reconfiguration: rebuilding the parallel topology, re-establishing collectives and re-warming the fabric on the new node set. It is roughly fixed per restart; elastic or nonuniform parallelism reduces it by avoiding a full topology rebuild.
• Reload & replay: loading the last checkpoint and recomputing the work since. Tiering attacks this term (load from RAM, not durable storage), and the Young/Daly interval bounds it (less work at risk between checkpoints).

Instrument all four and attack the binding one, because which term dominates shifts with scale. On a few-thousand-GPU job, reload and replay usually dominates, so tiering and interval tuning pay off most. On a six-figure-GPU job, isolation and scheduling grows until it rivals reload, which is why hot spares and lemon-node pre-ejection (keeping a clean pool so scheduling is instant) are where the next dollar goes. The classic waste is optimizing the wrong term: buying faster checkpoint storage when your time is going to a slow scheduler.

The recurring failures in training-reliability operations come from optimizing one term in isolation, or from treating a frequent-failure regime as a rare-failure one. Three are worth naming:

• Restarting on the same flaky node. With no lemon-node ejection, a degrading GPU re-kills the job every recovery cycle. The interruption count looks like bad luck; it is one bad node poisoning every attempt. The fix is cheap (eject before rescheduling) and the payoff is large (14%→4% large-job failure).
• Tiering only to durable storage. In the AWS measurement above, a conventional restart took 15–30 minutes to resume from the last saved checkpoint; peer recovery took under 2. With an interruption every three hours, the difference costs roughly 7–16% of wallclock. At a few thousand GPUs, where interruptions are rarer, this is survivable. At six-figure GPU counts, find out whether reload, scheduling or reconfiguration binds before concluding that 0.9 ETTR is out of reach. A RAM or peer tier pays for itself when its measured restore time and fault coverage cut the end-to-end loss.
• Choosing facility topology from checkpointability. Checkpointing can reduce the work lost to an interruption but does not prove that a path or hall outage meets the recovery SLO. Compare facility states and common modes against faster recovery, hot spares and lemon ejection in the Chapter 12.5 model.