Checkpointing for Large-Scale Training

At small scale, checkpointing is an afterthought: save the model every so often, restore if something breaks. At frontier scale it becomes one of the load-bearing decisions in the training system, because the arithmetic flips. Over a 54-day window of Meta's Llama 3 405B pre-training on up to 16,384 H100s, the job logged an unplanned interruption roughly once every three hours: 419 in all. Meta's Table 5 attributes 148 of them to faulty GPUs and 72 to HBM3 errors (Chapter 10.7). A fault that escapes local retry, spare substitution or the runtime’s fault-tolerance mechanism can stall a synchronous run completely: the whole machine rewinds to the last good checkpoint and replays the work in between. Count those job interruptions, not every component alarm. The question is no longer whether to checkpoint but how often, to where and at what bandwidth, and getting those three wrong directly reduces measured goodput.

This chapter is the canonical home for the checkpoint math the rest of the guide cross-references. It works decision by decision from what a checkpoint contains, through the failure model and Young's and Daly's interval formulas, to tiering, on-disk format, compression, and sizing the bandwidth and the incast it creates.

The first sizing number is how big a checkpoint is, and the starting point is a serialization recipe: ~14 bytes of checkpoint state per model parameter. VAST Data assumed this recipe to infer model sizes from a survey of 85k+ checkpoints, which carries about ±15% uncertainty, so verify it against what your framework actually writes. Those bytes are not the model weights, which are the smallest part. For the standard BF16-compute / FP32-master / Adam-optimizer recipe, the breakdown is:

• 2 bytes: the BF16 (or FP16) model weights used in the forward/backward pass.
• 4 bytes: the FP32 master copy of the weights the optimizer actually updates (mixed precision keeps a high-precision master to avoid drift).
• 4 bytes: Adam's first moment (the running mean of gradients), one FP32 value per parameter.
• 4 bytes: Adam's second moment (the exponential moving average of squared gradients), again FP32 per parameter.

That sums to 14. Stacks differ, though. Megatron and DeepSpeed write the BF16 weights next to the FP32 master, hence 14 bytes. PyTorch FSDP2 mixed precision with FP32 parameters stores only the sharded FP32 copy and casts down to BF16 at compute time, so its checkpoint comes to about 12 bytes per parameter. Gradients are usually not checkpointed: saves happen at optimizer-step boundaries, after the gradients have been applied, so a restart simply begins the next step.

The implication that catches teams out: under this recipe, a checkpoint is 7× the BF16 weight file you think you are saving. A 405B-parameter model is a ~0.8 TB weight file but a ~5.7 TB checkpoint, and a 1T-parameter model is a ~14 TB checkpoint. Optimizer state, not weights, dominates the I/O. So "just save the weights" is the wrong mental model for resumable training, and optimizer-state sharding (ZeRO/FSDP) is as much a checkpoint-I/O strategy as a memory strategy.

The decision this forces sits upstream of storage: change the optimizer or the precision recipe and you change the checkpoint size, and with it the bandwidth budget. FP32 Adam is 4 + 4 bytes for the moments, plus a 4-byte master weight and a 2-byte BF16/FP16 weight: 14 bytes per parameter. 8-bit Adam is 1 + 1 bytes for the moments, plus the same 4-byte master and 2-byte weight: 8 bytes per parameter. A memory-efficient optimizer that drops the second moment changes the arithmetic again. These are training-science choices with an often-unnoticed consequence for the storage and fabric you must provision, and the parameter-to-bytes mapping is the bridge between the two.

To size a checkpoint interval you need a failure model, and the one that matters for synchronous training is simple: a fault that escapes local recovery can force whole-job rollback. One GPU falling off the bus, one optical transceiver degrading, one HBM uncorrectable error: when redundant paths and local recovery cannot absorb the fault, a synchronous job spanning tens of thousands of GPUs can roll back to its last checkpoint and replay. That property defines synchronous training and is why the interval matters, since the interval controls the lost-work part of a job interruption.

A single failure costs lost GPU-hours in three parts. The wasted compute since the last checkpoint is on average half the interval. The detection-and-restart latency is the time to notice the fault, evict the bad node and reload state. The checkpoint overhead is paid during normal operation, so account for it separately from the incident duration. A long interval saves overhead but risks losing a lot of work per failure, and a short one loses little work but pays overhead constantly. The optimum is where the two balance, and it moves with the interruption rate you measure for the actual job.

The interval is not a matter of taste. Young's 1974 first-order approximation gives the square-root interval below, and Daly's 2006 higher-order model uses a fuller checkpoint/restart cost model. This is the canonical result the guide refers back to from Chapters 1.2, 10.7 and 14.4, and it follows directly from the failure model above. The intuition: wasted work per unit time grows with the interval (more to replay on failure), while checkpoint overhead per unit time shrinks with it (fewer saves). Minimize the sum and you get a square-root law:

τ_opt ≈ √( 2 · C · MTBF )

Here τ_opt is useful compute time between synchronous saves, C is blocking save time through durable completion, and M = MTBF is whole-job mean time to interruption. The failure-free wall-clock cycle is τ + C, so the save fraction is C / (τ + C). The optimal interval scales with the square root of both the save cost and the MTBF. Halve your checkpoint write time (faster tier, more drain bandwidth) and the optimal interval shrinks only by √2. The optimal overhead fraction, √(2C/M), falls by the same √2, so halving C buys a √2 improvement, not a twofold one. If individual accelerator and node hazards were independent and identically distributed and membership were the only scaling variable, doubling cluster size would approximately halve whole-job MTBF and shrink the square-root interval by √2. Real jobs also fail through correlated hardware, shared services, software, membership changes and detection policy, so size the interval from your fleet's measured whole-job MTBF, not by scaling a component or smaller-cluster observation.

Either model is only as good as the measured whole-job failure and recovery inputs you feed it, and most teams feed it a fantasy. A cluster whose burn-in telemetry shows elevated interruptions needs a different cadence, and a run that hits a lemon node (Chapter 10.7) sees its effective MTBF crater. Measure MTBF continuously from the fleet's own fault telemetry and re-derive τ as conditions change: a shorter interval early in a run, a longer one once burn-in clears. A static interval set at scoping time is wrong on day one and wrong again at steady state.

For the synchronous Poisson-interruption model, Daly (2006), equation 37 gives τ ≈ √(2CM)[1 + √(C/(2M))/3 + C/(18M)] − C when C < 2M, and τ ≈ M otherwise. Failures during restart are included: restart duration changes total completion time but cancels from this optimum. Asynchronous capture needs a separate protected-state timeline, because replacing C with a short local snapshot does not optimize durable recovery.

The biggest lever on checkpoint cost C is where the checkpoint lands and how much of the write blocks training. There is a clear spectrum, and most large runs use several tiers at once.

Synchronous checkpointing stops training, writes the full state to durable storage and resumes. It is simple and correct, but the most expensive: the GPUs sit idle for the entire write, so C is the full write time and grows with model size. At frontier scale, a synchronous write to object storage can stall the machine for tens of minutes. At that scale whole-job MTBF is a few hours: Meta's Llama 3 405B run was interrupted roughly every 3 hours. Take C = 30 min and M = 3 h. Young's interval then comes to about 104 minutes, and the saves alone consume over a fifth of wall-clock time before any replay. Daly's higher-order model, which matters when C is a sizeable share of M, gives about 85 minutes and a save share near 26%. Synchronous-only checkpointing remains viable even on a large cluster when the complete save fits its pause and recovery budgets.

Asynchronous checkpointing has four distinct timing terms. The blocking snapshot copies GPU state to host RAM or local NVMe and pauses training only for that capture. Training then resumes while the background drain moves the snapshot to durable storage; define an explicit drain-completion window D from snapshot release to the deadline for drain completion. The separate durable commit tail covers flush, metadata publication, verification and atomic visibility before the checkpoint is recoverable. Finally, add contention and concurrency headroom for simultaneous checkpoints and shared fabric or storage traffic.

Size the ideal drain floor as unique serialized bytes divided by D. Because D starts at snapshot release and ends at drain completion, capture and commit are already outside it. Subtract them only when converting a capture-start-to-commit budget into D, and add contention and concurrency headroom separately. In VAST's analysis of 40 production runs, asynchronous drains ran at 50–200 GB/s, and nearly every run kept its median checkpoint overlap (the share of training time with a background drain in progress) under 10%. VAST uses that 10% to size global checkpoint bandwidth. The figure describes those runs; your drain window still comes from your own state-age, pause and commit limits. Asynchronous checkpointing lets training continue while the captured state drains. Choose it when protected commit and interference fit the budgets better than a tested synchronous save.

In-memory / peer checkpointing goes further: it keeps the most recent checkpoint in the CPU DRAM of peer nodes rather than (or in addition to) durable storage. Aggregate host-memory bandwidth across the cluster dwarfs any storage tier, so the checkpoint can be taken every single step with near-zero throughput overhead (the GEMINI/in-memory line of work demonstrated optimal per-iteration checkpointing and >13x faster recovery). After a failure, a surviving peer holds a copy of the dead node's shard, so recovery is a memory-to-memory restore in seconds rather than a storage read. The cost is RAM: you reserve host memory across the fleet for redundant copies, and you need a placement strategy that guarantees any single node's failure leaves its shard recoverable from a survivor. This is the frontier-operator pattern: capture intervals approach the step duration, and where the surviving peers sit decides how much progress stays recoverable.

Independent of where the checkpoint lands is how it is laid out, and this is a genuine fork with a sharp scaling consequence. A monolithic checkpoint gathers the full model and optimizer state onto one rank (historically rank 0), which serializes one large file. It is easy to reason about and easy to load anywhere, but the all-gather does not scale: as the cluster grows, more ranks funnel state through one writer, and C grows with model size while write bandwidth stays pinned to a single node's link. Monolithic checkpoints are a small-model convenience that breaks down once gathering state into one writer exhausts its memory or misses the checkpoint write budget.

A sharded / distributed checkpoint writes each rank's own shard in parallel: every GPU drains its slice of weights and optimizer state to storage simultaneously. Aggregate write bandwidth then scales with the cluster, which is the only way to keep C small at frontier scale. This is the format behind PyTorch Distributed Checkpoint (DCP), DeepSpeed/Megatron sharded checkpoints, and the Orbax format on the JAX/TPU side. The cost is complexity. A sharded checkpoint is a directory of many files plus metadata describing the parallelism layout that produced it, and reading it back is a distributed operation, not a single load.

The consequence that bites teams hardest is resharding. A sharded checkpoint written under one parallelism layout, say TP=8, PP=16, DP=64, is not trivially loadable under a different one. If a failure forces a restart on fewer nodes, or you want to resume with a different tensor/pipeline split, a naive sharded format needs a slow offline reshard or cannot load at all. Modern distributed-checkpoint libraries solve this with a layout-independent logical format: the checkpoint records the global tensor shapes and each shard's coordinates, so the loader can re-partition on the fly to the layouts that library, state representation and model implementation support. Verify your checkpoint format survives a reshard before you need it to. Discovering at 3 a.m. that your only checkpoint cannot load on the 12,000 GPUs you have left (instead of the 16,000 you started with) is a goodput catastrophe that elastic, reshard-capable formats exist to prevent.

Because the checkpoint is dominated by optimizer state, an obvious lever is to make it smaller by compressing it or saving only part of it. For most runs the better move is to provision more drain bandwidth before you reach for compression. Lossless compression of BF16/FP32 tensors yields modest ratios at real CPU cost, and that CPU competes with the data loader (Chapter 9.5) for the same host cores. Halving the checkpoint size halves C, and the square-root structure of Young's first-order approximation then moves the optimal interval by only √2, a smaller win than the bandwidth and CPU spend often justify.

In specific cases, reducing checkpoint volume does pay. Optimizer-state quantization (8-bit Adam moments) cuts the two moment tensors from 4 + 4 to 1 + 1 bytes per parameter, taking the full recipe from 14 bytes to 8 while keeping the 4-byte FP32 master and 2-byte BF16/FP16 weight. It is a training-science decision with a storage payoff. Sparse / incremental checkpointing snapshots only the subset of state that changed (or a rotating subset of operators) each interval rather than the full state every time, which is especially attractive for Mixture-of-Experts models.

Two different mechanisms hide under that heading, and they carry different recovery contracts. Incremental checkpointing writes state that changed since its valid base, so reconstruction needs the base and every required delta. Keep that dependency chain protected, and verify reconstructed state before claiming unchanged restart fidelity. Selective expert checkpointing, the MoEtion line of work, exploits expert popularity skew to snapshot hot experts often and cold ones rarely. It buys high frequency by tolerating a bounded amount of dropped work on the cold path. Over a full batch most experts do see updates, so this second mechanism is not free: decide explicitly what may be lost on restart and which convergence test justifies losing it. The split is clean. For dense models, buy bandwidth; for MoE and 8-bit-optimizer regimes, the structure of the state itself makes reducing checkpoint volume the better lever.

Writing checkpoints is half the system; recovery is the half that determines how long a failure actually costs. A complete recovery runs in sequence. The system detects the fault, then isolates and evicts the bad node. Ideally a hot spare takes its slot, so the parallelism layout does not change. Next, it rebuilds the communicators, and on large clusters process-group initialization by itself can run to tens of minutes, per AWS. Then state reloads to every rank, and the job resumes. Detection latency is its own discipline (Chapter 10.7): silent data corruption and slowly degrading hardware can corrupt steps for a long time before a checkpoint is even triggered. The reload is where checkpoint architecture pays off. A peer/in-memory restore takes seconds and a local-NVMe re-read seconds to a minute. A cold read of a multi-terabyte sharded checkpoint from the durable tier takes minutes: the same up-to-30-minutes-to-under-a-minute spread the tiering decision controls.

In-place (checkpointless) recovery avoids the job restart. A replacement node joins a rebuilt process group and pulls its state from surviving peers (Chapter 14.4, Chapter 10.7). When the loss is correlated or spans several nodes, a durable checkpoint is still what you fall back on.

Define the background window from snapshot release to drain completion, as in the worked policy. To convert a whole capture-to-commit deadline, remove capture and commit once, then divide unique payload by the remaining drain time. Concurrent jobs share the delivered write budget, so sum the overlapping streams and include measured contention. Keep read bandwidth for recovery separate: ranks reload concurrently, but the traffic can be storage fan-out, client incast or both, depending on shard placement. Size the actual bottleneck cuts and the reload deadline. A write-optimized tier whose readers miss that deadline still fails recovery, even if its last checkpoint was durable.

The burst is the part that breaks naively designed clusters. A synchronized checkpoint write creates incast at shared storage destinations. Recovery reads create fan-out and receiving-side concentration that, if it shares the back-end fabric with the training collectives, will collide with the all-reduce traffic and tank the goodput it exists to protect. The co-design rule is to isolate checkpoint I/O from the compute fabric, with a dedicated storage rail or strict QoS separation on a converged fabric, so a checkpoint storm never starves an all-gather. Chapter 8.5 treats this fabric-and-storage co-design problem and Chapter 9.8 revisits it for storage sizing. The point here is that the checkpoint interval you choose creates a periodic incast whose magnitude you computed above, and the fabric must absorb it without disturbing training.

A defensible checkpoint policy is a coupled set of decisions, not a single setting, and together they define how much of a failure is recoverable. Size the unique state: about 14 bytes/parameter for BF16 weights plus FP32 master weights and Adam moments, then add the application state actually saved. Measure MTBF from live fault telemetry, and recalculate the interval with Young's first-order or Daly's higher-order model as burn-in clears. Make C small where asynchronous capture and drain reduce measured training pause, and use a sharded, reshard-capable format. Add an in-memory/peer tier when its surviving placement and timed reload let the common single-node failure recover inside the budget; the protected-state timeline still governs the durable copy. Choose the durable-copy cadence from the required failure-domain state age, with a commit protocol so you never load a torn checkpoint. Then size both the drain bandwidth and the recovery incast, and isolate them from the compute fabric. These mechanisms can materially improve measured goodput: Google reported a 6.59% increase on one 35K-chip TPU v5p workload.

Choose the policy whose captured progress, protected commit, pause fraction and recovery timeline all pass the failure you are designing for. A short local snapshot buys useful compute only while drain and publication keep the recoverable generation current; otherwise it buys an expanding recovery gap. Chapter 10.7 orchestrates the policy, Chapter 12.3 extends it to regional recovery, Chapter 13.9 tests it at acceptance and Chapter 14.4 runs it during incidents.