Checkpointing for Large-Scale Training

At small scale, checkpointing is an afterthought: save the model every so often, restore if something breaks. At frontier scale it becomes one of the load-bearing decisions in the whole training system, because the arithmetic flips. A 16,384-GPU run failed roughly once every three hours in Meta's Llama 3 405B snapshot — 419 unplanned interruptions over 54 days, 78% of them hardware-caused (Chapter 10.7). Larger synchronous jobs expose more components and shared dependencies, but their interruption cadence must be measured for the named fleet, job membership, software stack, and event definition rather than inferred from accelerator count alone. Every one of those faults is, for a synchronous job, a full-job stall: the whole machine rewinds to the last good checkpoint and replays the work in between. The question is no longer whether to checkpoint but how often, to where, and at what bandwidth — and getting those three wrong is the difference between 96% goodput and 80%.

This chapter is the canonical home for the checkpoint math the rest of the guide cross-references, built decision by decision — the anatomy of a checkpoint (why ~14 bytes/parameter, and what is in those bytes), the failure model that makes checkpointing a goodput problem rather than a durability one, Young's first-order approximation and Daly's higher-order checkpoint model that turns MTBF and save-cost into a frequency, the tiering spectrum from synchronous-to-object up through in-memory peer copies, the format fork between monolithic and sharded checkpoints, compression and sparsification, and finally how to size the bandwidth and isolate the incast it creates.

The first sizing number you need is how big a checkpoint is, and the answer is governed by a rule of thumb that holds remarkably well across mixed-precision LLM training: ~14 bytes of checkpoint state per model parameter (VAST Data, from a survey of 85k+ checkpoints). It is not the model weights — those are the smallest part. The breakdown, for the standard BF16-compute / FP32-master / Adam-optimizer recipe, is:

• 2 bytes — the BF16 (or FP16) model weights used in the forward/backward pass.
• 4 bytes — the FP32 master copy of the weights the optimizer actually updates (mixed precision keeps a high-precision master to avoid drift).
• 4 bytes — Adam's first moment (the running mean of gradients), one FP32 value per parameter.
• 4 bytes — Adam's second moment (the running variance), again FP32 per parameter.

That sums to 14. Gradients are usually not checkpointed — they are recomputed on restart from the restored weights and a forward/backward pass, so they cost replay time, not storage. The implication that catches teams out: a checkpoint is roughly 7x the size of the model you think you are saving. A 405B-parameter model is a ~0.8 TB weight file but a ~5.7 TB checkpoint; a 1T-parameter model is a ~14 TB checkpoint. Optimizer state, not weights, dominates the I/O — which is why "just save the weights" is the wrong mental model for resumable training, and why optimizer-state sharding (ZeRO/FSDP) is as much a checkpoint-I/O strategy as a memory strategy.

The decision this forces is upstream of storage: if you change the optimizer or the precision recipe, you change the checkpoint size and therefore the bandwidth budget. FP32 Adam is 4 + 4 bytes for the moments, plus a 4-byte master weight and a 2-byte BF16/FP16 weight: 14 bytes per parameter. 8-bit Adam is 1 + 1 bytes for the moments, plus the same 4-byte master and 2-byte weight: 8 bytes per parameter; a memory-efficient optimizer that drops the second moment changes the arithmetic again. These are training-science choices with a direct, often-unnoticed consequence on the storage and fabric you must provision. The parameter-to-bytes mapping is the bridge between the two.

To size a checkpoint interval you need a failure model, and the one that matters for synchronous training is starkly simple: any single component failure kills the whole job. One GPU falling off the bus, one optical transceiver degrading, one HBM uncorrectable error — and a job spanning tens of thousands of GPUs must roll back to its last checkpoint and replay. This is a defining property of synchronous training and the reason checkpoint interval matters: it controls the lost-work component of a job interruption. It does not select the facility topology; N, N+1, distributed, or 2N still follows from the permitted maintenance and fault states, interruption and recovery limits, post-event capacity, independence, common modes, and contract.

The cost of a single failure, in lost GPU-hours, has three parts: the wasted compute since the last checkpoint (on average half the interval), the detection-and-restart latency (the time to notice the fault, evict the bad node, and reload state), and the checkpoint overhead you paid continuously to have a checkpoint at all. Push the interval long and you save overhead but risk losing a lot of work per failure; push it short and you lose little work but pay overhead constantly. The optimum is where these balance, and it changes with the effective interruption distribution measured for the actual job. Published anchors from different fleets are not one arithmetic curve: correlated faults, shared services, software, detection policy, and job membership all change job-level MTBF. Re-estimate that input from telemetry as the fleet and workload change.

The interval is not a matter of taste. Young's 1974 first-order approximation gives the square-root interval below; Daly's 2006 higher-order model uses a fuller checkpoint/restart cost model. This is the canonical result this guide refers back to from Chapters 1.2, 10.7, and 14.4 — and it falls straight out of the failure model above. The intuition: the wasted-work cost per unit time grows with the interval (longer interval, more to replay on failure), while the checkpoint-overhead cost per unit time shrinks with the interval (fewer, more widely-spaced saves). Minimize the sum and you get a square-root law:

τ_opt ≈ √( 2 · C · MTBF )

where τ_opt is the optimal interval between checkpoints, C is the cost (in wall-clock time) of writing one checkpoint, and MTBF is the mean time between failures for the whole job. The optimal interval scales with the square root of both the save cost and the MTBF. Halve your checkpoint write time (faster tier, more drain bandwidth) and the optimal interval shrinks only by √2 — but because you also lose less work per failure, total overhead drops faster than the interval. If individual accelerator and node hazards were independent and identically distributed and membership were the only scaling variable, doubling cluster size would approximately halve whole-job MTBF and shrink the square-root interval by √2. Real jobs also fail through correlated hardware, shared services, software, membership changes, and detection policy, so size the interval from the named fleet's measured whole-job MTBF rather than scaling a component or smaller-cluster observation.

Either interval model is only as good as the measured whole-job failure and recovery inputs you feed it, and most teams feed it a fantasy. A burning-in cluster fails 3–4x more often in its first weeks than at steady state; a run that hits a lemon node (Chapter 10.7) sees its effective MTBF crater. The right practice is to measure MTBF continuously from the fleet's own fault telemetry and re-derive τ as conditions change — early in a run, interval shorter; once burn-in clears, interval can lengthen. A static interval set at scoping time is wrong on day one and wrong again at steady state.

The biggest lever on checkpoint cost C is where the checkpoint lands and how much of the write blocks training. There is a clear spectrum, and most large runs use several tiers at once.

Synchronous checkpointing stops training, writes the full state to durable storage, and resumes. It is simple and it is correct, but it is the most expensive: the GPUs sit idle for the entire write, so C is the full write time and it grows with model size. At frontier scale a synchronous-to-object checkpoint can stall the machine for tens of minutes — unacceptable when MTBF is also tens of minutes. Synchronous-only is now a small-cluster pattern.

Asynchronous checkpointing has four distinct timing terms. The blocking snapshot copies GPU state to host RAM or local NVMe and pauses training only for that capture. Training then resumes while the background drain moves the snapshot to durable storage; define an explicit drain-completion window D from snapshot release to the deadline for background drain completion. The separate durable commit tail covers flush, metadata publication, verification, and atomic visibility before the checkpoint is recoverable. Finally, add contention and concurrency headroom for simultaneous checkpoints and shared fabric or storage traffic. Size the ideal drain floor as checkpoint size divided by D, then shorten the available window for blocking and commit time and add measured headroom. VAST reported 50–200 GB/s drain rates across 40 production runs and a 10% overlap observation; those are reported workload results, not a universal fraction of the checkpoint interval. Asynchronous checkpointing is the default for any serious training run today.

In-memory / peer checkpointing goes further: it keeps the most recent checkpoint in the CPU DRAM of peer nodes rather than (or in addition to) durable storage. Because aggregate host-memory bandwidth across the cluster dwarfs any storage tier, the checkpoint can be taken every single step with near-zero throughput overhead (the GEMINI/in-memory line of work demonstrated optimal per-iteration checkpointing and >13x faster recovery). On a failure, a surviving peer holds a copy of the dead node's shard, so recovery is a memory-to-memory restore in seconds rather than a storage read. The cost is RAM: you reserve host memory across the fleet to hold redundant copies, and you need a placement strategy that guarantees the failure of any node leaves its shard recoverable from a survivor. This is the frontier-operator pattern — checkpoint frequency approaching the failure rate, so that almost no work is ever lost.

Independent of where the checkpoint lands is the question of how it is laid out, and this is a genuine fork with a sharp scaling consequence. A monolithic checkpoint gathers the full model and optimizer state onto one rank (historically rank 0), which serializes a single large file. It is easy to reason about and easy to load anywhere, but the all-gather is a bottleneck that does not scale: as the cluster grows, more ranks funnel state through one writer, and C grows with model size while the write bandwidth stays pinned to a single node's link. Monolithic checkpoints are a small-model convenience that breaks down past a few billion parameters.

A sharded / distributed checkpoint writes each rank's own shard in parallel — every GPU drains its slice of weights and optimizer state to storage simultaneously. Aggregate write bandwidth now scales with the cluster, which is the only way to keep C small at frontier scale. This is the format behind PyTorch Distributed Checkpoint (DCP), DeepSpeed/Megatron sharded checkpoints, and the Orbax format on the JAX/TPU side. The cost is complexity: a sharded checkpoint is a directory of many files plus metadata describing the parallelism layout that produced it, and reading it back is a distributed operation, not a single load.

The consequence that bites teams hardest is resharding. A sharded checkpoint written under one parallelism layout — say TP=8, PP=16, DP=64 — is not trivially loadable under a different layout. If a failure forces you to restart on fewer nodes, or you want to resume a run with a different tensor/pipeline split, a naive sharded format requires a slow offline reshard or cannot load at all. Modern distributed-checkpoint libraries solve this with a layout-independent logical format: the checkpoint records the global tensor shapes and each shard's coordinates, so the loader can re-partition on the fly to whatever parallelism the restart cluster has. Verify your checkpoint format survives a reshard before you need it to — discovering at 3 a.m. that your only checkpoint cannot load on the 12,000 GPUs you have left (instead of the 16,000 you started with) is a goodput catastrophe that elastic, reshard-capable formats exist specifically to prevent.

Because the checkpoint is dominated by optimizer state, an obvious lever is to make it smaller — compress it or save only part of it. For most runs the better move is to provision more drain bandwidth before you reach for compression, because lossless compression of BF16/FP32 tensors yields modest ratios at real CPU cost, and that CPU competes with the data loader (Chapter 9.5) for the same host cores. The square-root structure of Young's first-order approximation also means halving checkpoint size only shrinks C by a factor of two, which moves the optimal interval by √2 — a smaller win than the bandwidth and CPU spend often justify.

There are specific cases where reducing checkpoint volume does pay. Optimizer-state quantization (8-bit Adam moments) cuts the two moment tensors from 4 + 4 to 1 + 1 bytes per parameter, reducing the full recipe from 14 bytes to 8 while retaining the 4-byte FP32 master and 2-byte BF16/FP16 weight — a training-science decision with a storage payoff. Sparse / incremental checkpointing snapshots only the subset of state that changed (or a rotating subset of operators) each interval rather than the full state every time, which is especially effective for Mixture-of-Experts models where only a fraction of experts are touched per step — most of the optimizer state is unchanged between intervals, so re-writing all of it is pure waste. Lines of work like MoEtion exploit exactly this to get high-frequency, low-overhead checkpoints for MoE training. The split is clean: for dense models, buy bandwidth; for MoE and 8-bit-optimizer regimes, the structure of the state itself makes reducing checkpoint volume the better lever.

Writing checkpoints is half the system; recovery is the half that determines how long a failure actually costs. A complete recovery is: detect the fault, isolate and evict the bad node (ideally swapping in a hot spare so the parallelism layout is preserved), reload state to every rank, recompute the gradients the checkpoint did not store, and resume. Detection latency is its own discipline (Chapter 10.7) — silent data corruption and slow-degrading hardware can corrupt steps for a long time before a checkpoint is even triggered. The reload itself is where checkpoint architecture pays off: a peer/in-memory restore is seconds, a local-NVMe re-read is seconds-to-a-minute, and a cold read of a multi-terabyte sharded checkpoint from the durable tier is minutes — the same 30-minutes-to-under-a-minute spread the tiering decision controls.

Define the background drain window D explicitly, then size the ideal floor as B_drain = checkpoint size / D. If a 5.7 TB checkpoint has the full 40 minutes available to drain, the decimal-unit floor is 5,700 GB / 2,400 s ≈ 2.4 GB/s per concurrent checkpoint. In practice the usable drain window is shorter after the blocking snapshot and durable commit tail, so the required sustained rate is slightly higher; add measured headroom for storage and fabric contention and simultaneous checkpoints. Keep the read bandwidth for recovery as a separate requirement because recovery is an incast: every rank reads its shard at once, and restoring a multi-terabyte checkpoint quickly can demand hundreds of GB/s to multiple TB/s of aggregate read bandwidth from the storage tier (Chapter 9.2, Chapter 9.3), concentrated into a burst.

And that burst is the part that breaks naively-designed clusters. A synchronized, all-ranks-at-once checkpoint write or recovery read is a textbook incast that, if it shares the back-end fabric with the training collectives, will collide with the all-reduce traffic and tank the very goodput it exists to protect. The co-design rule is to isolate checkpoint I/O from the compute fabric — a dedicated storage rail, or strict QoS separation on a converged fabric — so a checkpoint storm never starves an all-gather. This is the fabric-and-storage co-design problem treated in Chapter 8.5 and revisited for storage sizing in Chapter 9.8; the point here is that the checkpoint interval you choose creates a periodic incast whose magnitude you computed above, and the fabric must be designed to absorb it without disturbing training.

A defensible checkpoint policy is a coupled set of decisions, not a single setting — together they define how much of a failure is recoverable. Size the state (~14 bytes/param × your optimizer recipe). Measure MTBF from live fault telemetry and recalculate the interval with the selected Young first-order or Daly higher-order model as burn-in clears. Make C small with asynchronous drains and a sharded, reshard-capable format, and add an in-memory/peer tier so the common single-node failure recovers in seconds. Keep a less-frequent durable copy for correlated failures, with a commit protocol so you never load a torn checkpoint. Then size both the drain bandwidth and the recovery incast, and isolate them from the compute fabric. Do all of that and you are operating in the 96% goodput regime; skip the tiering or mis-size the interval and you are leaving 6–16 points of goodput — and the proportional fraction of a power-bound, depreciating GPU fleet — on the floor.