Checkpoint overlap (share of training time with a background checkpoint drain running): nearly every one of 40 production runs kept its median under 10% in VAST's analysis, and VAST sizes global checkpoint bandwidth on 10%
10% overlapobserved
| Value kind | observed — Reported measurements, counts, and specifications keep the precision and scope stated by their source; an exact specification is not treated as a range. |
|---|---|
| Scope | VAST defines overlap as the fraction of training time during which background checkpoint I/O is running; its 800B-parameter example (3.6-minute median drain on a 40-minute interval) comes to about 9%. Across 40 runs on more than 1,024 GPUs, almost every run kept median overlap under 10%, and VAST concludes 10% is sufficient for its sizing model. The figure describes those runs; a project's drain window still comes from its own state-age, pause and commit limits. |
| As of | 2025 |
| Source | VAST Data, 'Optimizing Checkpoint Bandwidth for LLM Training' (overlap results and sizing model) |
| Review | checking…review by 2027-04-20 · standard cadence |
| Recorded changes | last 2026-10-19 · 2 revisions tracked |
| Claim id | checkpoint-vs-compute-overlap-target-found |
Where the guide uses it
← Full numbers register — every date-stamped figure in the guide, with revision history.