Google-reported checkpoint restore time with multi-tier checkpointing across a cluster of thousands of nodes, versus up to 30 minutes from object storage alone; one clock inside MTTR
restore <1 minobserved
| Value kind | observed — Reported measurements, counts, and specifications keep the precision and scope stated by their source; an exact specification is not treated as a range. |
|---|---|
| Scope | Vendor-reported general statement by Google for clusters with thousands of nodes, not a measurement tied to the 35K-chip TPU v5p workload (that workload is the source of the separate 6.59% ML Goodput figure). Excludes detection, fencing, allocation, worker launch and validation; not an end-to-end MTTR. |
| As of | 2025-06-17 |
| Source | Google Cloud, Save early and often with multi-tier checkpointing to optimize large AI training jobs, 17 June 2025. · Multi-tier checkpointing overview: restore in under a minute across a cluster with thousands of nodes |
| Review | checking…review by 2027-06-17 · standard cadence |
| Recorded changes | last 2026-10-16 · 4 revisions tracked |
| Claim id | mttr-achievable-with-multi-tier-checkpointing |
Where the guide uses it
← Full numbers register — every date-stamped figure in the guide, with revision history.