Overprovisioning needed for a month-long synchronous job on 608 H100 GPUs to hold 99.9% job-level availability falls from 5% to 2% (2.5x) as modeled recovery time drops from 2.2 hours to 5 minutes
5% → 2%modeled
| Value kind | modeled — Interpret this value according to its displayed kind, scope, source, and as-of date. |
|---|---|
| Scope | Discrete-event simulation driven by Delta's measured failure distributions (per-node availability about 99.4% for A100, 99.3% for H100). Recovery time is a swept parameter covering checkpoint load and spare availability; 2.2 h is Delta's measured H100 node service time, mostly drain and reboot. The 608 GPUs are the H100s in Delta's four-way GH200 nodes. It is not a stocked-FRU or inventory experiment. Earlier arXiv versions (March 2025) reported 5-20% and a 4x reduction from a 99.5% baseline; a June 2025 revision superseded those figures. |
| As of | 2025-12 |
| Source | Cui et al., Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100 GPUs, SC '25 (arXiv 2503.11901 v4) · Projected impact of availability on long-running and large-scale jobs (5% overprovisioning at 2.2 h recovery; 2% at 5 min) |
| Review | checking…review by 2027-06-01 · standard cadence |
| Recorded changes | last 2026-10-16 · 2 revisions tracked |
| Claim id | synchronous-job-overprovisioning-cut-when-gpu |
Where the guide uses it
← Full numbers register — every date-stamped figure in the guide, with revision history.