Projected mean time to failure for a 16,384-GPU synchronous job, from Meta's 1/(nodes × failure rate) formula at RSC-1's 6.50 failures per 1,000 node-days
~1.8 hr (projection)forecast
| Value kind | forecast — Forecasts are time-bound outlooks, not specifications; test decisions across plausible scenarios. |
|---|---|
| Scope | A projection, not an observation: Meta divides one by node count times per-node failure rate, and 2,048 nodes at RSC-1's 6.50 failures per 1,000 node-days gives about 1.8 h. v2 (February 2025) keeps the figure. Meta's observed MTTFs by job size, such as 7.9 h at 1,024 GPUs, are measured time over measured failures and imply a higher per-node rate; keep them out of any model built on this projection. The same formula gives 0.23 h at 131,072 GPUs, which holds only if nodes keep failing independently at an unchanged rate. The Llama 3 405B snapshot (419 unexpected interruptions in 54 days on up to 16,384 H100s, about 3 h between interruptions) counts a third event population. |
| As of | 2024-10 |
| Source | Meta, Revisiting Reliability in Large-Scale ML Research Clusters · Section on MTTF by job size: 'we project the MTTF for 16384 GPU jobs to be 1.8 hours and for 131072 GPU jobs to be 0.23 hours', from MTTF = (N_nodes × r_f)^-1 (v2, February 2025) |
| Review | checking…review by 2027-10-16 · standard cadence |
| Recorded changes | last 2026-10-16 · 4 revisions tracked |
| Claim id | mean-time-to-failure-for-a-16-384-gpu |
Where the guide uses it
← Full numbers register — every date-stamped figure in the guide, with revision history.