observed interruption attribution in one Llama 3 405B run: 148 faulty-GPU and 72 HBM3 events among 419 unplanned interruptions
148 GPU + 72 HBM interruptionsobserved
| Value kind | observed — Reported measurements, counts, and specifications keep the precision and scope stated by their source; an exact specification is not treated as a range. |
|---|---|
| Scope | Counts are event counts, not unique failed FRUs or equipment AFR. They describe the 54-day snapshot of one run on up to 16,384 H100s. |
| Caveat | Table 5's 18 counts sum to the 419 unexpected interruptions. One printed percentage is off: faulty GPU is printed as 30.1%, but 148/419 is 35.3%; the other rows reconcile. Use the counts. Network switch/cable is 35 events (8.4%); the NIC is classified as Host and NCCL watchdog timeouts as Unknown. |
| As of | 2024-07 |
| Source | Meta AI, The Llama 3 Herd of Models, §3.3.4/Table 5 · §3.3.4, Table 5: 466 total interruptions, 47 planned, 419 unexpected; rows for faulty GPU, HBM3, software, network, and the remaining categories |
| Review | checking…review by 2026-12-22 · standard cadence |
| Recorded changes | last 2026-08-24 · 3 revisions tracked |
| Claim id | share-of-interruptions-gpu-hbm-related-gpu-30-1 |
Where the guide uses it
- 12.5 Quantitative Reliability & Availability Modeling (RBD / FTA / Monte-Carlo)
- 13.8 GPU Node Burn-In, Diagnostics & Stress Validation
- 14.3 Component Failure Modes, Failure Rates & Fleet Reliability Data
← Full numbers register — every date-stamped figure in the guide, with revision history.