The AI-Cluster Reliability Rethink: Goodput vs Facility Availability

For sixty years the data-center industry optimized one number: availability — the fraction of time the facility is energized and cooled, expressed as a string of nines and certified against a tier. A Tier III site promises concurrent maintainability and roughly 99.982% uptime (about 1.6 hours of down a year); a Tier IV site adds fault tolerance and roughly 99.995% (about 26 minutes a year). That metric was correct for the workload it was built around: enterprise applications and web services where the unit of value is a transaction, an outage is a binary up/down event, and a single rack going dark is a contained, recoverable nuisance. Redundancy — N+1, 2N, block- and distributed-redundant power, dual cooling paths — exists to push that one number toward unity.

An AI factory breaks the assumption underneath the metric. A frontier training job is one tightly-coupled supercomputer running synchronously across tens of thousands of accelerators; a single failed GPU forces the entire job to restart from its last checkpoint. The facility can be at 100.000% availability — every breaker closed, every CDU pumping — and the cluster can still be throwing away a fifth of the money you spent on it, because the GPUs are idle waiting on a straggler, replaying lost steps, or stalled mid-checkpoint. Availability measures the building. It does not measure the work. The metric that measures the work is goodput, and this chapter is the argument that goodput — not the facility's nines — is the number that governs return, and therefore the number your redundancy budget should be optimizing.

Define the terms precisely, because the whole rethink lives in the gap between them. Facility availability is a property of the physical plant: the probability that power and cooling are present at the rack, measured at the building boundary, certified by topology (Uptime Tier, TIA-942 Rated level, EN 50600 Availability Class). Goodput is a property of the workload: the fraction of provisioned accelerator-time that produces preserved, useful progress. Google's formulation decomposes it as ML Goodput = Scheduling Goodput × Runtime Goodput × Program Goodput — resources available × time spent making preserved progress × effective FLOP utilization — and everything that is not goodput is badput: accelerator init, JIT compilation, data-loading stalls, checkpoint save and restore, wasted progress replayed after a failure, and infrastructure recovery during restarts (Google Cloud, 2024–2025).

The two numbers diverge because most badput is invisible to the facility. When the building loses power, both availability and goodput drop — they agree. But the dominant losses in a real cluster are not facility outages. Meta's published Llama 3 405B snapshot recorded 419 unplanned interruptions over 54 days on 16,384 H100s — roughly one every three hours — of which 78% were hardware-caused and 58.7% traced to GPUs or HBM (Meta, 2024). None of those is a building outage. Every one of them is goodput lost while the facility sat at a flawless 100% availability. The relationship runs one way: facility availability is a ceiling on goodput, never a floor. You can have perfect availability and mediocre goodput; you cannot have poor availability and good goodput. Optimizing the ceiling while the workload bleeds out below it is the central mis-allocation.

The deepest consequence of the goodput reframe is that redundancy migrates off the facility one-line diagram and into the compute stack. In the legacy model, resilience was almost entirely a facility property — you bought it as 2N power, dual cooling, and a generator yard, and the IT equipment was a passive load that the facility protected. In the AI model, the workload protects itself: synchronous training already tolerates node loss by checkpoint-and-resume, so the resilience that matters is how fast it recovers, which is a silicon, storage, and software question, not a switchgear question.

This changes the design-basis arithmetic at every layer. In the silicon: per-GPU capacitance (65 J/GPU on GB300, ~400 J/GPU on Vera Rubin) and rack BBUs ride through the millisecond-to-second transients that used to be the UPS's job, while the facility BESS is repurposed from outage ride-through to transient absorption (NVIDIA / SemiAnalysis, 2025–2026). In storage: multi-tier checkpointing collapses mean-time-to-recovery from the 15–30 minutes of a remote-filesystem restore to under two minutes by staging checkpoints in node-local and peer memory first (Google Cloud, 2025) — a goodput improvement the facility one-line cannot deliver at any redundancy tier. In software: elastic training continues on a shrunken world size instead of halting, and inference fabrics retry failed requests against healthy replicas. A dollar spent lifting facility power from N+1 to 2N prevents a class of outage the synchronous job already survives via checkpoint; the same dollar spent on a faster checkpoint tier or a hot-spare pool directly converts badput into goodput. The question is which layer resilience earns its keep in, not how many nines you buy.

Read the table as a spend-allocation guide, not a feature list. For a checkpointable training cluster the rational priority order runs roughly opposite to the legacy facility instinct: checkpoint tiering and hot spares first, cooling continuity second, facility power redundancy last. For an always-on inference fleet the order inverts again — facility availability and cooling continuity climb back up because an outage is unrecoverable revenue. The same building, two different workloads, two opposite redundancy budgets. Designing to goodput forces you to allocate the redundancy capital where the workload loses money, which is rarely where the facility tier chart tells you to spend it.

The most under-appreciated consequence of the density ramp is that the cooling system became the dominant single point of cluster-wide failure, and it lost its shock absorber at the same time. An air-cooled hall carried enormous thermal inertia: chilled-water volume, the air mass of the room, raised-floor plenum. A CRAH failure gave operators minutes of ride-through before inlet temperatures climbed — time to fail over, time to intervene. Direct-to-chip liquid cooling deletes that buffer. A GB200-class cold plate sits microns from a ~1,000 W die with a coolant inlet that must stay below ~25 °C and a delta-T held under ~10 °C; lose flow and junction temperature climbs not in minutes but in seconds, and the GPUs throttle — up to 50% — or trip to protect the silicon.

This relocates the availability problem. The facility can hold power at Tier-IV nines and still take the entire cluster down through a coolant-distribution-unit fault, a pump trip, or a control-loop oscillation, because the technology-cooling loop now has no inertia to coast on. The design-basis response is to push redundancy into the loop the way the legacy world pushed it into the power chain: N+1 (or 2N) CDUs, redundant pumps with automatic ride-through, dual-fed coolant loops, and leak-detection with negative-pressure containment so a breach throttles rather than floods (Vertiv / Equinix / Chilldyne, 2026). Concurrent maintainability — the Tier-III property the industry already values — has to be re-earned in the liquid path: you must be able to pull a pump or service a heat exchanger without dropping the rack. Skimp on cooling-loop redundancy and you have built a cluster whose availability is capped by its weakest pump, no matter how many nines the power chain carries. → Chapter 12.1 sets the topology vocabulary; the DLC continuity engineering is in Chapter 5.4.

The reliability rethink runs in both directions. The cluster's own reliability depends on the facility — but the facility has become a reliability problem for the grid, and that coupling now feeds back into the cluster's design-basis. AI training loads are phase-coherent and synchronized: tens of thousands of GPUs step from idle to peak and back together, every training step, producing load swings of hundreds of megawatts on sub-second timescales. Worse, the GPUs' own protection can drop the entire load at once. In a 2024 Virginia event, ~1,500 MW of data-center load tripped off on a single 230 kV fault, with roughly 1.5 GW dropped in 82 seconds — enough that the surviving generation had to absorb the imbalance, and enough that NERC issued a rare Level 3 Essential Actions Alert and now treats large data centers as grid actors with mandatory fault-ride-through obligations (NERC / Utility Dive, 2026).

Ride-through is no longer optional, and it is no longer purely a grid-interconnection concern — it is a goodput concern. A cluster that trips off on every grid disturbance to protect itself converts a recoverable grid event into a full cluster restart: maximum badput. The mitigation is the same transient-absorption stack that protects against the cluster's own load swings — per-GPU capacitance, rack BBUs, facility BESS, and intelligent power smoothing that has demonstrated ~30% reductions in peak grid demand on real training jobs — now also tuned to keep the cluster online through utility-side faults rather than dropping load (NVIDIA / SemiAnalysis, 2025–2026). The fork: engineer the facility to ride through grid disturbances (cost: storage and smoothing capex, plus a regulator-facing study) or accept that grid noise becomes cluster restarts (cost: goodput, plus a worsening relationship with a utility that can throttle your interconnection). → the full grid-interactive engineering — reactive support, frequency response, ride-through curves at the point of interconnection — is canonical in Chapter 4.10; the storage that backs it in Chapter 4.5.

None of this means the standards are wrong — it means they answer a question that is no longer the binding one. Uptime Tiers, TIA-942 Rated levels, and EN 50600 Availability Classes all certify the facility's availability, and none of them certify goodput. A Tier IV building tells a tenant the power and cooling will be present 99.995% of the time; it says nothing about whether the cluster inside it loses 10% or 20% of its bought GPU-hours to badput the facility never sees. The standards remain the right tool for the question they answer — they govern the ceiling — but they cannot be the design-basis metric for a workload whose return lives in the gap below that ceiling.

The practical reconciliation is a two-tier design-basis: certify the facility to the availability class the workload's floor requires (high for always-on inference, deliberately modest for checkpointable training), then run a separate goodput design-basis that governs the silicon/storage/software redundancy the standards never touch. This is also where the redundancy primer in Chapter 0.5 gets extended: N, N+1, 2N and the distributed-redundant topologies are still the vocabulary, but you now apply them in two places — the facility power/cooling chain and the compute resilience stack — and the goodput model decides which application earns the spend. The standards landscape and topology selection are detailed in Chapter 12.1; the SLA that contracts goodput rather than availability is the subject of Chapter 12.4.

Put the two metrics on one curve and the design decision becomes a spending question with a real crossover. On the x-axis, redundancy capital; on the y-axis, realized goodput. Early dollars spent on the workload's own dominant failures — checkpoint tiering, hot spares, cooling continuity, ride-through — buy steep goodput gains, because they attack the badput that actually dominates the loss budget (GPU/HBM faults, slow recovery, transient trips). Past a point, the next dollar can only go to facility availability — lifting power from N+1 toward 2N+ — and for a checkpointable training job that dollar buys almost no goodput, because the job already survives the outage class that spend prevents. The curve flattens, and the flattening point is where you should stop buying nines.

For an always-on inference fleet the curve has a different shape: facility availability stays on the productive part of the curve much longer, because every outage is unrecoverable revenue rather than a checkpoint replay, so 2N power and dual cooling keep paying. The right redundancy budget is therefore workload-specific and quantitative: it is the point where the marginal goodput per dollar of facility availability falls below the marginal goodput per dollar of compute-stack resilience — and finding that point for your actual failure environment is exactly what the reliability model does. It takes component failure rates (Chapter 14.3) and FMEA scenarios (Appendix F) as inputs and rolls them up to cluster-level availability and goodput, then runs the sensitivity analysis that locates the crossover. → the model is built in Chapter 12.5; the failure-rate inputs in Chapter 14.3.