The AI-Cluster Reliability Rethink: Goodput vs Facility Availability

For sixty years the data-center industry optimized one number: availability — the fraction of time the facility is energized and cooled, expressed as a string of nines and certified against a tier. A Tier III site promises concurrent maintainability; a Tier IV site adds fault tolerance — topology guarantees, not the percentage-uptime folklore still quoted from pre-2009 Uptime documents (Chapter 12.1). That metric was correct for the workload it was built around: enterprise applications and web services where the unit of value is a transaction, an outage is a binary up/down event, and a single rack going dark is a contained, recoverable nuisance. Redundancy — N+1, 2N, block- and distributed-redundant power, dual cooling paths — exists to push that one number toward unity.

An AI factory breaks the assumption underneath the metric. A frontier training job is one tightly-coupled supercomputer running synchronously across tens of thousands of accelerators; a single failed GPU forces the entire job to restart from its last checkpoint. The facility can be at 100.000% availability — every breaker closed, every CDU pumping — and the cluster can still be throwing away a fifth of the money you spent on it, because the GPUs are idle waiting on a straggler, replaying lost steps, or stalled mid-checkpoint. Goodput — the share of bought GPU-hours that becomes useful work — is the number that governs return, and therefore the one the redundancy budget should be optimizing.

Define the terms precisely, because the whole rethink lives in the gap between them. Facility availability is a property of the physical plant: the probability that power and cooling are present at the rack, measured at the building boundary, certified by topology (Uptime Tier, TIA-942 Rated level, EN 50600 Availability Class). Goodput is a property of the workload: the fraction of provisioned accelerator-time that produces preserved, useful progress. Google's formulation decomposes it as ML Goodput = Scheduling Goodput × Runtime Goodput × Program Goodput — resources available × time spent making preserved progress × effective FLOP utilization — and everything that is not goodput is badput: accelerator init, JIT compilation, data-loading stalls, checkpoint save and restore, wasted progress replayed after a failure, and infrastructure recovery during restarts (Google Cloud, 2024–2025).

The two numbers diverge because most badput is invisible to the facility. When the building loses power, both availability and goodput drop — they agree. But the dominant losses in a real cluster are not facility outages. Meta's published Llama 3 405B snapshot recorded 419 unplanned interruptions over 54 days on 16,384 H100s — roughly one every three hours — of which the paper attributes approximately 78% to hardware and reports 58.7% as GPU issues, while its table counts and percentages do not reconcile (Meta, 2024). Those are cluster-side interruptions, not evidence of a building outage; the paper does not report facility availability for the snapshot. The relationship runs one way: facility availability is a ceiling on goodput, never a floor. You can have perfect availability and mediocre goodput; you cannot have poor availability and good goodput. Optimizing the ceiling while the workload bleeds out below it mis-allocates the redundancy budget.

The deepest consequence of the goodput reframe is that resilience spans the facility and compute stack. Power paths, cooling paths, controls, silicon ride-through, checkpoint storage, spare capacity, orchestration, request retry, and fleet failover each address different failure states. Checkpoint-and-resume lowers the consequence of some training interruptions, but does not prove that a whole-hall or distribution-path interruption is acceptable.

Assign every defined maintenance and fault state to the mechanism that detects, contains, rides through, or recovers it. Per-GPU capacitance, rack BBUs, and facility BESS can cover named millisecond-to-second transients; multi-tier checkpointing and hot spares can reduce training replay and recovery; replica and region capacity can mask serving failures. Facility topology still carries the states that those mechanisms cannot bridge. Compare the incremental cost of N, N+1, distributed, or 2N with compute-stack and fleet resilience using transfer interruption, post-event loading, independence, common modes, recovery SLO, and contract.

The table is a spend-allocation guide. For a checkpointable training cluster the rational priority order runs roughly opposite to the legacy facility instinct: checkpoint tiering and hot spares first, cooling continuity second, facility power redundancy last. For an always-on inference fleet the order inverts again — facility availability and cooling continuity climb back up because an outage is unrecoverable revenue. The same building with two different workloads yields two opposite redundancy budgets. Designing to goodput forces you to allocate the redundancy capital where the workload loses money, which is rarely where the facility tier chart tells you to spend it.

The most under-appreciated consequence of the density ramp is that coolant flow became a dominant single point of cluster-wide failure, and the loop now operates on two different transient clocks. An air-cooled hall carried enormous thermal inertia: chilled-water volume, the air mass of the room, raised-floor plenum. A CRAH failure gave operators minutes of ride-through before inlet temperatures climbed — time to fail over, time to intervene. At maintained flow, coolant and metal mass may buffer a heat step; when flow stops, the selected rack's throttle, controlled-shutdown, emergency-shutdown and no-response states must be established from OEM transient data, controls tests or an engineering calculation, even where the rack's supported supply and return limits are separately declared.

This relocates the availability problem. The facility can hold power at Tier-IV nines and still take the entire cluster down through a coolant-distribution-unit fault, a pump trip, or a control-loop oscillation, because the technology-cooling loop's flow-loss ride-through is product- and implementation-specific and must be validated rather than compared using a universal seconds-versus-minutes rule. The design-basis response is to size CDUs, pumps, loops and protection from that validated transient, maintenance case and workload consequence; no universal N+1/2N topology is implied. so a breach throttles rather than floods (Vertiv / Equinix / Chilldyne, 2026). Concurrent maintainability — the Tier-III property the industry already values — has to be re-earned in the liquid path: you must be able to pull a pump or service a heat exchanger without dropping the rack. Skimp on cooling-loop redundancy and you have built a cluster whose availability is capped by its weakest pump, no matter how many nines the power chain carries. → Chapter 12.1 sets the topology vocabulary; the DLC continuity engineering is in Chapter 5.4.

The reliability rethink runs in both directions. The cluster's own reliability depends on the facility — but the facility has become a reliability problem for the grid, and that coupling now feeds back into the cluster's design-basis. AI training loads are phase-coherent and synchronized: tens of thousands of GPUs step from idle to peak and back together, every training step, producing load swings of hundreds of megawatts on sub-second timescales. A separate mechanism can drop the entire load: protection response to a grid fault. In a 2024 Virginia event, ~1,500 MW of data-center load tripped off during a six-fault, 82-second sequence on a 230 kV line — enough that the surviving generation had to absorb the imbalance, and enough that NERC issued a rare Level 3 Essential Actions Alert and now treats large data centers as grid actors expected to ride through faults — an expectation the alert recommends but does not yet enforce, with a penalty-backed reliability standard still in development (NERC / Utility Dive, 2026).

Ride-through has become a goodput concern as much as a grid-interconnection one. A cluster that trips off on every grid disturbance to protect itself converts a recoverable grid event into a full cluster restart: maximum badput. The mitigation is the same transient-absorption stack that protects against the cluster's own load swings — per-GPU capacitance, rack BBUs, facility BESS, and intelligent power smoothing that has demonstrated ~30% reductions in peak grid demand on real training jobs — now also tuned to keep the cluster online through utility-side faults rather than dropping load (NVIDIA / SemiAnalysis, 2025–2026). The choice is to engineer the facility to ride through grid disturbances — storage and smoothing capex, plus a regulator-facing study — or to accept that grid noise becomes cluster restarts, paid for in goodput and in a worsening relationship with a utility that can throttle your interconnection. → the full grid-interactive engineering — reactive support, frequency response, ride-through curves at the point of interconnection — is canonical in Chapter 4.10; the storage that backs it in Chapter 4.5.

None of this means the standards are wrong — it means they answer a question that is no longer the binding one. Uptime Tiers, TIA-942 Rated levels, and EN 50600 Availability Classes all certify the facility's availability, and none of them certify goodput. A Tier IV building tells a tenant the power and cooling ride through any single worst-case event; it says nothing about whether the cluster inside it loses 10% or 20% of its bought GPU-hours to badput the facility never sees. The standards remain the right tool for the question they answer — they govern the ceiling — but they cannot be the design-basis metric for a workload whose return lives in the gap below that ceiling.

The practical reconciliation is a two-tier design-basis: certify the facility to the availability class the workload's floor requires (high for always-on inference, deliberately modest for checkpointable training), then run a separate goodput design-basis that governs the silicon/storage/software redundancy the standards never touch. This is also where the redundancy primer in Chapter 0.5 gets extended: N, N+1, 2N and the distributed-redundant topologies are still the vocabulary, but you now apply them in two places — the facility power/cooling chain and the compute resilience stack — and the goodput model decides which application earns the spend. The standards landscape and topology selection are detailed in Chapter 12.1; the SLA that contracts goodput rather than availability is the subject of Chapter 12.4.

Put the mechanisms on one economic curve: incremental lifecycle cost on the x-axis and realized service value or goodput on the y-axis. Early dollars should attack the measured dominant loss states, which may include checkpoint/recovery, silicon spares, cooling continuity, ride-through, facility-path continuity, or fleet failover. The curve flattens only when the next mechanism prevents less expected loss per dollar than the alternatives. Checkpointing alone does not locate that point, because it does not prevent every facility state and its recovery may still breach the contract.

Serving fleets also require a state model: request retry and independent replica, zone, or region capacity can mask a site event, so a single-site outage is not automatically unrecoverable and 2N is not automatic. The right budget is the point where the marginal value per dollar equalizes across facility, compute-stack, and fleet resilience. The model must include failure rates, maintenance states, transfer interruption, post-event capacity, path and control independence, common modes, recovery, and contractual penalties. → Chapter 12.5.