The AI-Cluster Reliability Rethink: Goodput vs Facility Availability

For sixty years the data-center industry optimized one number: availability, the fraction of time the facility is energized and cooled over a declared window. Tier certification is a separate assessment of topology outcomes. A Tier III site promises concurrent maintainability and a Tier IV site adds fault tolerance. Those are topology guarantees, not the percentage-uptime folklore still quoted from pre-2009 Uptime documents (Chapter 12.1). The metric fit the workload it was built around: enterprise applications and web services, where the unit of value is a transaction, an outage is a binary up/down event, and one dark rack is a contained, recoverable nuisance. Redundancy (N+1, 2N, block- and distributed-redundant power, dual cooling paths) exists to push that number toward unity.

An AI factory breaks the assumption underneath the metric. A frontier training job is one tightly coupled supercomputer running synchronously across tens of thousands of accelerators. If a failed GPU takes out a required rank, a restart-all job falls back to its last surviving checkpoint, and replica-group recovery needs its own state and quorum rules. The facility can sit at 100.000% availability, every breaker closed and every CDU pumping, while the cluster can still throw away a tenth or more of the money you spent on it. The GPUs are idle waiting on a straggler, replaying lost steps, or stalled mid-checkpoint. Goodput, the share of bought GPU-hours that becomes useful work, governs return, so it is the number the redundancy budget should optimize.

The whole rethink lives in the gap between two metrics, so define them precisely. Facility availability is a property of the physical plant: the fraction of the observation window in which the required power and cooling conditions hold at the load interface. The facility classification is a separate statement about assessed topology and operating outcomes. ML Productivity Goodput is a property of the workload. Google decomposes it as ML Productivity Goodput = Scheduling Goodput × Runtime Goodput × Program Goodput, which is resource availability × retained forward-progress time × effective FLOP utilization.

Keep three labels apart, because they carry different denominators. Provider Goodput (scheduling × runtime) is the part a contract can commit. ML Productivity Goodput multiplies that by MFU. Meta's ETTR, or effective training time, is a productive-time observation that excludes MFU entirely. Take an illustrative Scheduling Goodput of 100%, Runtime Goodput of 90% and Program Goodput (MFU) of 40%. Provider Goodput is then 90% and ML Productivity Goodput is 36%: one fleet, two denominators, and numbers more than a factor of two apart. Low Program Goodput cuts FLOP productivity while retained-progress time stays unchanged, so the contract must say which metric and which code obligations it covers. In the scheduling and runtime terms, badput includes accelerator init, JIT compilation, data-loading stalls, checkpoint save and restore, wasted progress replayed after a failure, and infrastructure recovery during restarts (Google Cloud, 2024–2025).

Treat provider figures the same way. CoreWeave's October 2026 post reports mean time to failure improved about 10x and effective training time reached up to 98%. It frames the figures around a 1,024-GPU Hopper cluster benchmark but gives no time period, baseline or list of excluded downtime. Modeled numbers differ again: the April 2026 SemiAnalysis goodput-expense scenario prices extra cost under assumed fault-tolerance methods and is not a measured ETTR. Before you compare either with Meta's measured effective training time of just over 90%, pin down which metric it is (ETTR, Provider Goodput or ML Productivity Goodput) and what job size, window and exclusions sit behind it. Chapter 12.4 turns those answers into contract terms.

Availability and goodput diverge because most badput is invisible to the facility. When the building loses power, availability and goodput both drop, and they agree. But the dominant losses in a real cluster are not facility outages. Meta's published Llama 3 405B snapshot recorded 419 unplanned interruptions over 54 days on up to 16,384 H100s, roughly one every three hours. Table 5 attributes 148 events to faulty GPUs and 72 to HBM3 errors (Meta, July 2024). Those are cluster-side interruptions, not evidence of a building outage, and the paper does not report facility availability for the snapshot.

For the same GPUs, window and eligible-time denominator, facility availability is a ceiling on retained-work time, never a floor. You can keep every required facility service present and still throw away progress below it. The ceiling also does not compare one site with a surviving regional fleet or an exclusion-adjusted contract. State those boundaries, then price the lost work: polishing the ceiling while the workload bleeds below it misallocates the redundancy budget.

The deepest consequence of the goodput reframe is that redundancy migrates out of the facility and into the silicon and the software. In the availability model, every resilience dollar bought a redundant power or cooling path. In the goodput model, mechanisms above the facility boundary compete for recovered GPU-hours per dollar. Multi-tier checkpointing cuts a training restore from tens of minutes to under a minute. With fast health-check and drain, a hot-spare pool replaces a failed node in minutes, where the old route was a fabric re-cable. Per-GPU capacitance smooths millisecond-scale swings, while rack BBUs and facility BESS ride through the longer transients the cluster's own load swings cause. For serving, replica and region capacity masks a site event the way a second power path never could.

The facility chain still owns the states none of those can bridge, such as a distribution-path fault that drops a whole hall or a coolant-flow loss that trips every rack on the loop, and it must be sized for them. For a checkpointable fleet whose measured losses are dominated by replay and capacity waits, the checkpoint tier and spare pool move ahead of a second utility path once the required facility states are covered.

The table is a spend-allocation guide, and it reads differently for the four fleets an operator actually runs.

For frontier synchronous training, the ranking is checkpoint tier, spare pool, cooling continuity, transient ride-through, then facility path redundancy. The job already restarts every few hours, so a rare hall event adds little to a loss the first two rows have bounded. That order holds while measured losses are dominated by replay and capacity waits. A required-rank loss interrupts the run either way, so compare interruption frequency, durable progress and complete recovery before buying. For smaller checkpointable training the order holds and the facility rows fall further.

For frontier rack-scale inference on a single site, flip it. There is no checkpoint to fall back on, the SLA is measured in tail latency, and a hall outage is unmasked unless a second region absorbs it, so cooling continuity and the second power path move to the top. For replicated enterprise inference across regions, the fleet masks the site, provided a surviving replica also has weights, keys, session state and sufficient capacity. The facility rows are bought to the contract's floor and no further.

Two things hold across all four. Loss of coolant flow is a hall-wide outage that pump and CDU redundancy usually prevents at low cost relative to its consequence, and the contract, not the workload label, sets the floor the facility must certify. Chapter 12.3 selects regional recovery capacity.

torchft illustrates recovery for supported replicated training arrangements. Test the actual algorithm and surviving group, because its support does not extend automatically to every training stack.

The most under-appreciated consequence of the density ramp is that coolant flow became a dominant single point of cluster-wide failure, and the loop now operates on two different transient clocks. An air-cooled hall carried enormous thermal inertia in its chilled-water volume, the air mass of the room and the raised-floor plenum. After a CRAH failure there can be time to fail over or intervene, when the coupled thermal mass holds inlet temperature inside its limit. Measure that window for the actual failed component.

Liquid cooling splits the problem in two. If flow is maintained after a loss of heat rejection, the coolant and metal in the loop buffer a heat step for tens of seconds. If flow stops, only the thin film of coolant and the cold-plate metal at the die absorb heat, and the distant stored water no longer helps. No GPU vendor publishes a universal seconds-to-throttle figure for stopped flow. The nearest published measurement, a 2018 Binghamton study of CPU servers on direct liquid cooling, found that a cooling failure can force shutdown in under a minute (Alkharabsheh et al., Journal of Electronic Packaging). Your rack's throttle and trip clock comes from the OEM's transient data and a controlled commissioning pump-drop test; an air-hall runbook cannot supply it.

This relocates the availability problem. A facility can hold a fault-tolerant power topology and still lose the entire cluster to a coolant-distribution-unit fault, a pump trip or a control-loop oscillation, whenever the interruption outlasts what the racks on that loop can tolerate. The assessment record must say whether the in-rack path is included. The design-basis response is to make coolant continuity a first-class redundancy line: redundant pumps and heat exchangers in the CDU, protected pump power where the ride-through study shows it is needed (Chapter 5.12), isolation that lets one CDU fail without starving the loop, and a commissioning test that drops a pump at full load and proves the ride-through. That is the posture behind the ~99.999% fleet-wide CDU availability Google reported in April 2025 for its liquid-cooled TPU fleet since 2020, a vendor figure with no published definition or method (Chapter 5.11).

Concurrent maintainability, the Tier III property the industry already values, has to be re-earned in the liquid path: you must be able to pull a pump or service a heat exchanger without dropping the rack. Skimp on cooling-loop redundancy and you have built a cluster whose availability is capped by its weakest pump, however many nines the power chain carries. Chapter 12.1 sets the topology vocabulary, and the DLC continuity engineering is in Chapter 5.4.

The reliability rethink runs in both directions. The cluster's reliability depends on the facility, but the facility has also become a reliability problem for the grid, and that coupling now feeds back into the cluster's design basis. AI training loads are phase-coherent. Tens of thousands of GPUs can move between low and peak power in step, at checkpoints, collective waits and job starts or stops. Meta (2024) and Google (2025) both report data-center power swings on the order of tens of megawatts within a second, and larger sites can swing more.

A separate mechanism can drop the entire load: the protection response to a grid fault. In July 2024 a 230 kV line in Northern Virginia suffered six faults over 82 seconds, and the affected data-center facilities transferred to backup power. About 1,500 MW of their demand left the grid. Other generators had to absorb the imbalance. NERC answered on 4 May 2026 with a rare Level 3 Essential Actions alert. Its recipients run from transmission planners, owners and operators to the planning coordinators, balancing authorities and reliability coordinators. It asks them to model large computational loads, study what losing one would do, and, where they can, commission each load with its computing equipment installed. It also asks transmission owners to work with data-center customers, where possible, to improve facility ride-through, but it sets no ride-through requirement on data centers themselves.

On 22 July 2026 the pattern repeated at larger scale. This time one 230 kV fault in Ashburn was enough, and protection cleared it normally: about 3.8 GW of load still transferred to backup in two waves (PJM put it at nearly 4,000 MW). That is roughly 2.5 times the 2024 event. PJM responded in September 2026 with a proposal that existing and new large loads ride through normally cleared disturbances, and it plans to file it with FERC in November 2026. It is a proposal, not yet a rule.

The alert's Essential Actions are not mandatory standards. On 8 October 2026 the NERC Board approved registry criteria for the new Computational Load Owner and Operator roles, and NERC's computational-load standards passed their final ballot (announced 15 October). The standards still await NERC Board adoption, planned for December, and then FERC approval; NERC's filing is due by 31 December 2026, and none of it binds operators before FERC approves (NERC; PJM, 2026). The standards will cover sites of 50 MW or more supplied at 100 kV or above, and they set no ride-through curve. Chapter 4.3 tracks registration, and Chapter 4.10 owns ride-through.

Ride-through is now a goodput concern as much as a grid-interconnection one. A cluster that actually loses IT power, cooling or required state during a survivable grid disturbance turns that event into a full restart, paid in badput. A grid-visible transfer to backup while the IT keeps running costs no restart. Rack BBUs and facility BESS protect against both grid disturbances and the cluster's own load swings; the capacitor layer protects only against the swings.

NVIDIA specifies and controls its capacitor storage for smoothing fast load swings (NVIDIA / SemiAnalysis, 2025–2026): 65 J per GB300 GPU, and ~400 J per Vera Rubin GPU, where the capacitors form part of the Intelligent Power Smoothing system NVIDIA introduced with Vera Rubin. NVIDIA states the Vera Rubin system can reduce peak current demand by up to 25%. It gives the storage no utility-fault ride-through rating, so do not count it toward ride-through unless the OEM rates it for that. BBUs, BESS or UPS, with the right protection settings, are what keep the cluster online through utility-side faults rather than dropping load. The choice is to engineer the facility to ride through grid disturbances, which costs storage and smoothing capex plus a regulator-facing study, and to test delivered IT power, coolant response and recovery on the same event clock. Otherwise a survivable grid disturbance can become a cluster restart paid in goodput, while the utility still sees an unexplained load departure. The full grid-interactive engineering (reactive support, frequency response, ride-through curves at the point of interconnection) is canonical in Chapter 4.10, and the storage that backs it is in Chapter 4.5.

None of this means the standards are wrong. They answer a question that is no longer the binding one. Uptime Tiers, TIA-942 Rated levels, and EN 50600 Availability Classes assess their defined infrastructure requirements; none certifies a numerical goodput guarantee. A Tier IV building tells a tenant that the assessed power and cooling paths preserve the required service through the Tier IV fault conditions, with continuous cooling and separately stated outcomes for additional faults during maintenance. It says nothing about whether the cluster inside it loses 10% or 20% of its bought GPU-hours to badput the facility never sees. The standards remain the right tool for the question they answer, and they govern the ceiling. They cannot be the design-basis metric for a workload whose return lives in the gap below that ceiling.

The practical reconciliation is a two-tier design basis. First, certify the facility to the availability class the workload's floor requires. Higher continuity can pay for an unmasked interactive endpoint, while checkpointable training can price replay; both still owe the required states and customer obligations. Second, run a separate goodput design basis that governs the silicon, storage and software redundancy the standards never touch.

This also extends the redundancy primer in Chapter 0.5. N, N+1, 2N and the distributed-redundant topologies are still the vocabulary, but you now apply them in two places, the facility power and cooling chain and the compute resilience stack, and the goodput model decides which application justifies the spend. Chapter 12.1 details the standards landscape and topology selection, and Chapter 12.4 covers the SLA that contracts goodput rather than availability.

Put the mechanisms on one economic curve, with incremental lifecycle cost on the x-axis and goodput or realized service value on the y-axis. For a training fleet whose largest avoidable losses are addressed first, the curve starts steep and flattens as those losses disappear. Event frequency, recovery time and remaining shared causes set the actual slope and step order.

The first dollars above the required facility-state floor go to the largest avoidable loss per dollar. Multi-tier checkpointing attacks wasted progress and slow recovery: Google’s 17 June 2025 report gives a 6.59% ML Goodput increase on one 35K-chip TPU v5p workload. Separately, it says restore takes under one minute across clusters of thousands of nodes, against up to 30 minutes from object storage alone. Both claims are vendor-reported, and restore is not an end-to-end fleet MTTR guarantee. Spare pools and health checks rise when capacity waits dominate. Cooling continuity, transient ride-through and the second facility path compete on the events they prevent. Price each rung against the loss still left after the previous purchase, or pay twice to recover the same GPU-hours.

Serving fleets bend the curve differently. Request retry plus replica capacity in another zone can hide the loss of one facility. Capacity in another region can hide a regional event. A fleet with that spread can recover from a site outage at the fleet layer, so 2N is not automatic. Zones in one metro still share regional failure domains: the March 2026 drone strikes on AWS's Gulf facilities showed that multi-AZ design does not survive the loss of a region (Chapter 11.2; regional recovery is in Chapter 12.3). A single-site serving fleet has none of that, and for it the facility rungs climb back to the top. The right budget is where marginal goodput per dollar equalizes across the facility, the compute stack and the fleet, the point Chapter 12.5 locates for your failure environment.

Choose recovery mechanisms that preserve the workload’s required state and complete inside its deadline, then price them against the same facility events. Reserve and storage spending pays off through recovered work, and underfunding the required physical states leaves a loss the software cannot mask. Carry the event-to-job mapping from Chapter 8.9 into the model, and carry the resulting measurements to the useful-output accounting in Chapter 14.1.