Operational KPIs, Goodput & the Reliability Economics of AI Factories

Parts 1 through 13 of this guide build the machine. This part operates it — and operating it is where most of the lifetime money is won or lost, because the asset depreciates whether or not it is producing. The reflex inherited from twenty years of enterprise data-center operations is to manage the facility against availability: nines of uptime, hours of downtime per year, Tier classes. That reflex is not wrong so much as it measures the wrong thing. An AI factory is a depreciating compute asset whose return is set by how many accelerator-hours land on the revenue-bearing critical path. A hall can report high facility availability and still throw away a quarter of its compute to stragglers, restarts, silent corruption, and idle hot spares — and the income statement will not care that the UPS never dropped.

This chapter is the framework for Part 14: it reframes the operating objective from availability to goodput, defines the goodput stack (ETTR, ML goodput, MFU/MBU) at engineering depth, characterizes the AI failure environment that makes day-2 different from traditional IT, sets out the blast-radius problem that turns one component fault into a cluster-wide stall, and assembles the operations scorecard that the rest of Part 14 instruments. The deep treatment of why goodput beats availability as a design target lives in Chapter 12.2; this chapter is the operational counterpart — how you measure it, what it costs to move it, and what the scorecard looks like once the building is live.

The first day-2 decision is which number sits at the top of your operations dashboard, because it silently re-weights every reliability investment below it. Availability asks: what fraction of wallclock time is the facility energized and the IT reachable? It is a facility-centric, binary-per-component measure derived from observed events and repair times; Uptime Tier topology does not predict it. Goodput asks a different and harder question: what fraction of the accelerator-hours you paid for actually advanced a job on the critical path? The two diverge sharply for AI, and that divergence is what Part 14 instruments.

Consider a synchronous pre-training run on 16,384 GPUs. The facility never loses power; by the availability definition it is at five-nines. But a single GPU fails roughly every three hours at that scale (Meta's Llama 3 405B snapshot), and each failure idles all 16,384 accelerators until the job restarts from its last checkpoint. The 'available' facility is hemorrhaging goodput through a mechanism the availability metric cannot see, because the failure domain is the job, not the component. This is why the AI-cluster reliability model must include both facility states and workload recovery. Checkpointing, hot spares, and recovery automation may beat an additional facility path on marginal goodput for a named failure environment, but checkpointability does not prove that 2N is misallocated. Compare the defined maintenance and fault states, interruption, post-event capacity, independence, common modes, recovery, and contract before choosing the next investment.

'Goodput' is not one number; it is a stack of nested ratios, each measuring a different kind of waste, and conflating them is the most common source of dishonest operating reporting. Read from the outside in, each layer multiplies the one below it, and the product is what the dollars actually buy.

ETTR (Effective Training Time Ratio) is the outermost layer and the cleanest measure of cluster reliability: the ratio of productive runtime to the available wallclock time of a job run, ranging from 0 to 1, accounting for queueing delay, restart overhead, and re-computation of lost progress (Meta, Revisiting Reliability in Large-Scale ML Research Clusters, 2024–25). ETTR is model-agnostic — it does not care what fraction of a GPU's FLOPs a kernel extracts — which is exactly why it is the right SLA target between an operator and a training customer. Meta's largest reported jobs (>1024 GPUs) sustained average ETTR above 0.9 with one-hour checkpoint intervals on its named shared research clusters. A different fleet must derive cadence and restart targets from its own effective interruption distribution, checkpoint cost, recovery path, and ETTR objective; accelerator count alone does not produce a portable two-minute target.

ML goodput (Google's formulation) is the aggregate productive work completed per unit time across the cluster, with all non-productive time accounted as 'badput': program startup, data-loading stalls, checkpoint writes that do not overlap compute, failed steps, wasted progress since the last checkpoint, and scheduling gaps. Where ETTR is per-job, ML goodput is the fleet-level rollup, and it is the number the operations scorecard reports. Use 90% versus 96% only as a declared sensitivity scenario; the cited provider-tier/marketed examples do not establish an industry average or portable best-in-class boundary. At a 1 GW facility, that six-point scenario difference is 60 MW-equivalent of useful-work fraction, subject to the model's declared facility/IT boundary.

MFU (Model FLOPs Utilization) and MBU (Model Bandwidth Utilization) are the innermost layer — they measure efficiency while a job is running, independent of failures. MFU is the fraction of the hardware's peak FLOPs the model actually realizes (compute-bound training; commonly 30–50%, >50% considered strong on Hopper-generation reported runs; no comparable public band has been published for NVL72-class racks). MBU is the analogous ratio for memory bandwidth (the binding constraint on autoregressive inference decode). A job can have a 0.95 ETTR and a 35% MFU — perfectly reliable, half-idle on the math — which is why the three metrics must be reported together. Realized output = ETTR x MFU/MBU x peak, and an operator who quotes only one of the three is hiding the other two. → metric definitions in Chapter 0.3.

The table is a hierarchy of denominators, and comparing two facilities on different rows is the recurring reporting error. A neocloud quoting '99.99% uptime' is on the bottom row; a customer who actually loses 20% of their training run to stragglers and restarts is living on the top row. The contractual maturity of 2026 is that goodput, not uptime, is becoming the acceptance and SLA basis — operators gate production handoff on sustained NCCL/collective performance and low error rates over multi-day soaks, not power-on uptime, because each failed GPU can cost 15–20% performance and a mid-run failure destroys training economics. The industry is converging, slowly, on a contractual definition of 'goodput acceptance' (sustained collective bandwidth, an error budget, a soak window) that buyers can hold integrators to. → acceptance and IST in Chapter 13.6; goodput-vs-availability theory in Chapter 12.2.

Traditional enterprise IT operates a fleet of loosely-coupled, independently-failing servers: one box dies, a load balancer routes around it, and the blast radius is one request. The AI factory inverts every assumption behind that model, and the day-2 reliability program has to be rebuilt from the inverted premises.

The components fail far more often. Meta's research clusters log 6.50 failures per thousand node-days (RSC-1) and 2.34 per thousand node-days (RSC-2) — roughly 5x10⁻³ failures per node-day. In Meta's named research-cluster population, an 8-GPU job had about 47.7 days MTTF and a 1,024-GPU job about 7.9 hours; the Llama 3 405B run separately recorded 419 unplanned interruptions over 54 days on 16,384 H100s. These observations use different event populations and cannot be extended into a minute-level 131,072-GPU forecast without explicit job-membership, independence/common-mode, software, and detection assumptions. The takeaway holds regardless: failure rate rises with GPU count, so the largest jobs spend a meaningful fraction of their life recovering rather than computing. This is the arithmetic that makes recovery speed, not component MTBF, the dominant goodput lever at frontier scale. → fleet failure-rate data in Chapter 14.3; operational recovery in Chapter 14.4.

The failures are not all loud. The AI fleet has three failure classes, and the dangerous one is invisible. Hard failures (a GPU falls off the bus, a link drops) announce themselves and trigger a restart. Transient failures (a correctable ECC storm, a thermal throttle) degrade goodput without stopping the job. Silent data corruption (SDC) is the third and worst: a marginal device computes a wrong result with no error flag, quietly poisoning gradients or activations. Meta finds roughly 1 in 1,000 machines affected by SDC, and for large training runs an SDC event is expected every one to two weeks; Google estimates an SDC event every week or two during Gemini training. Soft-error rates have worsened with process shrinks — from roughly one failure per year at 65 nm to one per ~1.5 hours at 16 nm — so this is structurally getting harder. Detection is now a standing fleet program (Meta's Fleetscanner and Ripple run ~2.5 billion test seeds per month). → SDC mechanisms and detection in Chapter 14.3.

The defining structural feature of the AI failure environment is that the failure domain is not the failed component. In a tightly-coupled synchronous job, one GPU stalling on an all-reduce stalls the entire collective, and the whole job moves at the speed of its slowest straggler. The scale-up fabric amplifies this further: a failed NVSwitch tray degrades bandwidth for all 72 GPUs in an NVL72 domain, and a tensor-parallel group of 64 GPUs with 0.1% of GPUs failed drops to roughly 94% effective availability for the group. The blast radius of a single fault is the size of the coupling domain you chose at design time, so domain sizing is a reliability decision as much as a performance one.

The cascade goes wider than the job. Meta found that 16% of total failure-related goodput loss came from secondary preemptions — small jobs getting evicted to free resources for a large job's restart. One large-job failure ripples into idle time across unrelated workloads sharing the cluster. This is the day-2 reason operators weigh the NVL72-vs-NVL36x2 fork. NVIDIA forms the two-rack configuration as one 72-GPU NVLink domain from two directly connected 36-GPU L1 domains, so a single-rack GB300 NVL72 at a 135 kW rack TDP concentrates the same domain into one rack rather than doubling its blast radius. VR200 NVL72 raises the per-rack figure again to ~190 kW (Max Q) / ~230 kW (Max P), an analyst estimate (Kuo / Hashrate Index, 2026) rather than an NVIDIA datasheet figure. Where scale-up domain size does change, the trade is real: a larger domain lifts the tensor-/expert-parallel ceiling and the achievable MFU and widens the blast radius of a single tray or cold-plate fault, and the right answer depends on whether you manage against MFU or against goodput. → scale-up domain sizing in Chapter 8.2; lemon-node ejection in Chapter 14.4.

Day-2 reliability is an investment with a measurable return, and the return is denominated in goodput. The canonical economic anchor: a 1 GW AI factory carries roughly $8.5B/yr in all-in TCO (Epoch AI), and at the short, frontier-economic depreciation life the figure runs higher still. In a declared 96%-versus-90% sensitivity, six percentage points of an $8.5B/yr asset is on the order of $500M/yr; that is scenario arithmetic, not a measured industry gap. Underwrite each reliability intervention—checkpointing, spares, health checks, SDC scanning and node ejection—against the named fleet's measured badput reduction and implementation cost.

The economics also explain the hidden-tax structure that ClusterMAX exposed: holding GPU sticker price constant, a gold-tier neocloud's total cost runs lower than a silver-tier provider's by 5–15% on large training workloads, and a hyperscaler's 36-month total swelled to 1.10x a gold-tier neocloud's — a 10% hidden tax. Goodput loss is part of it (10.53% vs 6.14% of rented cluster-time re-bought to cover interruptions), but SemiAnalysis attributes the gap primarily to support and setup costs (EFA performance tuning) — none of it on the rate card. The cheapest GPU-hour on paper can cost the most once delivered, because reliability is priced in goodput, not in the quoted $/GPU-hr. For fault-tolerant workloads (single-node inference), the gap collapses toward zero — which is why the metric choice at the top of this chapter matters: the value of a goodput point is workload-dependent, and so is the reliability spend that is rational to chase it. → unit economics in Chapter 1.8.

An operations program is only as honest as its scorecard, and the scorecard is only as honest as its denominators. The day-2 KPI set spans three layers — the workload, the fleet, and the facility — and all three belong on the scorecard with named denominators rather than collapsed into a single flattering 'uptime' figure. The workload layer is goodput and ETTR (and MFU/MBU underneath); the fleet layer is failure rate per node-day, MTTR decomposition, lemon-node rate, and SDC detection rate; the facility layer is availability, PUE/WUE, and power/thermal headroom. Each layer has a different audience: the customer cares about goodput, the reliability engineer cares about the fleet layer, and the facility team and lenders care about the bottom layer.

The recurring day-2 failure is a scorecard that reports the bottom layer (because it is the one traditional DCIM measures well) and is silent on the top layer (because it requires IT/facility telemetry correlation the legacy stack never had to do). A facility can show a perfect availability scorecard while its customers are quietly losing a fifth of their compute to badput the operator never instrumented. Closing that gap — wiring the facility telemetry to the workload telemetry so that a thermal event, a power transient, and a training stall can be correlated to a single root cause — is the central task of the observability chapter that follows. → telemetry and IT/facility correlation in Chapter 14.2.

The same day-2 mistakes recur, each one a consequence of reaching for the facility-availability worldview when the workload obeys goodput economics. Three are worth naming:

• Choosing topology from checkpointability. Treating checkpoint-and-resume as proof that 2N is waste, or treating a serving label as proof that Tier-IV-class power is required. Model the named facility states, workload and fleet recovery, post-event capacity, common modes, and contract before allocating the marginal dollar. → Chapter 12.5.
• Reporting availability and calling it goodput. A green uptime dashboard over a fleet quietly losing 15–20% of its compute to stragglers, restarts, and badput. The denominator is wallclock-of-the-facility, not accelerator-hours-on-the-critical-path, and the gap is invisible until a customer measures their own ETTR and disputes the SLA.
• No standing SDC program. Treating silent corruption as a rare anomaly rather than a continuous fleet condition (~1 in 1,000 machines, an event every 1–2 weeks at scale). Without continuous scanning and checkpoint validation, multi-day rollbacks of poisoned runs are the ones the operator is structurally blind to.