Operational KPIs, Goodput & the Reliability Economics of AI Factories

Parts 1 through 13 of this guide build the machine. This part operates it, and operating it is where most of the lifetime money is won or lost, because the asset depreciates whether or not it is producing. Twenty years of enterprise data-center practice taught operators to manage the facility against availability: nines of uptime, hours of downtime per year, Tier classes. That habit measures continuity at a declared service boundary. It does not measure how much paid compute reaches the customer. An AI factory is a depreciating compute asset, and its return depends on how many accelerator-hours land on the revenue-bearing critical path. A hall can report high facility availability and still throw away a tenth or more of its compute to stragglers, restarts, silent corruption and idle hot spares. The income statement will not care that the UPS never dropped.

This chapter is the framework for Part 14. It pairs contracted availability with goodput, defines the operating metrics (ETTR, ML goodput, MFU/MBU) at engineering depth, and describes the AI failure environment that makes day-2 different from traditional IT. It then sets out the blast-radius problem, where one component fault stalls a whole cluster, and builds the operations scorecard that the rest of Part 14 instruments. Chapter 12.2 covers the service-boundary and continuity tradeoffs. This chapter owns the useful-output ledger: how you measure it, what it costs to move it, and what the scorecard looks like once the building is live.

The first day-2 decision is which number sits at the top of your operations dashboard, because it silently re-weights every reliability investment below it. Availability asks what fraction of the contracted time or requests met the agreed service conditions. Its boundary varies: facility-power availability counts energized time, reachability counts reachable IT, and correctness-aware service availability can also reject corrupt responses. Each boundary needs its own event log, and Uptime Tier topology predicts none of their percentages. Goodput is useful work delivered within the declared quality and service limits. As a time fraction it asks a harder question: what share of the accelerator-hours you paid for actually advanced a job on the critical path? It is not MFU, which measures how much of peak FLOPs a running job realizes. Availability and goodput diverge sharply for AI, and Part 14 instruments that divergence.

Consider a synchronous pre-training run at the scale of Meta's Llama 3 405B. Suppose the facility never loses power during such a run. That shows continuity over one window, not an annual five-nines result. Even so, the job stops roughly every three hours: Meta's snapshot ran on up to 16,384 H100s and logged 419 unexpected interruptions (of 466 in total) over 54 days, from GPU, network, software and other causes. Each interruption idles every accelerator in the job until it restarts from its last checkpoint. The 'available' facility is bleeding goodput through a mechanism the availability metric cannot see, because the failure domain is the job, not the component. So an AI-cluster reliability model has to cover both facility states and workload recovery. For a given failure environment, checkpointing, hot spares and recovery automation may buy more goodput than an extra facility path would. Goodput alone does not choose N, N+1, 2N or a Tier topology; Chapter 12.2 prices that trade.

'Goodput' is not one interchangeable number. ETTR, application goodput and MFU expose different losses, and conflating them can make a wasteful fleet look efficient. Read each numerator against its own denominator. Multiply ratios only after proving they are conditional factors on one event log and one time boundary, or the same lost hour gets counted twice.

ETTR (Effective Training Time Ratio) is productive runtime divided by the job run's available wallclock time, meaning time scheduled or eligible to be scheduled, including the eligible queue. It ranges from 0 to 1 and absorbs queueing delay, restart overhead and recomputation of lost progress (Meta, Revisiting Reliability in Large-Scale ML Research Clusters, 2024–25). ETTR is model-agnostic: it does not care what fraction of a GPU's FLOPs a kernel extracts, which is why it is the right SLA target between an operator and a training customer. Per Meta's revised paper of February 2025 (arXiv 2410.21680 v2), its largest, highest-priority jobs on the shared RSC research clusters have an estimated ETTR of 0.85–0.9. Runs of 2,048–4,096 GPUs lasting 2+ days on RSC-1 average an estimated 0.9. Meta derived both figures from observed job runs. Both estimates assume the Daly-Young optimum for checkpoint spacing, and 5 minutes each for restart and checkpoint writes. Another fleet must derive its own checkpoint cadence and restart targets from its own interruption distribution, checkpoint cost, recovery path and ETTR objective.

ML goodput (Google's formulation) is the fraction of application elapsed time spent on preserved training progress. All other time is 'badput': program startup, data-loading stalls, checkpoint writes that do not overlap compute, failed steps, progress wasted since the last checkpoint, and scheduling gaps. Google's ML Goodput library measures an individual application's preserved training progress. Its elapsed clock starts with the application, so it does not automatically include an earlier eligible queue. The operations scorecard rolls up compatible raw totals, weighted by allocated accelerator count when job widths differ. Google's Cloud TPU goodput documentation (checked October 2026) defines this application boundary. Take 90% versus 96% as two illustrative operating points, not measured provider results. Holding workload, peak-normalized productivity and the installed 1 GW of IT capacity fixed, six points equal 60 MW-equivalent of useful-output capacity. The plant still draws its workload-dependent power, so this is neither 60 MW saved electrically nor automatically sellable capacity. Chapter 1.3 maps useful output to demand and revenue.

MFU (Model FLOPs Utilization) and MBU (Model Bandwidth Utilization) are the innermost layer. They measure efficiency while a job is running, independent of failures. MFU is the fraction of the hardware's peak FLOPs the model actually realizes (compute-bound training). Meta’s Llama 3 405B Table 4 reports 41% BF16 MFU on 16,384 H100s at 8K sequence in July 2024. Published GB300 NVL72 measurements exist for named benchmark configurations, but there is no comparable fleet-wide operating band. Lambda's November 2025 white paper measures Llama 3.1 405B training at 52–56% MFU, using 8K sequences on 16 to 128 GPUs, after Lambda tuned the stack. At 128 GPUs that is 52.7%, against 28.6% for its TorchTitan baseline. MBU is the analogous ratio for memory bandwidth, the binding constraint on autoregressive inference decode. A job can have a 0.95 ETTR and a 35% MFU: almost never interrupted, yet using about a third of the peak FLOPs it pays for. Report the three metrics together. Useful model FLOPs = productive time × MFU during that time × declared peak FLOPs/s, with time in seconds and the same hardware, precision and sparsity convention throughout. Use MBU for bytes against peak bandwidth, not as a substitute FLOPs factor. A running-job average that already includes stalls cannot be multiplied by ETTR without reconciling its window. → metric definitions in Chapter 0.3.

The table uses different denominators, and comparing two facilities on different rows is the recurring reporting error. A neocloud quoting '99.99% uptime' is on the bottom row. A customer losing a tenth or more of a training run to stragglers and restarts lives on the top row. If a contract requires useful-output acceptance, gate production handoff on sustained NCCL/collective performance, low error rates and the agreed soak window, as well as service continuity. A GPU failure costs the job its measured interruption, recovery and discarded progress. Sustained collective bandwidth, an error budget and a soak window become enforceable when the buyer names the workload, test load and pass threshold. They are not a universal 2026 SLA. NVIDIA's open-source Cluster Readiness Engine (v0.7.0, announced October 2026) is one example of an open acceptance tool. It flags failing nodes but never quarantines them. Quarantine stays with the operator, so the contract must state who does it and who approves a node’s return to service. → acceptance and IST in Chapter 13.6; service boundaries in Chapter 12.2.

Trace. Available allocation = 100 GPUs × 10 h = 1,000 GPU-h. Reconcile 100 + 50 + 100 + 50 + 700 = 1,000 GPU-h; discarded work is charged once. ETTR = 700/1,000 = 70%. Useful output = 700 GPU-h × 0.40 = 280 peak-equivalent GPU-h. Recovery gives 750 × 0.40 = 300; kernel tuning gives 700 × 0.42 = 294. Select recovery at the stated equal cost: 20/C versus 14/C peak-equivalent GPU-h recovered per budget unit. These are output units, not electrical energy or cash.

Flip. At equal cost, kernels win above MFU = 300/700 = 3/7 (about 42.86%); recovery must reclaim more than 700 × (0.42 − 0.40)/0.40 = 35 GPU-h to beat the stated kernel change. At unequal costs C_R and C_K, recovery wins only when C_R/C_K < 20/14 = 10/7 (about 1.43), assuming the same benefit horizon. The ETTR side uses Meta’s scheduled-or-eligible clock and the goodput side Google’s application clock, each read with conditional MFU and weighted by allocated GPUs when job widths vary. Availability keeps its own service denominator; busy time cannot replace either account.

Traditional enterprise IT runs a fleet of loosely coupled, independently failing servers. One box dies, a load balancer routes around it, and the blast radius is one request. The AI factory inverts every assumption behind that model, so the day-2 reliability program has to be rebuilt from the inverted premises.

The components fail far more often. In Meta's research clusters, observed mean time to failure (MTTF) falls by roughly two orders of magnitude with job size, from about 47.7 days for an 8-GPU job to about 7.9 hours for a 1,024-GPU job. Meta's rate-based formula, 1/(nodes × failure rate per node), projects about 1.8 hours for a 16,384-GPU job at RSC-1's 6.50 failures per 1,000 node-days (Chapter 14.3 derives it). Extrapolating either figure to a larger job needs explicit assumptions about job membership, independent versus common-mode failures, software and detection. The takeaway still holds: failure rate rises with GPU count, so the largest jobs spend a meaningful fraction of their life recovering rather than computing. Recovery speed and failure rate are therefore both first-order goodput levers at frontier scale. Meta's revised model says a ~12,000-GPU run on RSC-1 hits ETTR ≈0.9 only if checkpoint writes take about 10 seconds, or if failures fall from 6.50 to roughly 1 per 1,000 node-days. → operational recovery in Chapter 14.4.

The failures are not all loud. The AI fleet has three failure classes, and the dangerous one is invisible. Hard failures (a GPU falls off the bus, a link drops) announce themselves and trigger a restart. Transient failures (a correctable ECC storm, a thermal throttle) degrade goodput without stopping the job. Silent data corruption (SDC) is the third and worst: a marginal device computes a wrong result with no error flag, poisoning gradients or activations. Meta’s 22 July 2025 discussion reports about one SDC fault per thousand devices, with no exposure period from which to infer machine prevalence or a run clock. Separately, its July 2024 Llama 3 report records six SDC-attributed interruptions during a 54-day campaign on up to 16,384 H100s. Those observations do not establish another fleet's rate or recurrence interval. Soft-error susceptibility rises as devices shrink and operating voltages fall, so the problem is getting harder. The number that matters is your own fleet's detected and silent error rate. Detection is now a standing fleet program: Meta’s March 2022 CPU-fleet report attributes about 2.5 billion test seeds per month to Ripple, with Fleetscanner counted separately. That test volume is not GPU-fleet coverage. → SDC mechanisms and detection in Chapter 14.3.

The defining structural feature of the AI failure environment is that the failure domain is not the failed component. In a tightly coupled synchronous job, one GPU stalling on an all-reduce stalls the entire collective, and the whole job moves at the speed of its slowest straggler. The scale-up fabric amplifies this: a failed NVSwitch tray degrades bandwidth for all 72 GPUs in an NVL72 domain. Take an illustrative tensor-parallel group that needs all 64 GPUs, each 99.9% available over the same window with statistically independent failures. Under the model in Chapter 12.5, it has roughly 94% all-device availability. Correlated failures and tolerated device loss need a different model. The blast radius of a single fault is the size of the coupling domain you chose at design time, so domain sizing is a reliability decision as much as a performance one.

The cascade goes wider than the job. Meta estimated that 16% of the goodput lost to hardware failures on its RSC-1 research cluster came from second-order preemptions: small jobs evicted to free resources for a large job's restart. One large-job failure ripples into idle time across unrelated workloads sharing the cluster. The old NVL72-versus-NVL36x2 fork is also closed. Analyst Ming-Chi Kuo reported in October 2024 that NVIDIA had halted NVL36x2, which split one 72-GPU NVLink domain across two racks, and it never ramped. One GB300 or Vera Rubin NVL72 rack holds the entire 72-GPU domain, and so its whole blast radius. Where scale-up domain size does change, the trade is real: a larger domain lifts the tensor-/expert-parallel ceiling and the achievable MFU, but it widens the blast radius of a single tray or cold-plate fault. The right answer depends on whether you manage against MFU or against goodput. → scale-up domain sizing in Chapter 8.2; lemon-node ejection in Chapter 14.4.

Day-2 reliability is an investment with a measurable return, denominated in goodput. For scale, a 1 GW AI factory's all-in TCO is roughly $8.5B/yr (Epoch AI), and at a short, frontier-economic depreciation life the figure runs higher. Allocating that fixed annualized cost in proportion to the stipulated six-point useful-time difference gives 0.06 × $8.5B, or about $500M/yr of cost-equivalent output. That assumes the same 1 GW IT configuration, workload and cost base, and it is a cost allocation, not recoverable cash. Chapter 1.3/1.8 test paid demand, marginal contribution, implementation cost and the cash-flow horizon before valuing an intervention. Underwrite each reliability intervention (checkpointing, spares, health checks, SDC scanning and node ejection) against your fleet's measured badput reduction and its implementation cost.

The economics also explain the hidden-tax structure that ClusterMAX exposed. Holding GPU sticker price constant, SemiAnalysis's April 2026 ClusterMAX model puts a gold-tier neocloud's total cost below a silver-tier provider's by 5–15% on its large-training assumptions, and a hyperscaler's 36-month total swelled to 1.10x a gold-tier neocloud's, a 10% hidden tax. Modeled goodput expense accounts for part of that gap: the hyperscaler profile carries 10.53% and gold 6.14%, each figure including that profile's assumed fault-tolerance overhead. Across all three profiles the expense spans about 6–21%, with silver at the top. SemiAnalysis attributes the gap primarily to support and setup costs (EFA performance tuning), none of it on the rate card. The cheapest GPU-hour on paper can cost the most once delivered, because reliability is priced in goodput, not in the quoted $/GPU-hr. For fault-tolerant workloads (single-node inference) the gap collapses toward zero. The value of a goodput point depends on the workload, and so does the reliability spend that is rational to chase it. → unit economics in Chapter 1.8.

An operations program is only as honest as its scorecard, and the scorecard is only as honest as its denominators. The day-2 KPI set spans three layers: the workload, the fleet and the facility. All three belong on the scorecard with named denominators, not collapsed into a single flattering 'uptime' figure. The workload layer is goodput and ETTR (with MFU/MBU underneath). The fleet layer is failure rate per node-day, MTTR decomposition, lemon-node rate and SDC detection rate. The facility layer is availability, PUE/WUE and power/thermal headroom. Each layer has a different audience: the customer cares about goodput, the reliability engineer about the fleet layer, and the facility team and lenders about the bottom layer.

The recurring day-2 failure is a scorecard that reports the bottom layer, because traditional DCIM measures it well, and stays silent on the top layer, because measuring it needs IT/facility telemetry correlation the legacy stack never had to do. A facility can show a perfect availability scorecard while its customers lose a tenth or more of their compute to badput the operator never instrumented. The central task of the observability chapter that follows is to close that gap by wiring facility telemetry to workload telemetry, so that a thermal event, a power transient and a training stall can be traced to a single root cause. → telemetry and IT/facility correlation in Chapter 14.2.

The same day-2 mistakes recur, each one a consequence of applying the facility-availability worldview to a workload that obeys goodput economics. Three are worth naming:

• Choosing topology from checkpointability. Treating checkpoint-and-resume as proof that 2N is waste, or treating a serving label as proof that Tier-IV-class power is required. Before spending the marginal dollar, price the facility path from its maintenance and fault states, post-event capacity and common modes, and the workload path from its recovery time, then check both against the contract. → Chapter 12.5.
• Reporting availability and calling it goodput. A green uptime dashboard sits over a fleet losing a tenth or more of its compute to stragglers, restarts and badput. The denominator is wallclock of the facility, not accelerator-hours on the critical path, and the gap stays invisible until a customer measures their own ETTR and disputes the SLA.
• No standing SDC program. Treating silent corruption as an anomaly that conventional error alarms will catch. Size screening capacity and workload checks against your own measured corruption window and the cost of losing trusted progress. Without continuous scanning and checkpoint validation, the multi-day rollbacks of poisoned runs are the ones the operator cannot see.

Choose the next reliability intervention by useful output recovered per dollar, after the service-availability and physical-continuity gates pass. Keep occupancy, execution efficiency and retained progress as separate totals: an idle spare costs capacity, a busy GPU can repeat discarded work, and an energized hall can still miss the customer's deadline.

Serving has its own ledger. Record offered requests, accepted requests, failures, cancellations, completed outputs and elapsed observation time, with the arrival mix and admission policy. Count as useful only output that meets the declared correctness and latency SLO, and include failed and timed-out requests in the service denominator rather than reporting latency only for survivors. Tokens/s needs its model, output-length and quality boundary, while GPU busy time reports resource activity. Utilization can rise while queueing violates latency. Do not multiply serving SLO attainment by an ETTR/MFU ratio unless their numerators and conditional windows actually compose. Chapter 10.6 instruments these accounts, and Chapter 12.2 defines the service boundary.