SLAs, Goodput Contracts & Availability Commitments

An SLA is a reliability model with money attached. Everything in Part 12 up to this point (the goodput-versus-availability rethink in Chapter 12.2, the redundancy primer in Chapter 0.5, the facility tier standards in Chapter 12.1) lives in the engineering domain, where being wrong costs an outage. The SLA moves all of it into the commercial domain, where being wrong costs a service credit, a churned tenant or a take-or-pay dispute. This chapter covers the translation: how a failure environment becomes a promise, how the promise is measured and attributed, and how the penalty structure disciplines the provider without bankrupting them in one bad month.

The recurring fork is the one Part 12 has been building toward. Availability asks whether the facility was energized, cooled and reachable. Goodput asks whether the customer's job made effective forward progress. For a web service the two questions have nearly the same answer. For an AI workload they diverge violently. The facility meter can read 100% available while one GPU with silent data corruption poisons every synchronous step. Once the fault is found, hours of work that looked productive are discarded and count as zero retained progress. Or a ninety-second cooling interruption can leave a clean checkpoint intact and still cost those service seconds, plus detection, restore and replay time. An SLA that measures the wrong one is worse than none: it gives the customer a number that does not track their pain, and the provider an exposure that does not track their control.

The legacy data-center SLA promises availability: a Monthly Uptime Percentage measured at the facility or the instance, with a service-credit ladder if it falls short. This is the world of the Uptime tiers (Tier III concurrently maintainable, Tier IV fault-tolerant), where each "nine" in the negotiated SLA is a downtime budget the operator commits not to exceed. Tier itself certifies topology, not an availability percentage (see Chapter 12.1). It is a clean, well-understood, auditable promise, and a useful boundary metric for a facility or instance service. For a managed inference endpoint, facility uptime and request success can diverge either way: an unreachable endpoint loses requests and revenue, while surviving replicas can mask facility downtime. Contract correct, on-time request completion within the agreed load as well (Chapter 10.11).

The training SLA is a different animal, because the customer is buying effective compute, not uptime. For this contract, Provider Goodput = Scheduling Goodput × Runtime Goodput: retained forward-progress time divided by contracted wall-clock accelerator time. Provider-attributable badput includes unavailable resources, failed steps that get rolled back, checkpoint and restore, stragglers, rescheduling, and work discarded because it ran on a silently corrupting device. Program Goodput (how efficiently the code uses the hardware, close to MFU) stays tenant-owned unless the provider accepts a reference-program target. The 90% versus 96% comparison used in this Part is a stipulated sensitivity, not an industry average or a best-in-class target. Value a measured badput reduction by the interruption and recovery time it actually removes, not by the availability nines it appears to add.

The two columns reward different capital. An availability commitment is defended with redundancy (2N power, N+1 cooling, dual fabric), and its cost lands in the MEP-scope construction budget. 2N duplicates the full power path, so price the BOM and TCO premium for your topology along with the operating posture. A goodput contract is defended with operational reliability: passive health-checks every few seconds, automatic node drain when a node degrades against a golden reference, a hot-spare pool, and multi-tier checkpointing.

None of those settings port between fleets, and the numbers are where the money is. The drain threshold follows from the reference workload, the cost of a false positive and the measured degradation distribution. The spare fraction follows from failure arrivals, repair and logistics time, topology-compatible placement and the depletion risk you accept. Swap and restore times follow from your SKU, fabric and checkpoint tier. Derive each from your own acceptance data, and keep the controls apart: a performance-degradation trigger bounds throughput loss, but it does not establish correctness or catch every SDC.

The mistake operators make is buying the first kind of insurance for a workload that needs the second. A training tenant values facility continuity, but also the work since its last checkpoint (up to one full checkpoint interval) that any interruption forces it to replay, plus detection and restart time. Compare the contracted interruption budget with complete restart time plus the age of the lost progress. → the goodput-vs-availability tradeoff curve in Chapter 12.2, quantified by the model in Chapter 12.5.

A commitment without a penalty is marketing. The penalty in a standard SLA is the service credit: a percentage of the monthly bill, refunded as credit (almost never cash) when the measured metric falls below a threshold. The ladder is a step function: miss the target band and you owe a credit; miss it badly and you owe more. The reference shape is Amazon EC2’s Region-Level SLA (May 25, 2022), a three-rung ladder: a 10% credit below 99.99% but at least 99.0% monthly uptime, 30% below 99.0% but at least 95.0%, and 100% below 95.0%, all on the policy’s affected-fee basis. Those are EC2 uptime terms. A GPU agreement needs its own node, rack or goodput boundary before those rungs can price a shortfall. Exhibit T below gives an assumed training agreement with exact inequalities and an event ledger.

The design decisions inside the ladder are where the money and the disputes live. Stepped vs linear: a stepped ladder is simple to administer but creates cliff incentives. A provider one minute on the wrong side of a threshold owes the same as one an hour past it, which can perversely make them stop fighting an outage once a rung is lost. A linear or finely stepped ladder tracks pain more honestly, at the cost of administrative complexity. Capped vs uncapped: a cap limits how much of the defined monthly fee is exposed. Without one, a bad month can create a liability larger than that fee, so price collateral, insurance availability and the provider's ability to pay before agreeing to an uncapped or multiplied remedy. Credits vs termination: the customer's real remedy for chronic underperformance is not the credit, which rarely exceeds a month's fee, but a termination-for-repeated-breach right (for example, three failed months in any six, the trigger Exhibit T uses). That clause is what actually disciplines a provider, and both sides should negotiate it hardest.

Choose the fee basis and overlap rule alongside the ladder. Two obligations measured on the same reservation can breach in the same event, so the agreement must say whether credits accumulate or the greater applicable remedy settles the overlap. Keep the raw event evidence even when the settlement pays only once.

An availability metric is observable at a boundary. The meter trips or the endpoint stops answering, and the parties can reconcile probe results, clocks, measurement intervals and exclusions. A goodput metric is the opposite: it is measured inside the customer's job, where provider and tenant share the failure surface, and the contract's commercial value turns on attribution, deciding for every minute of badput whose fault it was. This is the hardest part of an AI SLA, and the part most contracts leave dangerously vague.

The contractual measurement basis is goodput accounting: instrument the job to produce a defensible ledger of total GPU-time, productive GPU-time and each class of badput. The badput taxonomy is the heart of it, and each class has a natural owner. Hardware badput (a failed GPU, an HBM error, an SDC event auto-drained by the health-checker) is the provider's, because it is their silicon and their fleet management. Recovery badput (checkpoint/restore latency, node-swap time, re-scheduling delay) is shared, and the split depends on whether the provider or the tenant supplied the checkpointing stack. Workload badput (a tenant's inefficient parallelism, a bad hyperparameter that diverges, a job that simply ran slow) is the tenant's, and the provider must be able to fence it out or they are underwriting the customer's ML engineering. The contract must name these classes, name the attribution method (whose telemetry, and what arbitration if the logs disagree), and name the baseline.

Add one more class: grid-directed curtailment. Texas SB 6 (2025) requires loads of 75 MW or more that interconnect to ERCOT from January 2026 to accept curtailment in a firm load shed. It also lets ERCOT order a large load that has on-site backup to curtail or switch to that backup in an emergency. Decide in the contract whether each case is force majeure, the provider's responsibility or a backup-runtime obligation. Also decide how a job that rides through a transfer to backup is counted. → SB 6 in Chapter 3.2; ride-through in Chapter 4.10.

Exhibit T turns attribution into a payable decision: determine which reserved GPU-hours remain eligible, count overlapping failures once, then select the credit band using the exact fraction. The assumptions and the event ledger supply every input.

Contracted time is 1,024×720 = 737,280 GPU-hours. The raw tenant event overlaps provider hardware loss on [10,18), so only [60,80) is tenant-only: exclude 20×1,024 = 20,480 GPU-hours. Eligible time is 716,800 GPU-hours; retained progress is 640×1,024 = 655,360 GPU-hours. Eligible G = 655,360/716,800 = 32/35, displayed as 91.4%. Unadjusted resource productivity is 655,360/737,280, or 88.9%; it uses a different denominator.

The floor is max(92%, 0.95×96%) = 92%. Use the unrounded fraction to select the band: the month fails, and the 10% remedy is 0.10×$100,000 = $10,000. Pay max($10,000, $8,000) = $10,000, not their sum; the fee cap is not reached. Issue the future service credit and open the agreed attribution process with event IDs, timestamps, resource IDs, progress counters and device evidence. If logs conflict, use the agreed independent expert; unresolved provider evidence keeps the provisional treatment.

Flip: if evidence proves all four missing-telemetry hours were retained reference work, the numerator becomes 644×1,024 = 659,456 GPU-hours, and 644/700 = 92% exactly. The goodput credit becomes zero; the separate $8,000 availability credit remains payable. If only x of those hours are proven productive, the threshold is (640+x)/700 ≥ 0.92, hence x ≥ 4 hours. An exclusion needs its own evidence and a different denominator calculation. AWS’s published policy shows why fee scope and nonstacking must be explicit; the arithmetic here is this exhibit's own. Operational accounting stays in Chapter 14.1.

A goodput contract measured against "the industry says ~90%" is measured against nothing, and it invites a dispute the day the first shortfall is claimed. What makes it enforceable is a baseline captured at go-live: commissioning produces a quantitative fingerprint of the as-built cluster, and that fingerprint becomes the contractual reference the SLA is measured against. The acceptance gate does double duty, because it is also the moment the SLA's denominator is fixed.

The go-live fingerprint the SLA cites should include the NCCL all-reduce busbw the fabric actually achieved (measured for the specific SKU, topology, collective, message-size range, node count, software tuple and direction, against OEM qualification and the pre-production baseline). It should also include the per-node intra-node bandwidth, the fabric bit-error-rate floor, the burn-in record (the stress intensity and exposure applied, failures by mode and node-hours, pass/re-soak dispositions, and the contracted statistical stopping rule), and the measured goodput on a representative reference job.

Writing the SLA against this number ("goodput shall not fall below 95% of the commissioned baseline reference") turns an unfalsifiable marketing claim into an auditable commitment with an agreed starting point. Two further steps make it hold. State what the relative test resolves to in absolute terms, and set a floor beside it: Exhibit T's relative test resolves to 91.2% (95% of its 96% baseline), which its 92% absolute floor then overrides. Without a floor, a weak acceptance baseline legitimizes weak service for the life of the contract. The baseline also protects the provider: a tenant who later runs a pathological workload cannot claim the cluster regressed, because both parties signed a known-good reference job. The acceptance gate can also carry a deadline: ClusterMAX 3.0's template SLA lets the buyer terminate if the agreed acceptance date is missed. → acceptance scripts and pass/fail gates in Chapter 13.2.

For Exhibit T, attach the hardware SKU and count, topology, storage tier, software versions, checkpoint policy, and program and dataset hashes to the assumed 96% baseline. Re-run acceptance after any material change, using the same retained-progress definition and acceptance records. Freeze the existing floor during a regression investigation, so a slower reference cannot unilaterally lower the promise. Chapter 13.7 covers fabric acceptance, Chapter 13.8 node burn-in, and Chapter 13.9 the reference training run that fixes the goodput baseline.

The SLA is the contractual face of two things engineered elsewhere. Upstream of the customer, it is the productization of capacity: the service tiers, the pricing, the reserved-versus-on-demand structure and the onboarding commitments that turn a cluster into a sellable product. A reserved or take-or-pay commitment justifies a stronger SLA, because the customer is locked in and the provider can underwrite more. An interruptible (spot or preemptible) tier carries little or no availability promise by design, and on-demand capacity usually carries only the provider's standard instance-level SLA. The SLA tier and the commercial tier must be co-designed or they contradict each other. → customer delivery and productization in Chapter 10.9.

Downstream, for inference, the availability SLA is only the outer limit. The metric the customer experiences is the serving SLO: time-to-first-token, time-per-output-token and p99 latency under load. A 99.99% availability commitment is worthless to an inference tenant if the endpoint is technically "up" but blows its latency budget at every traffic peak. The contract must therefore reconcile the facility-availability layer with the serving-engineering layer. Availability is necessary but not sufficient, and the agreement can either contract deadline-qualified request success directly or separate uptime and latency obligations with an explicit credit-overlap rule. → serving-engineering SLOs and latency budgets in Chapter 10.11.

One rule protects the provider from their own sales team: never promise a tier of reliability the physical plant and the failure environment cannot deliver at a profit. The SLA is the output of the reliability model. Start from the design basis: the redundancy topology (Chapter 0.5), the facility tier (Chapter 12.1) and the measured component AFRs (Chapter 14.3). Run the availability-and-goodput model (Chapter 12.5), and only then write a commitment with margin between the modeled number and the promised number. Promise the modeled number with no margin and a single bad-luck month, well within the variance the Monte-Carlo predicts, turns into a service-credit hit you did not price.

The failure environment is harsher than the marketing instinct assumes. At cluster scale, interruptions are routine, but populations and event definitions matter. Meta's Llama 3 405B snapshot on up to 16,384 H100s logged 419 unexpected interruptions over 54 days, roughly one every three hours (Chapter 12.2). SemiAnalysis separately reported about seven days of MTBF for one 512-H100 cluster at a top-tier operator in October 2024. Neither observation is a per-GPU scaling law, so the contract model must use the measured interruption distribution of the service being sold. Set the goodput floor at the number the service's interruption model supports with margin and its operational reliability machinery can defend. Offer a premium tier only when extra hot spares and dedicated checkpointing measurably support it, and price the difference. How the provider recovers from a failure moves the customer's cost a long way. In SemiAnalysis's April 2026 model of a single pretraining run on 5,184 GB300s, goodput expense (the cost of the rental time a job loses to failures and recovery) came to 6.1% where a Gold-tier provider has in-job fault tolerance, 10.5% where a hyperscaler runs checkpointless training, and 20.9% where a Silver-tier provider restarts from checkpoints on hot spares (Chapter 14.1). The customer's move is to demand the measurement and attribution regime, because a high number with weak attribution is worth less than a modest number with airtight badput accounting.

Choose a service metric whose inputs both parties can reconstruct, an absolute floor the accepted workload supports, and a remedy with a defined fee basis and overlap rule. A higher headline target with weak attribution shifts the dispute without recovering work. A lower supported target reflects the service actually being sold. Price monthly breach exposure with Chapter 12.5 before accepting the obligation.