Inference Data Centers: Bursty, Distributed, Always-On

For most operators, inference is the business. Training builds the asset; inference is what earns against it, request by request, token by token, every hour of every day. By 2026 the compute has followed the money: inference is roughly two-thirds of all AI compute — up from about half in 2025 and a third in 2023 — and at large operators it is 80-90% of the served draw (Deloitte TMT Predictions 2026). Yet the reflex in the industry is still to design the inference fleet as if it were a training cluster with the dial turned down. That reflex is expensive. An inference data center optimizes a fundamentally different objective function, and nearly every subsystem decision flips sign because of it.

Chapter 1.1 split the archetypes on one axis: training-shaped facilities optimize a single tightly-coupled job; inference-shaped facilities optimize many independent requests against a latency SLO. This chapter works the consequences of that difference all the way down — the inference taxonomy and the economics shift, the reasoning-driven demand multiplier reshaping the decode-heavy future, latency-driven siting, the prefill/decode disaggregation that has become the defining 2026 serving pattern, the reliability posture, and a GPU-vs-ASIC selection that is genuinely contestable for the first time. The serving engineering — batching, scheduling, the disaggregation tax in detail — is owned by Chapter 10.11; here we cover what those choices mean for the building.

"Inference" covers at least three distinct workloads, and they pull the facility in different directions. The mistake that recurs is treating a fleet as homogeneous when its dominant sub-mode silently dictates the SLO, the prefill:decode ratio, the memory hierarchy, and the fleet size.

Interactive / online inference is a human (or an interactive agent) waiting on the output in real time. It is governed by two latency metrics — time-to-first-token (TTFT), set by the prefill of the prompt, and time-per-output-token (TPOT), set by decode — and it is bursty and always-on: traffic can swing from 30% to 90% of capacity in minutes on a diurnal-plus-spike pattern. The design basis is high availability, proximity to users, and enough headroom to absorb the peak without breaching the SLO.

Batch / offline inference — embeddings generation, document and corpus processing, synthetic-data creation, evaluation sweeps, nightly re-scoring — has no user waiting. It is throughput-bound, not latency-bound, so it tolerates queuing, interruption, and aggressive oversubscription, and it is the natural consumer of spot capacity, off-peak power, and curtailable interconnections. It is the cheapest inference to host and the most flexible to schedule, which makes it the load you shift to soak up the headroom the interactive fleet leaves on the table.

Agentic / long-context inference is the fast-growing third mode and the one that breaks naive sizing. An agent issues many model calls per user action, carries long and growing context (tool outputs, retrieved documents, prior turns reaching toward 1M+ tokens), and interleaves reasoning with tool use. It inflates the prefill share (huge prompts), explodes the KV-cache footprint (long context held live across many concurrent sessions), and shifts the GPU:CPU ratio toward more host work for orchestration and tool calls. A fleet sized on single-turn chat assumptions is undersized for agents on both memory and prefill compute.

The structural fact of 2026 is that inference is now the larger and faster-growing share of AI infrastructure, and it grows differently from training. McKinsey's base case has inference capacity rising from ~20.9 GW to ~93.3 GW by 2030 — a ~35% CAGR — against training's ~23.1 GW to ~62.2 GW at ~22%. The crossover has happened: inference is both the bigger pool and the steeper curve. The consequence for siting and procurement is direct. Training capacity concentrates in a handful of gigawatt campuses chasing cheap firm power; inference capacity distributes toward demand, into more, smaller, latency-sited halls. As the trade press puts it, training built the campuses; inference will choose the markets.

The other structural fact is deflation. Ramp's customer cohort paid about $2.50 per million tokens in March 2025, down from about $10 per million roughly one year earlier; this is not a universal market average — a Jevons-paradox dynamic where unit cost collapses while aggregate spend rises because demand more than compensates. For an inference operator this is the dominant business risk: a fleet scoped to today's $/Mtoken can be underwater in a year if its tokens-per-dollar does not improve on the same curve. So the inference design basis fixates on efficiency per token — tokens-per-watt and tokens-per-dollar at real batch sizes — rather than peak throughput. The full unit-economics build-up and the deflation risk are scored in Chapter 1.8.

The table is a cascade, like the one in Chapter 1.1: the left two columns are what you choose; everything to the right is consequence. Choose an interactive-dominant fleet and you have chosen geo-distribution, a high KV-cache budget, and a willingness to oversubscribe power but not availability. Its revenue-critical service objective often justifies stronger facility topology, but the required maintenance and fault outcomes, not the workload label, select that topology. A batch-dominant fleet is the opposite building — consolidated, curtailable, interruption-tolerant. The agentic column is the one most operators under-provision today, because it looks like interactive chat until the context lengths and call counts reveal a memory-and-prefill problem the single-turn sizing never anticipated.

The largest force reshaping inference demand since 2025 is the rise of reasoning models that spend compute at inference time to improve answers — the post-o1, post-R1 paradigm. A reasoning model emits a long internal chain of thought before its visible answer, turning a request that once decoded a few hundred tokens into one that may decode tens of thousands. This is test-time compute, and it has three structural consequences for the fleet that compound:

• The prefill:decode ratio shifts toward decode. Long chains of thought are autoregressive generation, which is memory-bandwidth-bound, not compute-bound. Decode now dominates the token budget for reasoning traffic, which changes which silicon and which memory hierarchy you want — and tilts the disaggregation math below.
• KV-cache pressure inflates. Every token of context and every token generated must keep its keys and values resident for attention. Long-decode reasoning, multiplied across many concurrent sessions, makes the KV-cache a first-class capacity constraint — often the binding one — not an afterthought. This is the demand driver behind the new inference-memory tier in Chapter 9.7.
• Effective demand per request multiplies. If the average request decodes 10-50x more tokens, a fleet sized on the old token-per-request assumption is undersized by roughly the same factor at constant request volume. Reasoning is, in effect, a demand multiplier hiding inside a flat user count.

Treat the decode-heavy future as the base case rather than a tail risk. It argues for memory-bandwidth-rich silicon, a deep KV-cache hierarchy, and fleet headroom for a per-request token budget that keeps climbing. The serving-engineering levers that absorb this — continuous batching, chunked prefill, speculative decoding — live in Chapter 10.11.

Training siting chases the cheapest firm megawatt and the coldest free-cooling climate, indifferent to where users are. Inference inverts the hierarchy: the binding constraint is the latency budget back to the user, and that turns siting into a coverage problem. Interactive inference lives or dies on the 30/50/100 ms perceptibility thresholds (treated in full for the edge in Chapter 1.5); a centralized fleet two time-zones from its users cannot meet a sub-50 ms latency budget no matter how fast the silicon, because the speed of light in fiber is not negotiable — roughly 0.5 ms per 100 km one-way (about 1 ms round-trip), before any switching or queuing.

The consequence is a different geographic footprint. Where training consolidates into a few gigawatt campuses, inference distributes into more, smaller, latency-sited regions — typically Tier-1 and Tier-2 metros within fiber reach of population centers. Siting closer to users buys proximity and pays for it in energy cost (metro power runs 2-4x stranded-rural power) and in the operational overhead of running many sites instead of one. Site too centrally and you breach the SLO in distant regions; site too widely and you fragment your fleet below the scale where batching efficiency and utilization hold up. The fiber-and-latency screen that governs this is Chapter 3.6; the market-cluster playbook is Chapter 3.13; the reordered siting hierarchy is Chapter 3.1.

The defining serving architecture of 2026 is the recognition that the two phases of a request want different hardware. Prefill — processing the entire prompt to produce the first token and the initial KV-cache — is compute-bound: it saturates the accelerator's matrix engines and scales with prompt length. Decode — generating each subsequent token autoregressively — is memory-bandwidth-bound: each step reads the whole model and the growing KV-cache to emit one token, leaving the matrix engines mostly idle. Run both on the same GPU and they interfere: a long prefill stalls the decode stream of every other request sharing the device, and you cannot tune the hardware for either because it is doing both.

Disaggregated serving splits them onto separate pools — a prefill pool sized for compute, a decode pool sized for memory bandwidth and capacity — connected by a fast fabric that ships the KV-cache from prefill to decode. By early 2026 this is supported across every major open-source engine (vLLM, SGLang, TensorRT-LLM) and orchestrated by frameworks like NVIDIA Dynamo and llm-d, with NIXL the standard KV-transfer transport over RDMA/NVLink. The payoff is higher goodput per dollar and the ability to mix silicon — expensive compute-dense parts for prefill, cheaper memory-rich parts for decode. The cost is the disaggregation tax: every request now pays a KV-cache transfer across the fabric, which only nets out when the pools are large enough and the fabric fast enough that the transfer is cheaper than the interference it eliminates.

The facility consequence is that the KV-cache becomes a first-class, tiered resource — held in HBM where it is hottest, spilled to local NVMe and Ethernet-attached flash as it cools, reused across requests that share a prefix. That memory hierarchy is its own subsystem now, engineered in Chapter 9.7. For the building, disaggregation means the inference hall is no longer a uniform sea of identical nodes; it is a heterogeneous fleet whose pool ratios you tune to your prefill:decode mix — which, per the reasoning shift above, is moving.

Two of inference's defining properties — loose coupling and uncorrelated per-request load — open headroom that a synchronous training job forbids, and capturing it is one of the clearest efficiency levers in the fleet.

Fabric. Many local inference requests fit inside one node or a small scale-up domain, so some designs validate upper-tier 2:1–3:1 oversubscription; a named 2:1-versus-1:1 model estimates roughly 31% lower back-end cost. Those are scoped examples, not an inference default. Distributed MoE, KV movement, or prefill/decode disaggregation can require high bisection. Derive each tier from measured request, KV and expert-parallel traffic, placement, failure headroom, and the tail-latency SLO, then validate it on the target fabric. Topology and oversubscription are engineered in Chapter 8.5.

Power. Training's synchronous steps make thousands of GPUs draw their peak in lockstep, leaving only ~3% power-oversubscription headroom — the facility must provision near the synchronized peak. Inference's per-request peaks are uncorrelated, so the aggregate load is far smoother, opening ~21% headroom: you can provision more served capacity behind the same megawatts and reclaim power from low-priority work with simple priority-based capping (Uptime Institute Journal; arXiv power-profile studies). The result is density-per-megawatt that training cannot match — but it must be engineered with capping and ride-through, because reasoning and agentic traffic make the per-request peaks lumpier than single-turn chat. The grid-facing transient behavior this implies is in Chapter 12.2's reliability frame.

Training and inference expose different outage consequences, but neither workload label selects a facility topology. Checkpoint-and-resume can reduce a training job’s lost work and recovery cost; request retry, replica capacity, zone routing, and regional failover can mask an inference-site interruption. In both cases, define the maintenance and fault states, allowable transfer interruption, post-event capacity, path and control independence, common modes, recovery SLO, and contractual consequence.

An always-on inference business may place high value on uninterrupted single-site service, but that does not automatically require 2N or Tier-IV-class power: sufficient independent replica, zone, or region capacity can satisfy the end-to-end serving objective with a different site topology. Conversely, a training contract or recovery limit can require path continuity despite checkpointing. Select the site and fleet designs together, then prove the named states. The quantitative comparison is in Chapter 12.5.

Training has effectively one merchant answer, the rack-scale NVLink domain: GB300 NVL72 is the 2026 volume platform, with VR200 NVL72 in full production behind it (NVIDIA Q2 FY27, August 2026). The workload rewards raw FLOPS, the largest scale-up domain, and the most mature collective-comms software. Inference is the first archetype where the silicon choice is a genuine, contestable fork, because the scoring metric changes from peak FLOPS to tokens-per-watt and tokens-per-dollar at your real batch sizes, context lengths, and SLOs. On that scoreboard, purpose-built inference ASICs become competitive, and several are now in volume. The GPU they are scored against is that same rack: the 72-GPU NVLink domain is what makes prefill/decode disaggregation economic, which is why Azure built its first GB300 cluster for OpenAI reasoning-model inference (October 2025) and AWS's P6e-GB300 instances reached GA in December 2025.

Google's Ironwood (TPU v7) is explicitly "the first TPU for the age of inference": 192 GiB of HBM3E at 7.4 TB/s per chip, 4,614 FP8 TFLOPS, scaling to 9,216-chip pods delivering 42.5 FP8 ExaFLOPS — a memory-bandwidth-rich design aimed squarely at the decode-heavy future. AWS's Inferentia/Trainium line and the hyperscaler XPUs (Maia, MTIA) target the same tokens-per-dollar objective for captive fleets. At Cloud Next 2026 (April) Google previewed the eighth-generation split behind it — TPU 8t (training: 3D torus, 9,600-chip superpods, Google-claimed up to 2.7x training $/perf vs Ironwood) and TPU 8i (inference: a new all-to-all 'Boardfly' interconnect, up to 80% better inference $/perf at low-latency MoE serving; both up to 2x perf/W — vendor figures, Apr 2026). Ironwood remains the GA part in 2026. These are surveyed in Chapter 7.4; the selection-and-TCO methodology is Chapter 7.11.

An ASIC can win decisively on tokens-per-watt for a stable, high-volume model family — but it carries software-ecosystem and lock-in risk (the CUDA moat is real, and a custom part with a thin software stack can leave throughput on the table), and it is a bet that the model architecture you optimize for stays put. Against the Ramp customer cohort's roughly $10/M → $2.50/M decline through March 2025 (not a universal market average) and reasoning reshaping the prefill:decode mix, a part optimized for last year's workload can be the wrong part this year. The rational posture for most operators is to score the fork at their own batch sizes and context lengths — peak-FLOPS datasheets are nearly useless here — and to keep the silicon decision as reversible as the procurement decision, because both move faster than a facility's depreciation clock.

The recurring inference mis-scopes all trace to one root cause: designing the inference fleet as a de-rated training cluster instead of from its own objective function. Four are worth naming:

• Sizing fabric from the word inference. A named 1:1-versus-2:1 model estimates a ~31% back-end cost difference, but the workload label does not set the ratio. Node-local serving may validate upper-tier oversubscription; distributed MoE, KV movement, or prefill/decode disaggregation may require more bisection. Derive each tier from measured traffic, placement, failure headroom, and the tail-latency SLO.
• Sizing the whole fleet to single-turn chat. Reasoning and agentic traffic multiply the per-request token budget and explode KV-cache pressure; a fleet sized on old token-per-request assumptions is undersized on memory and decode by the same factor. Size to the decode-heavy base case.
• Centralizing inference for power economics. Consolidating an interactive fleet onto one cheap-power gigawatt campus breaches the latency SLO for distant users. Distribute toward demand and pay the energy premium knowingly.
• Buying peak FLOPS instead of tokens-per-watt. Selecting inference silicon on datasheet FLOPS ignores that decode is memory-bound; the chip that wins on paper can lose 2x on realized goodput at your batch sizes and context lengths.