Inference Data Centers: Bursty, Distributed, Always-On

For most operators, inference is the business. Training builds the asset; inference is what earns against it, request by request, token by token, every hour of every day. By 2026 the compute has followed the money: inference is roughly two-thirds of all AI compute — up from about half in 2025 and a third in 2023 — and at large operators it is 80-90% of the served draw (Deloitte TMT Predictions 2026). Yet the reflex in the industry is still to design the inference fleet as if it were a training cluster with the dial turned down. That reflex is expensive. An inference data center optimizes a fundamentally different objective function, and nearly every subsystem decision flips sign because of it.

This chapter is the inference half of the master fork introduced in Chapter 1.1: training-shaped facilities optimize one tightly-coupled job; inference-shaped facilities optimize many independent requests against a latency SLO. We work the consequences of that one difference all the way down — the inference taxonomy and the economics shift, the reasoning-driven demand multiplier that is reshaping the decode-heavy future, latency-driven siting, the prefill/decode disaggregation that has become the defining 2026 serving pattern, the reliability and uptime posture, and the GPU-vs-ASIC selection that is finally a real fork. The serving engineering — batching, scheduling, the disaggregation tax in detail — is owned by Chapter 10.11; here we cover what those choices mean for the building.

"Inference" is not one workload. It is at least three, and they pull the facility in different directions. The mistake that recurs is treating a fleet as homogeneous when its dominant sub-mode silently dictates the SLO, the prefill:decode ratio, the memory hierarchy, and the fleet size.

Interactive / online inference is a human (or an interactive agent) waiting on the output in real time. It is governed by two latency metrics — time-to-first-token (TTFT), set by the prefill of the prompt, and time-per-output-token (TPOT), set by decode — and it is bursty and always-on: traffic can swing from 30% to 90% of capacity in minutes on a diurnal-plus-spike pattern. The design basis is high availability, proximity to users, and enough headroom to absorb the peak without breaching the SLO.

Batch / offline inference — embeddings generation, document and corpus processing, synthetic-data creation, evaluation sweeps, nightly re-scoring — has no user waiting. It is throughput-bound, not latency-bound, so it tolerates queuing, interruption, and aggressive oversubscription, and it is the natural consumer of spot capacity, off-peak power, and curtailable interconnections. It is the cheapest inference to host and the most flexible to schedule, which makes it the load you shift to soak up the headroom the interactive fleet leaves on the table.

Agentic / long-context inference is the fast-growing third mode and the one that breaks naive sizing. An agent issues many model calls per user action, carries long and growing context (tool outputs, retrieved documents, prior turns reaching toward 1M+ tokens), and interleaves reasoning with tool use. It inflates the prefill share (huge prompts), explodes the KV-cache footprint (long context held live across many concurrent sessions), and shifts the GPU:CPU ratio toward more host work for orchestration and tool calls. A fleet sized on single-turn chat assumptions is undersized for agents on both memory and prefill compute.

The structural fact of 2026 is that inference is now the larger and faster-growing share of AI infrastructure, and it grows differently from training. McKinsey's base case has inference capacity rising from ~20.9 GW to ~93.3 GW by 2030 — a ~35% CAGR — against training's ~23.1 GW to ~62.2 GW at ~22%. The crossover has happened: inference is both the bigger pool and the steeper curve. The consequence for siting and procurement is direct. Training capacity concentrates in a handful of gigawatt campuses chasing cheap firm power; inference capacity distributes toward demand, into more, smaller, latency-sited halls. As the trade press puts it, training built the campuses; inference will choose the markets.

The other structural fact is deflation. The market price of a million tokens has fallen on the order of ~10x per year for a given capability tier (LLMflation; Introl/a16z synthesis) — a Jevons-paradox dynamic where unit cost collapses while aggregate spend rises because demand more than compensates. For an inference operator this is the dominant business risk: a fleet scoped to today's $/Mtoken can be underwater in a year if its tokens-per-dollar does not improve on the same curve. That is why the inference design basis fixates on efficiency per token — tokens-per-watt and tokens-per-dollar at real batch sizes — rather than peak throughput. The full unit-economics build-up and the deflation risk are scored in Chapter 1.8.

The table is a cascade, like the one in Chapter 1.1: the left two columns are what you choose; everything to the right is consequence. Choose an interactive-dominant fleet and you have implicitly chosen geo-distribution, 2N-class uptime, a high KV-cache budget, and a willingness to oversubscribe power but not availability. Choose batch-dominant and you have chosen the opposite building — consolidated, curtailable, interruption-tolerant. The agentic column is the one most operators under-provision today, because it looks like interactive chat until the context lengths and call counts reveal a memory-and-prefill problem the single-turn sizing never anticipated.

The largest force reshaping inference demand since 2025 is the rise of reasoning models that spend compute at inference time to improve answers — the post-o1, post-R1 paradigm. A reasoning model emits a long internal chain of thought before its visible answer, turning a request that once decoded a few hundred tokens into one that may decode tens of thousands. This is test-time compute, and it has three structural consequences for the fleet that compound:

• The prefill:decode ratio shifts toward decode. Long chains of thought are autoregressive generation, which is memory-bandwidth-bound, not compute-bound. Decode now dominates the token budget for reasoning traffic, which changes which silicon and which memory hierarchy you want — and tilts the disaggregation math below.
• KV-cache pressure inflates. Every token of context and every token generated must keep its keys and values resident for attention. Long-decode reasoning, multiplied across many concurrent sessions, makes the KV-cache a first-class capacity constraint — often the binding one — not an afterthought. This is the demand driver behind the new inference-memory tier in Chapter 9.7.
• Effective demand per request multiplies. If the average request decodes 10-50x more tokens, a fleet sized on the old token-per-request assumption is undersized by roughly the same factor at constant request volume. Reasoning is, in effect, a demand multiplier hiding inside a flat user count.

The strategic read: treat the decode-heavy future as the base case, not a tail risk. It argues for memory-bandwidth-rich silicon, a deep KV-cache hierarchy, and fleet headroom for a per-request token budget that keeps climbing. The serving-engineering levers that absorb this — continuous batching, chunked prefill, speculative decoding — live in Chapter 10.11.

Training siting chases the cheapest firm megawatt and the coldest free-cooling climate, indifferent to where users are. Inference inverts the hierarchy: the binding constraint is the latency budget back to the user, and that turns siting into a coverage problem. Interactive inference lives or dies on the 30/50/100 ms perceptibility thresholds (treated in full for the edge in Chapter 1.5); a centralized fleet two time-zones from its users cannot meet a sub-50 ms TTFT no matter how fast the silicon, because the speed of light in fiber is not negotiable — roughly 0.5 ms per 100 km one-way (about 1 ms round-trip), before any switching or queuing.

The consequence is a different geographic footprint. Where training consolidates into a few gigawatt campuses, inference distributes into more, smaller, latency-sited regions — typically Tier-1 and Tier-2 metros within fiber reach of population centers. Every kilometer closer to users you buy proximity and pay for it in energy cost (metro power runs 2-4x stranded-rural power) and in the operational overhead of running many sites instead of one. Site too centrally and you breach the SLO in distant regions; site too widely and you fragment your fleet below the scale where batching efficiency and utilization hold up. The fiber-and-latency screen that governs this is Chapter 3.6; the market-cluster playbook is Chapter 3.13; the reordered siting hierarchy is Chapter 3.1.

The defining serving architecture of 2026 is the recognition that the two phases of a request want different hardware. Prefill — processing the entire prompt to produce the first token and the initial KV-cache — is compute-bound: it saturates the accelerator's matrix engines and scales with prompt length. Decode — generating each subsequent token autoregressively — is memory-bandwidth-bound: each step reads the whole model and the growing KV-cache to emit one token, leaving the matrix engines mostly idle. Run both on the same GPU and they interfere: a long prefill stalls the decode stream of every other request sharing the device, and you cannot tune the hardware for either because it is doing both.

Disaggregated serving splits them onto separate pools — a prefill pool sized for compute, a decode pool sized for memory bandwidth and capacity — connected by a fast fabric that ships the KV-cache from prefill to decode. By early 2026 this is supported across every major open-source engine (vLLM, SGLang, TensorRT-LLM) and orchestrated by frameworks like NVIDIA Dynamo and llm-d, with NIXL the standard KV-transfer transport over RDMA/NVLink. The payoff is higher goodput per dollar and the ability to mix silicon — expensive compute-dense parts for prefill, cheaper memory-rich parts for decode. The cost is the disaggregation tax: every request now pays a KV-cache transfer across the fabric, which only nets out when the pools are large enough and the fabric fast enough that the transfer is cheaper than the interference it eliminates.

The facility consequence is that the KV-cache becomes a first-class, tiered resource — held in HBM where it is hottest, spilled to local NVMe and Ethernet-attached flash as it cools, reused across requests that share a prefix. That memory hierarchy is its own subsystem now, engineered in Chapter 9.7. For the building, disaggregation means the inference hall is no longer a uniform sea of identical nodes; it is a heterogeneous fleet whose pool ratios you tune to your prefill:decode mix — which, per the reasoning shift above, is moving.

Two of inference's defining properties — loose coupling and uncorrelated per-request load — open headroom that a synchronous training job forbids, and capturing it is one of the clearest efficiency levers in the fleet.

Fabric. Most inference requests fit inside a single node or a small scale-up domain, so the back-end fabric does not carry the all-reduce traffic that forces training to 1:1 non-blocking. Inference runs comfortably at 2:1-3:1 oversubscription, and a 2:1 "optimized" fabric cuts back-end network cost by roughly 31% (SemiAnalysis; contested, single-source). Building a non-blocking fabric for an inference business spends that 31% on bisection bandwidth that never carries traffic — the exact anti-pattern named in Chapter 1.1. The money saved is better spent on geo-distribution and uptime. Topology and oversubscription are engineered in Chapter 8.5.

Power. Training's synchronous steps make thousands of GPUs draw their peak in lockstep, leaving only ~3% power-oversubscription headroom — the facility must provision near the synchronized peak. Inference's per-request peaks are uncorrelated, so the aggregate load is far smoother, opening ~21% headroom: you can provision more served capacity behind the same megawatts and reclaim power from low-priority work with simple priority-based capping (Uptime Institute Journal; arXiv power-profile studies). The consequence is real density-per-megawatt that training cannot match — but it must be engineered with capping and ride-through, because reasoning and agentic traffic make the per-request peaks lumpier than single-turn chat. The grid-facing transient behavior this implies is in Chapter 12.2's reliability frame.

This is where inference and training are most starkly opposed, and where copying the training posture costs the most. A synchronous training job already tolerates failure by checkpoint-and-resume — a node dies, the job restarts from the last checkpoint, and 2N facility power buys nines the workload does not value (the over-provisioned-redundancy anti-pattern of Chapter 1.1). An always-on inference business is the inverse: every minute of outage is lost revenue and a breached SLA, and there is no checkpoint to resume from — the request is simply dropped and the user sees an error.

So the posture flips. Inference justifies 2N / Tier-IV-class facility power with N+1 cooling on standby — the ~26 min/yr unavailability of a Tier-IV target (99.995%) is a business requirement, not gold-plating. But the deeper move, and the one that distinguishes a well-scoped inference fleet, is to buy resilience at the fleet level rather than only the facility level: geo-redundancy. Because inference is stateless per-request and distributed for latency anyway, a failed site can fail over to another region, so the marginal availability of any single hall matters less than the redundancy of the whole footprint. The right design often spends on a second region before it spends on the last increment of single-site uptime — N+1 across sites can beat 2N within one. The reliability rethink that reframes facility availability against fleet goodput is Chapter 12.2.

Training has effectively one merchant answer — the latest flagship GPU — because the workload rewards raw FLOPS, the largest scale-up domain, and the most mature collective-comms software. Inference is the first archetype where the silicon choice is a genuine, contestable fork, because the scoring metric changes: not peak FLOPS but tokens-per-watt and tokens-per-dollar at your real batch sizes, context lengths, and SLOs. On that scoreboard, purpose-built inference ASICs become competitive, and several are now in volume.

Google's Ironwood (TPU v7) is explicitly "the first TPU for the age of inference": 192 GiB of HBM3E at 7.4 TB/s per chip, 4,614 FP8 TFLOPS, scaling to 9,216-chip pods delivering 42.5 FP8 ExaFLOPS — a memory-bandwidth-rich design aimed squarely at the decode-heavy future. AWS's Inferentia/Trainium line and the hyperscaler XPUs (Maia, MTIA) target the same tokens-per-dollar objective for captive fleets. These are surveyed in Chapter 7.4; the selection-and-TCO methodology is Chapter 7.11.

An ASIC can win decisively on tokens-per-watt for a stable, high-volume model family — but it carries software-ecosystem and lock-in risk (the CUDA moat is real, and a custom part with a thin software stack can leave throughput on the table), and it is a bet that the model architecture you optimize for stays put. Against an inference market deflating ~10x/year and reasoning reshaping the prefill:decode mix, a part optimized for last year's workload can be the wrong part this year. The rational posture for most operators is to score the fork at their own batch sizes and context lengths — peak-FLOPS datasheets are nearly useless here — and to keep the silicon decision as reversible as the procurement decision, because both move faster than a facility's depreciation clock.

The recurring inference mis-scopes all trace to one root cause: designing the inference fleet as a de-rated training cluster instead of from its own objective function. Four are worth naming:

• Training fabric for an inference business. A 1:1 non-blocking back-end for a workload whose requests fit inside a node wastes ~31% of back-end cost on bisection bandwidth that never carries traffic. Oversubscribe 2:1-3:1 and spend the savings on geo-distribution and uptime.
• Sizing the whole fleet to single-turn chat. Reasoning and agentic traffic multiply the per-request token budget and explode KV-cache pressure; a fleet sized on old token-per-request assumptions is undersized on memory and decode by the same factor. Size to the decode-heavy base case.
• Centralizing inference for power economics. Consolidating an interactive fleet onto one cheap-power gigawatt campus breaches the latency SLO for distant users — the speed of light does not negotiate. Distribute toward demand and pay the energy premium knowingly.
• Buying peak FLOPS instead of tokens-per-watt. Selecting inference silicon on datasheet FLOPS ignores that decode is memory-bound; the chip that wins on paper can lose 2x on realized goodput at your batch sizes and context lengths.