Inference Serving Engineering: SLOs, Batching, Disaggregation & Goodput-Optimal Scheduling

Every other layer of the stack in this guide exists to put a working accelerator under a token. This chapter covers the last hop: the software that turns a powered, healthy, scheduled GPU into served tokens a user will pay for. It is where the capital you spent on power, cooling, fabric and silicon either earns its $/token or leaks away as idle bubbles, missed deadlines and over-provisioned headroom. It is also where the objective is almost always misstated. The naive goal is throughput, tokens per second per GPU. The naive fix for latency complaints is to chase p99 latency. Each is wrong on its own, because it optimizes one axis while wrecking the other. The right objective is goodput: the rate of tokens from requests that also met their quality and latency limits. A server doing 50,000 tok/s, where a third of the output tokens belong to requests that blew their deadline, delivers about 33,000 tok/s of goodput. That is less than a server doing 38,000 tok/s where every request landed, and only goodput tracks revenue.

The chapter defines the SLO (TTFT, TPOT, ITL), then works through batching, chunked prefill and the defining split of the 2025-2026 stack, prefill/decode (P/D) disaggregation, whose tax is the KV-cache transfer. It then sizes a bursty, always-on fleet with queueing theory, prices speculative decoding, routes on prefix-cache hits, and compares the engines and the orchestrators above them. The ranking of the three latency targets decides every downstream lever.

Autoregressive inference is two physically different workloads behind one API. Prefill ingests the whole prompt in a single forward pass and computes the KV cache for every input token at once. Once the prompt or batch supplies enough work to saturate the GPU, it is compute-bound; below that point, fixed launch and scheduling costs dominate. Decode then emits one token at a time, and each step reads the entire model's weights and the growing KV cache from HBM to produce a single token. While the batch is too small to amortize each weight read across enough sequences, decode is memory-bandwidth-bound and leaves tensor-core capacity idle. These are operating regimes, not invariants. Prompt length, batch, quantization, model, GPU and backend can move either phase across the compute/memory boundary, so profile the target workload before choosing hardware or scheduling policy.

The user-facing SLO is expressed in three numbers, and you must know which one binds before you tune anything:

• TTFT (time-to-first-token) — the wait from request arrival to the first streamed token. Queueing delay plus the prefill forward pass dominate it, so it grows with prompt length and with how loaded the prefill path is. Users feel it as "is it responding?". Interactive chat typically targets the low hundreds of milliseconds, and voice or agentic loops need far tighter.
• TPOT (time-per-output-token) — the average steady-state decode-step time once generation is underway. It sets the streaming speed, and so the perceived reading pace. A TPOT of ~40-50 ms (~20-25 tok/s, roughly 900-1,100 words per minute) streams comfortably faster than human reading speed (~200-300 wpm). Reasoning workloads that emit thousands of decode tokens make TPOT the dominant contributor to total request latency.
• ITL (inter-token latency) — the distribution of gaps between consecutive tokens, not just the mean. Batching interference shows up here: a smooth TPOT average can hide ugly ITL stalls when a heavy prefill joins a batch of in-flight decodes and freezes the stream for tens of milliseconds. SLOs that specify only mean TPOT and ignore p95/p99 ITL get gamed by exactly this.

The decision that cascades from here is the ranking of these targets. A batch-summarization product cares about end-to-end completion and tolerates a slow first token, so TTFT is loose. A live voice agent cares about first-token responsiveness above all, so TTFT binds hard. A long-form reasoning product lives and dies on TPOT/ITL because it emits a wall of tokens. Whichever binds tells you which physical phase to protect, and that ranking chooses between chunked prefill and disaggregation below.

Static batching waits for N requests, runs them in lockstep and returns when the longest finishes. It is the obvious approach and it is catastrophic for generation: requests finish at wildly different lengths, so the whole batch is held hostage by its longest sequence while finished slots sit idle. Continuous (in-flight) batching is the fix that defines modern serving. The scheduler works at the granularity of a single decode step, evicting completed sequences and admitting waiting ones every iteration, so the GPU never waits for a batch to drain. Paired with paged KV-cache management (PagedAttention and its descendants), which allocates KV memory in non-contiguous blocks to avoid the fragmentation of static pre-allocation, continuous batching lets a single GPU keep dozens to hundreds of sequences in flight at high utilization. It is table stakes in 2026; an engine that lacks it is not a serious contender.

Continuous batching solves the decode-side bubble but brings back the conflict from the callout above: when do you admit a new prompt's prefill into a batch that is busy decoding? The two classical policies are both bad. Prefill-first admits new prompts eagerly to minimize their TTFT, but a long prefill then freezes every in-flight decode stream: good TTFT, terrible ITL. Decode-first drains decodes before admitting prefill, which protects the streams but lets new requests queue and inflates TTFT. Chunked prefill dissolves the dilemma. It splits a long prompt's prefill into bite-sized token chunks and interleaves them with ongoing decode steps inside a fixed per-iteration token budget. No single iteration is dominated by a giant prefill, so decode streams keep flowing (ITL stays smooth) while new prompts make steady prefill progress (TTFT stays bounded). The cost is a modest hit to peak prefill throughput and a tuning knob, the per-step token budget, that trades TTFT against TPOT. Set it against your SLO ranking, not a vendor default.

Chunked prefill lets aggregated serving, one pool of GPUs doing both phases, hold both TTFT and TPOT reasonably well across a range of traffic. It runs out of room in one regime. With strict TPOT/ITL and long prompts, interleaving still ties the phases together, and long prefills keep stretching decode steps. The phases also want different hardware. Prefill is compute-bound and rewards raw FLOPs and large tensor parallelism; decode is memory-bound and rewards HBM capacity, bandwidth and high batch concurrency. Forcing both onto one homogeneous tier means provisioning for the worse case on both axes and wasting one of them. In 2026 this became a procurement choice. NVIDIA's Groq 3 LPX racks, announced in full production in August 2026 with first cloud deployments slated for late 2026, take the decode tier while Rubin GPUs keep prefill. Such a split adds a KV hop between device types and a second, non-CUDA toolchain, and the performance figures are NVIDIA's own (→ 7.2).

Prefill/decode (P/D) disaggregation is the architectural answer: physically split the fleet into a prefill tier and a decode tier, each independently sized, parallelized and scaled. A request runs prefill on the prefill tier, then its KV cache is handed to a decode worker, which streams the output. The wins are real and measured. A long prompt stops stalling streams already in flight, and queues at the prefill tier no longer depend on decode load. First-token latency still pays the KV transfer. The decode tier runs uninterrupted at the big batch it wants, so TPOT/ITL stay smooth. You can also scale the two tiers independently: the runtime-reconfigurable xPyD pattern, x prefill workers feeding y decode workers, lets you re-balance as your traffic's prompt-to-output ratio shifts.

Which mode wins depends on which target binds. TaiChi (2025) reports that aggregation is best when TTFT is tight and TPOT relaxed, and disaggregation is best when TPOT is strict and TTFT relaxed. When both bind, neither wins, and TaiChi's hybrid reaches up to 77% more goodput. vLLM's 2026 guide agrees on direction: missed ITL p99 is the main reason to disaggregate, and you should stay collocated if TTFT is the binding target.

The catch is the disaggregation tax. The KV cache computed during prefill must reach the decode worker before generation can begin, so the transfer sits on the critical path of TTFT. For a long prompt the cache is large. Per token it is 2 (K and V) × layers × KV heads × head dimension × bytes per element; for Llama 3.1 70B in BF16 that is 2 × 80 × 8 × 128 × 2 B = 320 KiB, so a 32k-token prompt moves about 10 GiB. Grouped-query attention counts KV heads, not query heads, and MLA models such as DeepSeek-V3 store less. The transport therefore matters enormously. Over fast intra-node NVLink the tax is small. Across a node boundary it demands RDMA and a purpose-built transfer library: NVIDIA's NIXL is the de facto transport that llm-d and Dynamo both lean on, moving KV asynchronously across memory and storage tiers. Over a slow link the transfer can erase whatever TTFT benefit isolating prefill gave you. So disaggregation pays only at sufficient scale and on a fast fabric, and the KV-cache hierarchy is its own engineering problem. Chapter 9.7 covers the transfer mechanics, the memory tiering and the offload-to-flash economics; this chapter governs when to pay the tax, not how to plumb it.

Mixture-of-experts (MoE) models change the decode layout. DeepSeek-V3/R1, Kimi K2, gpt-oss and Llama 4 send each token to a few of many experts, so decode usually runs attention data-parallel and spreads the experts across many GPUs (wide expert parallelism, EP). Every MoE layer then does an all-to-all dispatch and combine of tokens, and that collective sets how wide EP can usefully go. On an 8-GPU HGX node, EP wider than 8 sends the all-to-all over the scale-out NICs at roughly an order of magnitude less bandwidth per GPU than NVLink. That is why rack-scale NVLink matters for MoE inference: a 72-GPU NVLink domain keeps EP32 and wider off the NICs. NVIDIA reports up to 1.8× more per-GPU throughput for DeepSeek-R1 at EP32 than at EP8, at 100 tokens/s per user with disaggregated serving and multi-token prediction (NVIDIA, October 2025, vendor-measured).

Wide EP also needs expert load balancing, because a popular expert saturates its GPU while others idle; engines place or replicate hot experts from observed routing (EPLB), statically or online. In P/D the usual pairing is wide-EP decode with a narrower prefill tier: SGLang's May 2025 DeepSeek deployment on 96 H100s benchmarked prefill at EP32 and decode at EP72. EPLB's redundant experts made that decode width work: 288 expert slots instead of 256 divide evenly across 72 GPUs. Placement and fabric are covered in Chapter 10.2 and Chapter 8.5.

Size the CPU front end too. vLLM (v0.31+) can split tokenization and output parsing into a CPU-only render tier. Plain rendering is cheap, about 15 ms of CPU for a 9k-token prompt. Parsed streaming of long reasoning output measured about 9x the CPU and about 15% higher median end-to-end latency at matched load (vLLM project benchmark, September 2026), and until a parser-state cache lands it can saturate that tier. Before concluding that disaggregation does not pay, measure KV transfer time in a disaggregated trial: if decode-side transfers average hundreds of milliseconds, fix the fabric first.

With the levers on the table, the objective they serve can be stated precisely. Define goodput as output tokens per second from completed requests that met the full quality, error and latency requirements. A request that met all of them contributes every token. Tokens from a request that missed any one are excluded from accepted output, because they consumed capacity while failing the customer. Serving then becomes a constrained optimization: maximize goodput subject to TTFT ≤ target and TPOT ≤ target at the required percentile. Track request attainment separately: compliant logical requests divided by all offered eligible logical requests, counting rejects, errors, timeouts and exhausted retries as misses. That keeps token weighting from hiding failures on short requests.

Practitioners often get one consequence wrong: the throughput-maximizing operating point can be past the goodput-maximizing one. As batch size and load rise, raw tokens/sec keeps climbing toward saturation. But TTFT and TPOT degrade non-linearly as queues build, and at some load the marginal tokens you add are all breaching the SLO. Goodput then peaks before throughput and falls, because beyond the knee every admitted request slows the ones already running, converting good tokens into bad ones. Pushing admitted load past that point is a goodput error: you report a throughput number while shipping deadline misses. Find the knee on a measured curve, under your own offered load and length mix, and choose the least-cost point below it that still keeps failure headroom. Headroom that looks like waste on a utilization dashboard is the SLO doing its job. This mirrors the reliability-overhead framing in training goodput, applied to latency rather than failures; the economic version governs $/token in Chapter 1.8 and Chapter 7.11.

Online inference traffic is bursty and always on. Arrivals are roughly Poisson at short timescales with strong diurnal and event-driven swings, and there is no batch window to hide behind because a user is waiting. Sizing the fleet is a queueing problem, and the result that matters surprises people: tail latency explodes as utilization approaches 1, and it does so non-linearly. A transparent M/M/1 mean-value illustration gives mean response time E[T] = S/(1−ρ) and mean queue wait E[W_q] = ρS/(1−ρ), where S is mean service time. At ρ=0.80, mean response is 5S and mean queue wait is 4S; at ρ=0.95 they are 20S and 19S. This is not an M/M/c result and not a p95/p99 prediction. For a multi-server illustration, state c, λ and μ and calculate Erlang C under its assumptions; size production tails from measured arrival and service-time traces, batching, cache state, routing, failures and the actual SLO. Because TTFT contains queue wait, a TTFT SLO is really a cap on ρ: you cannot run a latency-bound fleet near saturation and meet a tail SLO, however good your engine is.

Three decisions follow. (1) Size to the goodput knee, not to average load. Provision enough replicas that peak diurnal demand still sits below the utilization ceiling your TTFT SLO implies, which means carrying idle headroom by design. (2) Exploit the c in M/M/c. In a homogeneous model, a shared work-conserving queue can reduce stranded capacity relative to isolated queues, because a free eligible replica can take the next request. In production, cache locality, model residency, batching, heterogeneity and failure domains can reverse part of that gain, so compare routing policies on measured traces and tail SLOs. (3) Shed or downgrade rather than breach. When a burst exceeds capacity, admission control that rejects the marginal request or routes it to a smaller model preserves goodput for everyone else, whereas blindly admitting converts the whole fleet into badput. Chapter 12.4 develops the SLA framing of these targets: what you promise, what you measure, what a breach costs.

When the batch is too small to amortize each weight read, decode is sequential and memory-bandwidth-bound: every token needs a full pass over the weights, and the FLOPs sit idle waiting on HBM. Speculative decoding exploits that idle compute. A cheap draft mechanism (a small draft model, a few extra prediction heads as in Medusa/EAGLE/MTP, or n-gram lookahead) proposes several tokens ahead. The full model then verifies all of them in a single parallel forward pass and accepts the longest correct prefix. When the draft is good, you get multiple tokens per expensive model pass instead of one: fewer sequential decode steps and lower TPOT. Under exact speculative sampling, verification is exact, so the output keeps the target model's own distribution and quality is unchanged. That guarantee is per method, not per family. Medusa's typical-acceptance scheme deliberately relaxes distribution matching and Medusa-2 retrains the backbone, so on those paths measure task quality, not acceptance rate alone. Parallel decoding schemes (lookahead, Jacobi-style) generalize the idea without a separate draft model.

The economics are a clean trade with a sharp boundary. Speculation spends FLOPs to buy back memory-bound stalls. It therefore pays when you have spare FLOPs: at low-to-moderate batch sizes, where decode is firmly memory-bound and the tensor cores are starved. The win there is real: 2-3x effective decode speedup on amenable workloads, at high acceptance rates. As batch size climbs, the decode step becomes compute-bound because weight reads are now amortized across many sequences. The spare FLOPs vanish, the extra verification work becomes pure overhead, and at high batch speculation can reduce throughput. So the setting is regime-dependent: on for low-batch, latency-bound work, off (or adaptive) at high batch. "Adaptive" became a shipping mechanism in 2026. DeepSeek's DSpark drafter landed natively in vLLM (Jul 2026), and its confidence-scheduled adaptive verification (Aug 2026) trims the verification set per step, holding the Pareto frontier from concurrency 1 to 256 on 8×B300 in vLLM's published benchmark. The static-k collapse at high batch is the reason the adaptive mechanism exists, not a law it repeals (current limits: SM100 sparse-MLA backends, no LoRA/PP). Acceptance rate is the other knob: a weak draft is rejected often and wastes the verification pass, so the draft-model choice is itself an economic decision. For agentic and reasoning workloads that emit long, low-batch decode streams under tight TPOT, speculation is one of the largest levers available. For a saturated high-throughput batch endpoint it is often net-negative.

By 2026 the serving layer has three widely used candidate engines and a layer of disaggregated orchestrators above them. The engine choice trades portability and iteration speed against peak hardware-locked performance against programmable structured workloads. It is reversible (you can re-platform an endpoint) but not cheap, because operational tooling, kernels and tuning accrete around whatever you pick.

vLLM is an open engine with multiple hardware backends. It popularized PagedAttention and continuous batching and runs on NVIDIA, AMD and other backends; check model, operator and backend support for your exact deployment. TensorRT-LLM targets NVIDIA GPUs. Its 1.3 and 1.4 pre-releases (1.3 from rc21, July 2026; 1.4.0rc0, October 2026) have only the PyTorch backend, as the migration documentation describes, so the historical convert/build-engine recipe does not apply there. The latest stable release, 1.2.1 (April 2026), still has the TensorRT engine backend. NVIDIA's release notes list the removal under 1.2, so check the tag you actually install. Benchmark the model you plan to deploy instead of assuming leadership. SGLang is a high-throughput engine whose RadixAttention prefix cache is first-class; its structured-generation frontend suits agentic, multi-turn, branching and tool-calling workloads where request structure can be exploited. Above the engines sit the disaggregated orchestrators: NVIDIA Dynamo (KV-aware routing, xPyD disaggregation, NIXL transport, multi-engine backends including TensorRT-LLM, vLLM and SGLang), the community llm-d (Kubernetes-native disaggregated serving on NIXL) and KServe (the broader CNCF model-serving CRD and autoscaling layer). They provide fleet-level disaggregation, prefix-aware routing and SLO-based autoscaling across replicas, subject to the integrations and versions you select.

Two 2026 shifts changed that layer. First, NIXL stopped being NVIDIA-only plumbing: AMD merged its RIXL fork upstream in June 2026 and deprecated it, and DDN's storage plugin ships in the standard NIXL 1.3 wheel. The vendor-locked piece is now TensorRT-LLM, not the KV transport. Second, KV offload reached object storage as a shipping recipe. vLLM/LMCache park KV blocks in Azure Blob over NIXL, and Microsoft measured up to 2.8× lower TTFT versus recompute and up to 7.3× faster model cold-loads via Dynamo ModelExpress (vendor-measured, AKS). The cache transport is privileged network surface: LMCache's multiprocess transport allowed unauthenticated remote code execution when bound to a routable address (CVE-2026-105192). It had no fix at disclosure on 7 October 2026; LMCache 0.5.6 (15 October 2026) fixes it, so upgrade and bind the transport only to localhost or to an internal network you fully trust (Chapter 11.6). The hierarchy itself stays canonical in Chapter 9.7.

Every lever in this chapter resolves to one number the business cares about: $/token at the SLO. Serving efficiency governs inference unit economics because the denominator of $/token is goodput: tokens from requests that cleared every quality and latency limit. The numerator includes the cost of every provisioned GPU, plus power, fabric, standby, KV movement and operating service over the same window. At unchanged full provisioned cost, an illustrative 30% gain in accepted output cuts unit cost by 1−1/1.30, about 23%. Include extra compute, reserve and operating cost before taking that gain. Higher accepted output lowers $/token only if total cost grows more slowly, and extra decode replicas or a draft model can change the numerator as well as the denominator. Precision is often the cheapest goodput lever: FP8 or NVFP4 weights and an FP8 KV cache cut the bytes each decode step moves and fit more sequences per GPU, but they count only once quality holds on your own tasks (Chapter 7.10). Deloitte forecast that inference would account for roughly two-thirds of AI compute in 2026, which makes serving efficiency a central lever in the power-bound era: more goodput per megawatt is more revenue per scarce input. The price buyers pay keeps falling. Ramp's effective token-price index fell from a March 2026 peak of $1.15 to $0.68 per million tokens on 9 September 2026, a purchase price across Ramp's business customers, not provider cost or a universal market average. Ramp's earlier cohort comparison, about $10 per million falling to about $2.50 in the year to March 2025, comes from a different dataset and method, so do not splice the two series.

Serving efficiency therefore deserves treatment as a capital decision, not a software-team detail. The measured service curve and failure headroom determine how much hardware you must own to serve a given SLO-bound demand. The engine and disaggregation choices determine your peak goodput per GPU, and the autoscaler determines how close to the knee you can safely run. Each is a lever on the $/token that ties this chapter to the economics in Chapter 1.8 and Chapter 7.11, and to the SLA framing in Chapter 12.4.