Inference & KV-Cache Storage: The New Memory Hierarchy

For a decade the storage conversation in AI was about training — feed the GPUs fast enough (ingestion), and don't lose the run (checkpointing). Inference was assumed to be storage-light: load the weights once, serve from HBM, done. That assumption is dead. The workload that now consumes roughly two-thirds of AI compute (Chapter 1.3) created a new storage tier with no precedent in the training stack: the KV-cache — the per-token attention state every autoregressive request must keep resident for as long as it is generating. Reasoning models that emit tens of thousands of decode tokens, agents that carry long histories, and RAG pipelines with multi-thousand-token system prompts have turned that state from a rounding error into the dominant consumer of the most expensive memory in the building.

This chapter treats the KV-cache as what it has become: a memory-hierarchy problem with a storage answer. We walk the hierarchy from HBM down through local DRAM, CXL-expanded DRAM, NVMe, Ethernet-attached flash, and object storage; we make the present-tense case for CXL as a real tier (and the rule for when it beats NVMe); and we cover the KV offload-and-reuse machinery — Dynamo/KVBM, NIXL, LMCache, prefix caching — plus the model-weight-serving and cold-start problem that rides the same hierarchy. Every tier you add buys capacity and reuse at the price of latency on the miss path, and the wrong placement shows up directly as a blown time-to-first-token SLO.

Start with the arithmetic. The KV-cache stores two vectors (a key and a value) per token, per layer, per attention head. The bytes scale linearly with context length and batch size, and they must sit in the same HBM as the weights. The numbers are large even for mid-size models: a single Llama 3 70B request at 128K context needs roughly 42 GB of KV-cache — more than half of an 80 GB H100, for one user. Llama 3.1 405B burns about 516 KB per token of context; at long context and modest batch the cache routinely exceeds the model itself. Past ~128K tokens the KV-cache dominates HBM; at 1M tokens it can eat 70–90% of GPU VRAM (DigitalApplied / Spheron KV-cache analyses, 2026).

That collides with a hard ceiling. HBM is the scarcest, most expensive memory in the system and the binding supply constraint on the whole industry (entire 2026 HBM production sold out; Chapter 16.2). You cannot buy your way out by stacking more HBM — there is none to buy, and it would cost more than the GPU. So the cache must spill, and the only question is where. Architectural compression (GQA, MLA, FP8/FP4 KV quantization) buys a 4–40x reduction and is the first line of defence — but it changes the model, not the infrastructure, and it does not eliminate the spill for long-context, high-concurrency serving. Once you accept the spill, you are no longer doing memory management; you are doing tiered storage, on the request's critical path, with a latency budget measured in milliseconds.

The new memory hierarchy is a ladder of declining cost and rising latency. Each rung holds colder, larger, cheaper KV state than the one above it. The engineering job is to keep the hottest, most-reused blocks high and let everything else fall — and to size the bandwidth between rungs so that a promotion on a hit does not blow the budget the hit was supposed to save.

NVIDIA's BlueField-4 Context Memory Storage Platform (CMX, formerly ICMSP) formalized this ladder into named tiers — G1 (GPU HBM), G2 (host CPU DRAM), G3 (node-local NVMe), a new G3.5 (an Ethernet-attached flash tier optimized specifically for KV-cache), and G4 (external durable NVMe) — and it is the clearest sign that the industry now treats KV-cache placement as a storage-architecture decision, not an inference-engine implementation detail (NVIDIA / Blocks & Files, 2026). The table below is the decision surface.

The hierarchy is a set of cliffs, not a smooth gradient. The two that matter most: the jump from DRAM/CXL (nanoseconds, byte-addressable, load/store) to NVMe (microseconds, block-addressable, DMA) is roughly a 1,000x latency step — that is the line between "memory" and "storage," and it is where the programming model changes. The jump from node-local to network-attached (G3 to G3.5/G4) trades single-node capacity for fleet-wide reuse, at the price of a fabric round-trip — which is why this tier only pays off when the same context is reused across many GPUs, not within one. Place a hot block one cliff too low and the latency of a hit approaches that of a miss; place a cold block one cliff too high and you have evicted a block someone else needed to make room for one nobody will reuse.

CXL stopped being a roadmap slide. By 2026 it is a deployable tier with two distinct uses that get conflated and should not be. Memory expansion adds byte-addressable DRAM to a single host beyond its DIMM-channel limit, over the PCIe physical layer with cache-coherent load/store semantics — the OS and the inference engine see more memory, full stop. Memory pooling disaggregates DRAM into a shared resource that multiple hosts (or GPUs, via the host) can attach, enabling KV blocks to be reused across servers without recomputation. Commercial CXL pools reached 100 TiB scale in 2025 with larger 2026 deployments, and Samsung's CMM-D-in-a-CXL-switch work is explicitly pitched at KV-cache offload (Samsung CMM-D white paper, June 2026; Introl, 2025).

The decisive property is that CXL keeps the cache memory-semantic. NVMe offload forces a block-mode DMA round-trip and a copy; CXL-expanded DRAM is a load away. Published results bear this out: CXL memory pools delivered ~3.8x speedup over 200G RDMA and >5x over SSD-based KV caching for inference, and offloading KV to CXL cut GPU memory usage by up to 87% while still meeting latency targets (ITME / CXL-KV research, arXiv 2026). The cost is real estate and complexity: CXL load latency (~250–600 ns) is several times host-local DRAM, the controllers and switches are a new BOM line, and pooling needs a memory-fabric topology that most halls were not built for (Chapter 8.5 for the fabric framing).

A hierarchy is inert without a manager that moves blocks across it and decides what to keep. In 2026 that machinery converged on a recognizable stack. PagedAttention (vLLM) made the KV-cache a paged, non-contiguous structure — the precondition for everything else, because you cannot offload or share a cache you cannot address in blocks. On top of it, prefix caching reuses the KV of a shared prompt prefix across requests: for RAG and agent workloads with long, repeated system prompts, the prefix's blocks stay resident instead of being recomputed per request, and vendors report serving on the order of 10x more users on the same GPU once prefix reuse and offload are combined (Spheron, 2026).

The transport layer is where the storage and networking disciplines fuse. NVIDIA Dynamo introduced the KV Block Manager (KVBM), which decouples KV memory management from the inference engine and orchestrates movement across G1–G4; it uses NIXL (the NVIDIA Inference Transfer Library) as the unified transport across NVLink, RDMA NICs, and GPUDirect Storage, and integrates LMCache for reuse and eviction. The reported wins are large: Dynamo 1.0 cited ~7x throughput on reasoning/MoE workloads, and NIXL+GDS demonstrated ~10x faster prefill for large-context scenarios by streaming KV from storage instead of recomputing it (NVIDIA GTC 2026; Spheron/LMCache, 2026). Open alternatives — LMCache standalone, Mooncake-class disaggregated KV pools, llm-d — chase the same pattern. The architectural consequence is that KV reuse pulls you toward prefill/decode disaggregation: separate the compute-bound prefill pool from the bandwidth-bound decode pool, generate KV once in prefill, and transfer it to decode over the fabric — which makes the KV-transfer path a primary fabric-sizing input, not an afterthought (Chapter 8.5; Chapter 9.3 for the GPUDirect Storage data path).

The KV-cache is not the only thing riding this hierarchy. Multi-model and autoscaled inference fleets must also load weights onto GPUs that were just freed — and the same tiers govern how fast that happens. A 70B model in FP8 is ~70 GB; loading it from a slow object store across the network is a multi-minute stall during which a freshly-scaled GPU earns nothing. This is the cold-start tax, and it is the mirror image of the KV problem: where KV is about keeping per-request state warm, weight serving is about keeping per-model state reachable fast enough to scale.

The placement logic is the same ladder applied to weights. The hot path keeps the working set of frequently-served models on node-local NVMe (G3) or a fast network-flash tier so a scale-up event is a fast block read, not a cold object pull; the cold path keeps the long tail of rarely-served models and durable artifacts in object storage (Chapter 9.6, which treats object as the inference cold tier and model-distribution backbone). The consequence is concrete: under-provision the weight-serving tier and your autoscaler's response time is gated by storage, not by GPU availability — you scale slower than your traffic spikes, breach SLOs during the ramp, and over-provision idle GPUs to compensate. The KV hierarchy and the weight hierarchy are the same physical media, contending for the same bandwidth, and they must be sized together.

With the tiers, the transport, and the workloads in place, the remaining decision is policy: what gets promoted, what gets evicted, and where each block lands. This is where most of the realized goodput is won or lost, because the hardware ladder only sets the ceiling — the policy determines how close you get to it.

Three policy axes matter. Eviction: LRU is the default, but reuse-aware policies that weight by prefix-sharing frequency keep high-fanout system prompts resident far longer than a naive recency scheme would, which is the whole point of prefix caching. Placement: the manager must route each block to the tier whose latency matches its expected reuse interval — the CXL-vs-NVMe rule above, applied per block rather than per fleet. Coherence and sharing: a pooled or networked tier (CXL pool, G3.5 Ethernet-flash) lets many GPUs reuse one copy of a context, which is enormously efficient for shared prefixes but introduces a consistency and lifetime-management problem that node-local caches never had. Get the policy wrong and the symptoms are unambiguous: thrashing between tiers (blocks promoted and evicted before reuse), cache pollution (cold blocks crowding out warm ones), or a TTFT tail that tracks the slowest tier instead of the fastest hit. The fleet-level sizing of these tiers — how much DRAM, CXL, NVMe, and network-flash per GPU, and the bandwidth between them — is the storage:compute co-design problem taken up in Chapter 9.8.

The recurring mistakes all come from importing a training-era storage mental model into an inference fleet, or from treating the KV-cache as an implementation detail instead of an architecture:

• Recompute-everything by default. Disabling or under-sizing KV offload because "storage is slow" — then paying full prefill FLOPs on every shared-prefix request. For RAG and agent workloads this is the single largest source of wasted inference compute, and it scales with how good your traffic is (more repeated prompts = more waste).
• One tier too low for the reuse rate. Parking warm, millisecond-reused KV on NVMe to save DRAM, and discovering that the ~100 µs fetch blew the TTFT SLO. The fix is the CXL-vs-NVMe rule — match tier latency to reuse interval — not buying more GPUs.
• Sizing the weight-serving tier as an afterthought. Provisioning HBM and fabric for steady state, then watching the autoscaler stall on cold model pulls from object storage during every traffic spike. Cold-start is a storage-bandwidth problem; size G3 for it or scale slower than your traffic.
• Shared KV across an untrusted boundary. Turning on cross-request prefix sharing or session persistence for the throughput win without scoping it to a trust boundary — converting a performance tier into a multi-tenant data path. → Chapter 11.5.