Inference & KV-Cache Storage: The New Memory Hierarchy

For a decade, storage in AI meant training: feed the GPUs fast enough (ingestion) and don't lose the run (checkpointing). Inference was assumed to be storage-light: load the weights once, serve from HBM, done. That assumption is dead. Inference is the workload Deloitte’s November 2025 outlook forecasts at roughly two-thirds of 2026 AI compute (Chapter 1.3), and it created a storage tier with no precedent in the training stack: the KV-cache, the per-token attention state every autoregressive request must keep resident while it generates. Reasoning models that emit tens of thousands of decode tokens, agents that carry long histories, and RAG pipelines with multi-thousand-token system prompts have turned that state from a rounding error into the dominant consumer of the most expensive memory in the building.

This chapter treats the KV-cache as a memory-hierarchy problem with a storage answer. It walks the hierarchy from HBM down through local DRAM, CXL-expanded DRAM, NVMe, Ethernet-attached flash and object storage. It makes the present-tense case for CXL as a deployable tier and for when it beats NVMe, then covers the offload-and-reuse machinery (Dynamo/KVBM, NIXL, LMCache, prefix caching) and the model-weight-serving and cold-start problem that rides the same hierarchy. Every tier you add buys capacity and reuse at the price of latency on the miss path, and the wrong placement shows up directly as a blown time-to-first-token SLO.

Start with the tensor layout. Logical KV bytes for a dense attention cache are 2 × layers × KV heads × head dimension × bytes per element × retained tokens × sequences. The leading two counts keys and values. Grouped-query attention uses the KV-head count, not the query-head count. The result is aggregate state across all participating ranks. Llama 3.1 405B, for example, has eight KV heads of dimension 128 across 126 layers, so a BF16 cache holds 504 KiB for each retained token. Derive each rank's share from tensor, pipeline and context parallelism, including any replicated KV heads. Then add allocator, alignment and layout overhead and compare the total with the HBM left after local weights and workspaces. The worked prefix case below uses Meta's Table 3 dimensions and an assumed precision, because model parameters alone cannot determine KV bytes.

HBM is scarce and expensive, yet an aggregate total cannot by itself show that a deployment must spill. Only the per-rank comparison above settles that, once data parallelism is counted as well. If the allocation does not fit the service's batch, context and latency limits, the options include architectural compression (GQA or MLA), FP8 or FP4 KV quantization, lower concurrency, recomputation, or offload to a slower tier. Compression can be large: DeepSeek-V2 reported a KV cache 93.3% smaller (about 15×) than its DeepSeek-67B baseline, from MLA plus roughly 6-bit KV quantization. Offload is then a tiered-storage decision on the request's critical path, so measure transfer bandwidth, reuse and tail latency for your workload.

The memory hierarchy is a ladder of falling cost and rising latency. Each rung holds colder, larger, cheaper KV state than the one above it. The job is to keep the hottest, most-reused blocks high, let the rest fall, and size the bandwidth between rungs so that promoting a block on a hit does not spend the budget the hit was supposed to save.

NVIDIA's BlueField-4 Context Memory Storage Platform (CMX, formerly ICMSP) turned this ladder into named tiers: G1 (GPU HBM), G2 (host CPU DRAM), G3 (node-local NVMe), a new G3.5 (an Ethernet-attached flash tier built specifically for KV-cache) and G4 (external durable NVMe). It is the clearest sign that the industry now treats KV-cache placement as a storage-architecture decision, not an inference-engine implementation detail (NVIDIA / Blocks & Files, 2026). NVIDIA introduced STX, the reference architecture that contains CMX, in March 2026 and said partner platforms would ship in the second half of the year. None had announced general availability as of mid-October 2026, so evaluate a partner system once one ships. The table below is the decision surface.

The hierarchy is a set of cliffs, not a smooth gradient. Two matter most. The first is the jump from DRAM/CXL (nanoseconds, byte-addressable, load/store) to NVMe (microseconds, block-addressable, DMA). Memory access gives way to queued storage I/O and bulk transfer, which is the line between "memory" and "storage" and the point where the programming model changes. The second is the jump from node-local to network-attached (G3 to G3.5/G4). It trades single-node capacity for fleet-wide reuse at the price of a fabric round trip, and it can buy either shared reuse or capacity beyond one node, so measure both the compute it saves and the transfer contention it adds before choosing it. Place a hot block one cliff too low and the latency of a hit approaches that of a miss. Place a cold block one cliff too high and you have evicted a block someone needed to make room for one nobody will reuse.

One new rung is worth watching: High Bandwidth Flash (HBF), which stacks NAND on the accelerator package and connects over UCIe. Sandisk and SK hynix published the OCP HBF base-die specification (v0.7.0) in August 2026. It describes 512 GiB cubes with user-bandwidth grades from 0.384 to 3.072 TB/s, which are design targets, not measured devices. Sandisk reports a first die taped out, with memory samples planned for the second half of 2026 and samples of the first inference devices carrying HBF in early 2027. NAND wears with every write, so HBF is a better home for read-mostly state, such as weights and shared prefixes, than for write-heavy decode KV. Nothing deployable exists yet.

By 2026 CXL is a deployable tier with two distinct uses that get conflated. Memory expansion adds byte-addressable DRAM to a single host beyond its DIMM-channel limit, over the PCIe physical layer with cache-coherent load/store semantics. The OS and the inference engine simply see more memory. Memory pooling disaggregates DRAM into a capacity resource that many hosts draw allocations from. That is flexible assignment, not shared bytes. CXL 2.0 pooling gives each region to one host at a time. Concurrent multi-host access to the same physical KV blocks is memory sharing, a CXL 3.x capability that needs matching switch, device and software support. Settle the CXL generation and topology before you count on cross-server prefix reuse, since pooling and sharing buy very different things. CXL 4.0 (CXL Consortium, November 2025) doubles the link rate to 128 GT/s, the PCIe 7.0 rate, and adds bundled ports that gang several links between a host and an accelerator. PCIe 8.0, at 256 GT/s, reached Draft 0.5 in May 2026, with the final specification due in 2028. Neither has devices you can buy for a 2026–27 build.

XConn and MemVerge said at OCP 2025 that 100 TB commercial CXL memory pools were available in 2025, with larger deployments expected in 2026, and demonstrated a pool serving KV cache through Dynamo/NIXL (vendor statement). Samsung's CMM-D-in-a-CXL-switch work is explicitly pitched at KV-cache offload (Samsung CMM-D white paper, June 2026). Samsung now says its own AI Memory Node, a CMM-D "L3.5" tier in production at its US campus for an internal coding agent, delivers 5–7x KV-cache utilization. In its OCP 2026 demo, elastic pooling cut time-to-first-token by up to 38% and raised goodput up to 3.3x against static allocation (Samsung, October 2026; no test configuration published, and no baseline given for the 5–7x).

The decisive property is that CXL keeps the cache memory-semantic. NVMe offload forces a block-mode DMA round trip and a copy; CXL-expanded DRAM is a host load away. But that is a CPU-side property, and the consumer here is a GPU. Budget the whole path: NUMA placement, the host-to-device hop and the runtime support that has to exist. Do not read CXL load latency as the accelerator's fetch time. On Grace and Vera platforms, the CPU's own LPDDR5X over NVLink-C2C is the first spill tier, ahead of CXL. Test that path before you add a CXL tier. The published evidence for how much CXL is worth is thin. The speedup and GPU-memory-saving multipliers circulating for CXL KV pools do not trace back to a paper with a stated configuration, baseline path, workload and latency target. Treat them as unestablished and measure the fetch path on your own stack before you buy to them.

FMS 2026 added a proposed near-memory rung. SK hynix's CMM-Ax device, driven by Marvell's Structera A controller, moves overflow KV into CXL DRAM and computes attention there. The two companies project decode throughput up to 5.5× that of one GPU, and 3.6× that of two, for Llama-3-8B at 1M tokens. They scaled the result from a 32 GB prototype up to a projected 512 GB device, and the figures are a projection, not a measurement on a 512 GB device (Marvell and SK hynix, 2026-08-05). Marvell also proposed a pod-level optical shared-memory tier ("Photonic Fabric": up to 32 TB of warm KV at 50 m reach), a watch item and not a deployable 2026 SKU.

For scale on the Ethernet-flash rung, a Vera Rubin SuperPOD-scale CMX complex works out to about 9.6 PB of flash. That is Blocks & Files arithmetic: 16 enclosures × 4 BlueField-4 × the ~150 TB per BlueField-4 that Jensen Huang cited at CES 2026, not a datasheet value. It is why the KV tier is now a NAND-procurement line as well as a latency class. CXL has its own cost. Its load latency (~250–600 ns) is several times that of host-local DRAM, the controllers and switches are a new BOM line, and pooling needs a memory-fabric topology that most halls were not built for (Chapter 8.5 covers the fabric). OCP's Composable Memory Systems project targets non-blocking pooled CXL at under 400 ns round trip. Its Redfish-based fabric-management architecture for composing pooled regions is a Seagate contribution (v1.1, November 2024). Ask a pool vendor whether its fabric manager exposes those APIs or only a proprietary console.

One token takes 2 × 80 × 8 × 128 × 2 = 327,680 bytes aggregate. A full sequence therefore occupies 327,680 × 131,072 = 40 GiB aggregate, and ideal head sharding gives 5 GiB/rank. With 20% overhead, each full sequence needs 6.0 GiB/rank, so floor(60/6.0) = 10 sequences fit. Eleven would need 66 GiB and fail on capacity before any throughput claim. Reserve growth beyond the retained-token ceiling separately.

The reusable prefix occupies 327,680 × 8,192 = 2.5 GiB aggregate, or 320 MiB/rank = 0.33554432 GB/rank. One fetch takes 0.33554432 GB / 20 GB/s + 0.0050 s ≈ 22 ms. That is within the 50 ms budget and faster than the assumed 150 ms recompute, so fetch this authorized prefix and keep the budget for its uncached suffix. The result justifies a transfer choice, not a user-count multiplier.

Flip: two simultaneous promotions need 2 × 0.33554432/20 + 0.0050 ≈ 39 ms and pass. Three need about 55 ms and fail. The crossover for three is 3 × 0.33554432/(0.050 − 0.0050) ≈ 22 GB/s, so the assumed 20 GB/s path admits at most two promotions under this sharing model. Schedule earlier prefetch, cap simultaneous misses or add delivery capacity; recompute at 150 ms cannot rescue this deadline. Test queue behavior on all participating ranks at that admission limit, then hand the allowed concurrency to Chapter 1.3 and the transfer demand to Chapter 8.5.

Which tier can deliver 20 GB/s per rank? Where an x86 host's GPUs attach at PCIe 5.0 x16, as on H100- and H200-class HGX boards, both CXL memory and GPUDirect NVMe reach the GPU over that one link: about 63 GB/s in each direction before protocol overhead, so assume 50 GB/s delivered. At about 14 GB/s per Gen5 drive, NVMe needs two drives per rank to meet this case's 20 GB/s and four to fill the link. At 50 GB/s the 320 MiB/rank prefix moves in about 6.7 ms from either tier, plus 10–100 µs of flash access or well under a microsecond of CXL load latency. For a prefix this size the link sets the time, and the media gap is under 2% of the fetch. CXL pulls ahead on small blocks fetched at high request rates, where NVMe runs out of IOPS, and on write churn heavy enough to wear out flash. Eight ranks fetching at once need 160 GB/s from the tier: about a dozen Gen5 drives, but a fraction of one host's DDR5 bandwidth, which is why host DRAM is the first spill tier ahead of both. A GPU with a PCIe 6.0 host link can take GPUDirect NVMe faster than this. CXL and host memory, though, still pass through the CPU, so behind a CPU whose root ports run at PCIe 5.0 their ceiling does not move.

A hierarchy does nothing without a manager that moves blocks across it and decides what to keep. In 2026 that machinery converged on a recognizable stack. PagedAttention (vLLM) made the KV-cache a paged, non-contiguous structure, the precondition for everything else, because you cannot offload or share a cache you cannot address in blocks. On top of it, prefix caching reuses the KV of a shared prompt prefix across requests. For RAG and agent workloads with long, repeated system prompts, the prefix's blocks stay resident instead of being recomputed per request. The saved prefill work can increase service capacity only while prefix transfer, suffix computation and decode stay within the admitted workload's SLO.

The transport layer is where the storage and networking disciplines fuse. NVIDIA Dynamo introduced the KV Block Manager (KVBM), which decouples KV memory management from the inference engine and orchestrates movement across G1–G4. It uses NIXL (the NVIDIA Inference Transfer Library) as the unified transport across NVLink, RDMA NICs and GPUDirect Storage, and integrates LMCache for reuse and eviction. Dynamo, NIXL and GDS sit in different parts of the serving stack: the manager decides KV residency, the transfer layer moves buffers, and the execution engine defines the compatible KV layout and consumes it. Pin all three releases and measure the complete request path on the model you will serve. A GB200 NVL72 versus B200 comparison that also changes expert parallelism cannot isolate a storage-reuse gain, and neither can a prefill multiplier quoted without its conditions.

NIXL stopped being NVIDIA-only plumbing in mid-2026. NIXL 1.3 (Jul 2026) ships DDN's Infinia plugin inside the standard wheel and the prebuilt Dynamo container, the first storage vendor in the official package. AMD's RIXL fork was merged upstream on 2026-06-04 and deprecated, after validating end-to-end VRAM transfers on MI300X/MI355X (SemiAnalysis). The transfer library is becoming the inference data-movement ABI that storage vendors and the second GPU vendor target, which softens the certified-appliance lock-in calculus in Chapter 9.2. Open alternatives (LMCache standalone, Mooncake-class disaggregated KV pools, llm-d) chase the same pattern. On flash, KV offload is a churny write load, and NVMe Flexible Data Placement (FDP) lets the host group blocks that expire together so the SSD rewrites less during garbage collection (Samsung demonstrated FDP inside LMCache in October 2026). The OCP Datacenter NVMe SSD Specification leaves FDP optional, so name it in the SSD RFQ when you buy KV-offload flash (Chapter 9.3).

The architectural consequence is that KV reuse pulls you toward prefill/decode disaggregation: separate the compute-bound prefill pool from the bandwidth-bound decode pool, generate KV once in prefill, and transfer it to decode over the fabric. That makes the KV-transfer path a primary fabric-sizing input, not an afterthought (Chapter 8.5; Chapter 9.3 for the GPUDirect Storage data path).

The KV-cache is not the only thing on this hierarchy. Multi-model and autoscaled inference fleets must also load weights onto GPUs that were just freed, and the same tiers govern how fast that happens. A 70B model in FP8 is ~70 GB, and loading it from a slow object store across the network is a multi-minute stall during which a freshly scaled GPU earns nothing. This is the cold-start tax, and it mirrors the KV problem: KV is about keeping per-request state warm, while weight serving is about keeping per-model state reachable fast enough to scale.

The placement logic is the same ladder applied to weights. The hot path keeps the working set of frequently served models on node-local NVMe (G3) or a fast network-flash tier, so a scale-up is a fast block read, not a cold object pull. The cold path keeps the long tail of rarely served models and durable artifacts in object storage (Chapter 9.6, which treats object as the inference cold tier and model-distribution backbone). If you under-provision the weight-serving tier, storage sets your autoscaler's response time, not GPU availability. You scale slower than your traffic spikes, breach SLOs during the ramp, and over-provision idle GPUs to compensate. The KV hierarchy and the weight hierarchy are the same physical media contending for the same bandwidth, so size them together.

With the tiers, the transport and the workloads in place, the remaining decision is policy: what gets promoted, what gets evicted, and where each block lands. Policy decides most of the realized goodput, because the hardware ladder only sets the ceiling. Three axes matter. Eviction: LRU is the default, but reuse-aware policies that weight by prefix-sharing frequency keep high-fanout system prompts resident far longer than a naive recency scheme would, which is the whole point of prefix caching. Placement: the manager must route each block by modeled fetch time, hit probability and prefetchability against the remaining TTFT and recompute budgets. That is the CXL-versus-NVMe rule above, applied per block rather than per fleet. Coherence and sharing: a pooled or networked tier (CXL pool, G3.5 Ethernet-flash) can extend capacity or let authorized GPUs reuse compatible context. CXL pooling by itself allocates capacity and does not establish simultaneous sharing. Genuine sharing is enormously efficient for shared prefixes, but it brings a consistency and lifetime-management problem that node-local caches never had.

Get the policy wrong and the symptoms are unambiguous: thrashing between tiers (blocks promoted and evicted before reuse), cache pollution (cold blocks crowding out warm ones), or a TTFT tail that tracks the slowest tier instead of the fastest hit. Sizing the tiers across the fleet (how much DRAM, CXL, NVMe and network flash per GPU, and the bandwidth between them) is the storage:compute co-design problem taken up in Chapter 9.8.

Key reuse by the effective model/adapter revision, token IDs, position convention, KV dtype/layout and authorized sharing group. Prefix identity must include the preceding context; equal trailing tokens alone are not enough. On your runtime, issue an allowed repeated prefix and verify both reuse and equivalent output. Change the model revision, adapter, token prefix or tenant sharing key and require a miss. Revoke a session and confirm its cached state stops being served under your retention policy. vLLM prefix-cache identity and salting is one implementation; test the configured behavior rather than assuming a cache hit is permission.

Vectors occupy 100,000,000 × 768 × 4 = 307.2 GB, so one index needs 307.2 + 64 + 32 = 403.2 GB. Two replicas with an old/new rebuild need 403.2 × 2 × 2 = 1,612.8 GB, and at 80% occupancy you provision 1,612.8/0.8 ≈ 2.0 TB. The 2.1 TB reservation passes this footprint. Flip: 1,536-dimension vectors occupy 614.4 GB, so the rebuild needs (614.4 + 64 + 32) × 4/0.8 ≈ 3.6 TB and fails on capacity. At the same allowances the 2.1 TB reservation holds at most about 810 dimensions at FP32, or about 4.2 bytes/element at 768 dimensions. FP16 storage or product quantization moves the other way, and any precision choice or index compression must also preserve retrieval quality.

Choose a candidate index only after a labeled query set passes both assumed gates. Use exact search over the same authorized corpus as the filtered-recall reference, and report recall by tenant and filter class plus query latency at admitted concurrency. Pin the embedding model, normalization, distance metric, chunking and corpus version. Delete a source chunk and revoke its permission, then query every serving replica and result cache: forbidden content must stop reaching the prompt by the propagation deadline you set. Return index generation and deletion watermarks with the results. Chapter 10.10 covers which data may enter the index, and Chapter 1.3 sets the request SLO.

The recurring mistakes come from importing a training-era storage mental model into an inference fleet, or from treating the KV-cache as an implementation detail instead of an architecture:

• Recompute-everything by default. Disabling or under-sizing KV offload because "storage is slow", then paying full prefill FLOPs on every shared-prefix request. For RAG and agent workloads this is a leading source of wasted inference compute, and it grows with how good your traffic is: more repeated prompts means more waste.
• One tier too low for the transfer budget. Placing a multi-gigabyte warm prefix on NVMe based on media latency alone, then discovering that access latency, bytes divided by effective bandwidth, and queueing blew the TTFT SLO. Compare complete fetch time with the remaining TTFT budget and recompute time at admitted concurrency before buying more GPUs. Add capacity only where that comparison shows it removes the limiting resource.
• Sizing the weight-serving tier as an afterthought. Provisioning HBM and fabric for steady state, then watching the autoscaler stall on cold model pulls from object storage during every traffic spike. Cold start is a storage-bandwidth problem when weight transfer dominates, with verification, deserialization and warm-up added to the clock. Size G3 or stage weights ahead, or the fleet scales slower than its traffic.
• Shared KV across an untrusted boundary. Turning on cross-request prefix sharing or session persistence for the throughput win without scoping it to a trust boundary, which converts a performance tier into a multi-tenant data path. → Chapter 11.5.

Choose KV placement from per-rank capacity and the complete transfer deadline, with admission limits that survive simultaneous misses. Count retrieval indexes, source chunks and model weights in the same physical capacity and bandwidth budget as the KV-cache, while testing their different correctness requirements. A cache hit that arrives late, belongs to another sharing group or uses incompatible model state has no serving value. A larger tier pays only when it increases the authorized requests completed within the SLO.