Storage in the AI Lifecycle: Why It Determines GPU Efficiency

The accelerator is the most expensive depreciating asset in the building, and storage exists to keep it from sitting idle. Storage is a GPU-efficiency problem, not a capacity problem. When the data path cannot feed the accelerators fast enough they stall, and a stalled GPU burns depreciation and energized megawatts to produce nothing. What matters is goodput, the share of wall-clock time that advances the job rather than waiting on I/O or recovering from a failure, not how many petabytes you can store. → Chapter 14.1.

The rest of Part 9 rests on refusing the most common mistake in storage design: treating "AI storage" as one thing to procure. It is four I/O personalities that share the same flash and the same fabric while pulling in opposite directions: high-throughput sequential ingestion, bursty large-sequential checkpointing, metadata-heavy many-small-files access, and transfer-time-critical KV-cache offload for inference. Each wants something different (throughput, write bandwidth, metadata ops or prefix transfer time), and shared infrastructure has to meet their demands at the same time.

The unit economics are unforgiving. A debt-financed GPU cluster's break-even utilization comes from its realized GPU-hour price, its financing terms, and its fixed power and facility cost. A quoted market crossover is one scenario with one set of those inputs, and Chapter 1.8 derives the financing crossover itself. What matters in this chapter is the denominator. Break-even is measured in billable, productive GPU-hours, and a cluster can be fully booked and still be paid for time its accelerators spend waiting on I/O.

Storage is one of the few subsystems that can change how much paid accelerator time produces useful work without anyone touching a GPU. Underfeed the accelerators with a naive data path and a vision-training job waits on samples while its most expensive assets depreciate. Feed the same job with a sharded, sequential, prefetched pipeline and the prefetch runs ahead of the step instead of the accelerators waiting on it. Measure accepted training progress and batch wait, and keep recovery losses in the same window. Count the pipeline's cost against the gain: extra throughput helps pay the cluster's debt only when it produces useful steps, because device activity alone can hide retries, padding and replay.

Failure is the second lever. At scale, hardware fails constantly: Meta's Llama 3 405B run on up to 16,384 H100s logged 419 unexpected interruptions (of 466 total) over 54 days, a mean time to interrupt of roughly 186 minutes, and a synchronous training job must restart from its last checkpoint when any node drops. Storage makes that restart cheap or catastrophic. If checkpointing is slow, you checkpoint less often, lose more work per failure, and bleed goodput. If recovery reads are slow, every restart stalls the whole cluster. The subsystem therefore does two jobs at once: feeding the GPUs in steady state and insuring the run against the inevitable interruption. The two have completely different I/O signatures, and that split seeds the four-personality framing.

"AI storage" cannot be procured as one system, because the AI lifecycle generates four workloads whose I/O profiles contradict one another. They share infrastructure (the same parallel filesystem, the same flash, often the same fabric), but tuning the shared substrate for one de-optimizes it for the others. Hold these four in mind as the lens for every chapter that follows.

1. Ingestion (training-data reads) is high-throughput, large-sequential, read-dominated and bandwidth-bound. The data loader streams shards of training examples to the GPUs every step, and all it wants is sustained read GB/s. It tolerates latency because you prefetch ahead of need, and it does not care about write performance. The failure mode is a CPU-side or storage-side bottleneck that cannot sustain the per-GPU read target, which directly caps GPU utilization. → Chapter 9.5.

2. Checkpointing (state writes) is bursty, large-sequential, write-dominated, and tolerant of latency but not of duration. Every checkpoint interval the whole cluster pauses (or, done right, overlaps a brief stall) to flush model weights plus optimizer state, about 14 bytes per parameter, to durable storage. It is the inverse of ingestion: it wants peak write bandwidth in short bursts that finish fast, so the GPUs get back to computing. NVIDIA's reference tiers write at about half their read rate (124 against 250 GB/s per B300 scalable unit at the Enhanced tier), but that ratio is a comparison point, not this job's write requirement. Checkpoints must drain inside their permitted window, so size writes from the unique state and that deadline. → Chapter 9.4.

3. Many-small-files / metadata (preprocessing, LOSF) is the personality that breaks naive systems. Datasets of millions of tiny files (individual images, JSON records, audio clips) turn a bandwidth problem into a metadata problem: opens, stats, lookups and directory operations that hammer the filesystem's metadata service rather than its data path. This is the classic Lots-Of-Small-Files (LOSF) problem. It is where a metadata service sized or placed for bandwidth workloads saturates first, and where spreading metadata across many servers (or removing the small files) earns its keep. Throughput benchmarks tell you nothing about this personality; metadata ops/sec is the relevant number. → Chapter 9.2.

4. KV-cache offload (inference) is the newest personality, and it does not fit the training-only mental model. Long-context, reasoning and agentic inference generate enormous key-value caches that must be retained, retrieved and reused across requests. What binds is how fast a reused prefix gets back into HBM, because it has to arrive before the time-to-first-token budget runs out. A 128k-token Llama-3.1-70B prefix (GQA, BF16) holds about 43 GB of KV, so moving it is limited chiefly by bandwidth and transfer size, and tail latency adds to that. Reuse only pays when loading the cache is faster than recomputing the prefill. KV-cache offload to NVMe or Ethernet-attached flash is a memory-hierarchy problem, and it is why inference is now a first-class storage workload rather than an afterthought. → Chapter 9.7.

These personalities are not a menu. Most facilities run several at once on shared hardware, so the work is placement and tiering: route each personality to the tier that fits its binding metric, and isolate the ones that would otherwise collide. The most consequential collision in the building is checkpoint-write incast hitting ingestion reads and, worse, the training collectives if checkpoints land on the back-end fabric. The fix is a local-NVMe fast tier that absorbs the checkpoint burst and drains it asynchronously, for cases where direct durable writes cannot fit the save budget. A fabric separate from the one carrying all-reduce buys isolation. Converging them saves ports only if reads, drains and collectives still hit their throughput and latency targets together. A checkpoint staged on local NVMe survives only the failures that leave its node intact, so placement becomes a co-design problem across storage and network. → fabric isolation in Chapter 8.5; checkpoint mechanics in Chapter 9.4.

Storage in an AI facility is a hierarchy, not a single pool, and each tier exists because the one above it is too expensive or too small to hold everything. The decision at each boundary has the same shape: how much cost-per-terabyte do you pay to gain bandwidth and drop latency? From fastest to cheapest:

• Node-local NVMe scratch is the fastest tier and the only one that needs no network, because it sits inside the GPU server. It is the natural home for the checkpoint fast tier (absorb the burst locally, drain asynchronously) and for hot data-loader caching. It is scratch: not durable, not shared, wiped on reschedule. PCIe 5.0 remains an option for drives and backplanes, and Micron announced mass production in February 2026 of its 9650, a PCIe Gen6 enterprise SSD, with E1.S and E3.S form factors. Test the drive, backplane, firmware and cooling together before counting on Gen6 bandwidth at the node. → Chapter 9.3.
• Shared inference-context tier: Ethernet-attached flash where a pod shares KV-cache for reuse. NVIDIA calls it G3.5 and describes its CMX platform on BlueField-4 STX as the first rack-scale implementation. In March 2026 NVIDIA said partners would offer STX-based platforms in the second half of 2026; as of mid-October 2026 none had announced general availability. Size it by prefix transfer time, not capacity. → Chapter 9.7.
• Networked parallel or distributed filesystem (the primary hot tier) is the all-flash, NVMe-native, RDMA-attached shared namespace that feeds the whole cluster: WEKA, VAST, DDN EXAScaler/Lustre, IBM Storage Scale/GPFS. Ingestion reads and durable checkpoints land here, and the throughput-versus-metadata tradeoff is fought here. → Chapter 9.2.
• Object capacity tier (the data lake) is S3-compatible and increasingly QLC-flash-backed rather than HDD. It holds the full corpus, dataset versions and cold checkpoints, costs less per TB at lower bandwidth, and is the backbone for model distribution and the inference cold tier. → Chapter 9.6.
• Archive is the cheapest, coldest tier, for retention, lineage and compliance copies. Retrieval latency runs from minutes to hours, and nobody minds because nothing in the training loop touches it.

The distinction that matters more than the tier names is scratch vs durable vs archive. Scratch (local NVMe, ephemeral) is fast and cheap per IOP but loses data on failure: right for caches and checkpoint staging, fatal for anything you cannot reconstruct. Durable storage (the parallel FS, replicated or erasure-coded) survives failures and holds your insurance copies. Archive is durable, cheap and slow. Misclassify data across these, for example by treating the local-NVMe checkpoint stage as durable and skipping the async drain, and a node failure takes your checkpoint with it, which defeats the point of checkpointing.

Just as the workload archetype cascades into the whole facility in Chapter 1.1, the dominant I/O personality cascades into the storage build, and the cascade is causal. The personality sets the binding metric; the binding metric sets the media and filesystem; those set the fabric and placement; and that sets the cost structure and the failure blast radius.

Ingestion-dominant (large LLM/vision pre-training) demands sustained read bandwidth: an all-flash parallel FS sized to the per-GPU read target, fronted by local-NVMe caching, with object capacity behind it. The fabric carries large-sequential reads that prefetch ahead of need, so latency is forgiving but bandwidth is everything.

Checkpoint-dominant (frontier-scale synchronous training) turns on write bandwidth and fast drains: a local-NVMe fast tier to absorb the burst, async/tiered checkpointing to overlap it with compute, and isolation of checkpoint incast from the training collectives, so the insurance system does not sabotage the thing it insures.

Many-small-files-dominant (multimodal preprocessing, classic vision datasets) calls for distributed metadata and sharded data formats. The fix is often upstream of storage entirely: repackage millions of tiny files into a few large shards (WebDataset, TFRecord, Parquet, MDS), so the metadata problem evaporates and the workload becomes a clean sequential-read personality again. AWS recommends shards of roughly 100 MB to 1 GB when loading from S3. In its own S3 loading test, sequential tar shards kept the GPU near 100% busy, while random per-object reads starved it until the loader ran more than 16 workers. This is the cheapest big win in the data path. → Chapter 9.5.

KV-cache-dominant (long-context / agentic inference) needs a new memory hierarchy entirely (HBM, DRAM, NVMe and Ethernet-attached flash), tuned for how fast prefixes load and how often they are reused, not for bulk capacity. This personality barely existed in the training-only mental model and now reshapes inference fleet economics. → Chapter 9.7.

The last decision in this chapter can outrank all the others before the storage tier is even specced: whether your data is movable on the program's budget and schedule. At petabyte scale it usually is not, for economic rather than physical reasons. Cloud egress runs ~$0.087–0.11/GB at the 2026 first-tier list price (negotiated rates lower), tapering toward ~$0.05/GB at the deepest volume tiers. Moving a ~220 TB image download of LAION-5B costs ~$14,800 at AWS's published tiers, every time. At the ~$0.05/GB deep-volume rate, one complete read of a 5% slice of a 50 PB corpus moves 2.5 PB and costs about $125,000. Annual cost is that per-read figure multiplied by the number of reads, after caching, retries and selective replication, plus destination storage, requests and verification time. That bill, plus sovereign and data-residency rules, means the dataset frequently cannot be relocated to wherever the cheapest GPUs are. The dataset has gravity: it pulls compute toward itself. The tariffs and the per-petabyte arithmetic live in Chapter 9.8.

This inverts the naive build logic. Instead of "site the cluster on cheap power, then move the data to it," data gravity forces move-compute-to-data. Storage placement becomes a primary data-center siting input rather than an afterthought, and it can constrain the power-first siting hierarchy of Chapter 1.1 before the interconnection queue is ever consulted. The fork is to replicate the dataset to multiple sites and eat the storage cost, or to anchor compute at the data and eat the power-and-siting cost. It is one of the highest-leverage strategic decisions in the whole build, and Chapter 9.8 treats it in full, with the multi-site and egress economics.

Start from the workload list in Chapter 1.7 (feed rate, metadata load, recoverable state, KV transfer time, bytes retained) and pick the fewest tiers that cover it. One shared system is enough when it carries all of those peaks at once and can still restore a checkpoint after losing a node or a drive. Split the tiers when checkpoint incast slows ingestion reads or the training collectives, or when a restore would overrun its recovery window. Buy by petabytes alone and the accelerators starve; give every light workload its own tier and you pay for ports, power and support that buy no measurable gain.