NVMe Tiers, GPUDirect Storage & the CPU-Bypass Data Path

Chapter 9.2 chose the file system. This chapter chooses the data path underneath it: the physical and software route a byte takes from flash to a GPU's HBM. That path, not the file system's headline aggregate, decides whether the accelerators stay fed. A parallel file system can advertise multiple terabytes per second across a cluster and still starve an individual GPU, because the classic I/O path sends every read through the host CPU. The NIC DMAs data into a kernel bounce buffer in system DRAM, the application stages it in host memory, and the GPU's DMA engine then copies it into device memory. Each hop adds latency and uses CPU cycles and DRAM bandwidth. At the small, highly concurrent requests that dominate AI access patterns, the hops leave the PCIe bus underused while the GPU waits. When the trace shows host staging limiting delivery, the bounce buffer is the bottleneck, not the file system.

The engineering that follows is a sequence of bypasses. Each one deletes a piece of the host from the path, and each is a real fork with a downstream cost. First comes the media hierarchy: how many tiers of flash you run, where they sit, and which I/O personality each absorbs. Next is GPUDirect Storage (GDS), which takes the data path off the CPU. Then the 2026 frontier, SCADA and BlueField-4, takes the control path off the CPU too: the GPU or a DPU initiates I/O while the host CPU stays out of the per-request loop. Underneath all of it sit the silicon choices (TLC or QLC, PCIe 5.0 or 6.0, E1.S or E3.S) and the fabric question of where storage traffic rides. Each carries its cost in goodput, watts or stranded capacity.

AI storage is a stack of media, not one medium, because the four I/O personalities of Chapter 9.1 pull in incompatible directions: sequential ingest, bursty checkpoint writes, metadata-heavy small-file access and latency-critical KV-cache retrieval. No single tier serves all four economically. The hierarchy, from hottest to coldest:

Node-local NVMe scratch is the closest persistent medium to the GPU: one to a handful of NVMe drives inside the GPU server, on the host PCIe complex, adding no network traffic. It absorbs staleness-tolerant, node-private work: the local checkpoint tier (write to local flash in seconds, drain to the global store asynchronously), data-loader shuffle buffers and spill space. Its usable bandwidth is bounded by the populated drives, the PCIe tree, the controller and application concurrency, and its great virtue is that it never touches the storage fabric. Skip it and every checkpoint and shuffle becomes networked traffic that collides with training collectives. → Chapter 9.4 treats the local-NVMe checkpoint tier in depth.

Networked all-flash (the hot tier) is the parallel or distributed file system itself: WEKA, VAST, DDN/Lustre, IBM Storage Scale. It is built on TLC NVMe when write endurance and sustained service fit the job, and reached over RDMA when the file system and client support it. This tier is sized to the per-GPU read target and the checkpoint write budget, and it is where 9.2's file-system choice lives. For scale, NVIDIA's reference architectures give this tier an Enhanced performance guideline of 250 / 124 GB/s read/write for each DGX B300 scalable unit (512 GPUs in its Ethernet design) and 280 / 140 GB/s for each 576-GPU Vera Rubin NVL72 unit. Both come to about 0.49 GB/s of read per GPU. These are guidelines, not floors; Chapter 9.1 shows how to set your own target from your loader's measured rate.

QLC capacity flash (the warm tier) trades endurance and write speed for density: read-mostly datasets, model registries, and the all-flash data lake that replaces HDD for active data when its read rate and rack density justify the flash premium. The Ethernet-attached context tier is the 2026 newcomer: petabyte-scale flash addressed as inference KV-cache memory (NVIDIA CMX/ICMSP-class). It did not exist in the training-only storage stack, and Chapter 9.7 treats it as a memory-hierarchy problem, not a file-system one.

Map personality to tier, and resist collapsing them. Checkpoint writes on a QLC capacity tier whose rated host writes or full-drive drain rate fall short will burn its endurance or saturate its write path, so size the pool for the state after a host fails, and test it there. KV-cache served from the cold object tier blows the latency budget. The hierarchy answers a sizing question, not a procurement convenience, and its placement column meets the fabric decision at the end of this chapter.

GPUDirect Storage is the first and most established bypass. It sets up a direct DMA path between storage and GPU memory and removes the bounce buffer in host DRAM: the NIC (or local NVMe controller) DMAs data straight into HBM across the PCIe root complex, with no CPU-mediated copy. The host CPU still initiates the transfer. It owns the control path, issuing the I/O and managing the file system, but it is no longer in the data path. The payoff is threefold: lower latency (one fewer copy), reclaimed CPU cycles and DRAM bandwidth, and far higher effective bandwidth on the small, concurrent requests that defeat the buffered path. Published GDS results are configuration aggregates, not a promise for one GPU behind one link. A single 200 Gb/s link carries at most 25 GB/s, and that is the raw decimal line rate before any protocol overhead. Any networked result above that figure ran over several of those links, or over one faster link. Ask what drive count, link count, PCIe layout and concurrency produced it. Benchmark payload GB/s and GPU stall time end to end on your own topology.

Adopting GDS is not free, and the failure mode is concrete. It needs a supported stack end to end: a GDS-aware file-system client, a supported NIC, a clean PCIe topology, and applications that issue I/O through the cuFile API rather than ordinary POSIX read(). Put the GPU and the NIC or NVMe drive under the same PCIe switch or root complex, so the DMA does not cross the CPU's inter-socket link. If any required condition is missing, cuFile drops to compatibility-mode host staging, and current releases leave that fallback on by default (allow_compat_mode is true in cufile.json). You pay for GDS-capable hardware and get a bounce buffer. Other unsupported paths fail outright. Some file-system settings also send part of the I/O down the POSIX path on purpose, such as Lustre's posix_gds_min_kb for small requests. During acceptance testing, set allow_compat_mode to false so an ineligible path fails, and use the eligibility checks and per-process transfer counters to see which path ran. The fork is whether your file system and data loaders are GDS-native: a WEKA, VAST or GPFS deployment with a GDS client and a DALI/cuFile loader gets the bypass, while a generic NFS mount with a PyTorch loader doing buffered reads does not, whatever NICs you bought. → loader integration in Chapter 9.5.

Prove the bypass with the installed tools. Keep the gdscheck.py -p eligibility output, run a checksum-verified disposable read through GDSIO's direct and host-staged transfer modes, and collect gds_stats for that process. Confirm flags with the installed version's help. Inspect direct operations, compatibility/POSIX fallback, routing and errors, then repeat on the real loader: a GDSIO result does not prove a buffered application used cuFile. Compare delivered bytes and step wait at identical I/O sizes and concurrency. NVIDIA GDS diagnostics separate path support from path use.

For node-local scratch, choose the GDS mode before the RAID layout. Upstream P2PDMA mode (CUDA 12.8 and later, kernel 6.2 or newer on Ubuntu) drops the nvidia-fs driver and the NVMe driver patches. NVIDIA does not support it on RAID0 or multipath NVMe, or on drives formatted with end-to-end protection metadata. A striped mdraid scratch volume therefore needs the nvidia-fs mode or per-drive namespaces, or it runs staged.

GDS removed the host from the data path but left it owning the control path: the CPU still decides what to read and when. The 2026 generation removes the control path as well, by two routes that are easy to conflate and architecturally different.

SCADA (GPU-initiated I/O) moves the storage control path onto the GPU. GDS left the CPU as the initiator. SCADA lets the GPU itself submit storage requests and reap their completions, which takes the host CPU out of the per-request data and control loop. Host software still sets up access: it creates the queues, registers memory and runs the SCADA services NVIDIA supplies. For remote storage, a server built by the storage provider answers the GPU's requests. The motivating problem is inference: a GPU sustaining more than a thousand parallel threads cannot wait on a CPU to orchestrate millions of tiny reads against a multi-petabyte context dataset. At Computex 2026 Wiwynn demonstrated an early SCADA server built around a Vera CPU. It carries four ConnectX-9 SuperNICs and four RTX PRO 6000 GPUs, plus a bay of up to 96 liquid-cooled E3.S drives (PCIe 6.0, about 2.9 PB). That is a showcase, not a measured result. SCADA is, as of October 2026, still at the SDK stage. On 30 September 2026 NVIDIA made its cuObject libraries generally available and introduced a SCADA Server SDK, which lets storage providers build servers that serve GPU-issued requests over RDMA, and IBM has demonstrated a Storage Scale prototype. The public numbers are vendor demos, such as Micron's 230M IOPS at SC25: 512-byte random reads from 44 Gen6 drives into three H100 GPUs, about 118 GB/s of payload, the small-request regime SCADA exists for. No production deployment, and no measured SCADA-versus-GDS comparison, has been publicly reported.

BlueField-4 (DPU-offloaded storage) takes the other route. It moves the storage, networking and security control plane onto a DPU between the network and the GPU, so the host CPU is bypassed by offload rather than by GPU initiation. NVIDIA's BlueField-4 STX architecture, announced at GTC in March 2026, is built around a storage-optimized BlueField-4 processor that pairs a Vera CPU with a ConnectX-9 SuperNIC. It sends data through a dedicated accelerated-storage layer over RDMA on Spectrum-X Ethernet. The first BlueField-4 DPU, announced at GTC Washington D.C. in October 2025, combines a 64-core Grace CPU with ConnectX-9 for 800 Gb/s and about 6x the compute of BlueField-3. The STX processor is a separate profile. Twelve storage vendors, among them DDN, Dell, HPE, IBM, NetApp, VAST and WEKA, are building STX platforms that NVIDIA expects from partners in the second half of 2026. As of October 2026 none had been reported shipping, so treat availability and performance as vendor statements until a specific platform is delivered and measured. The same silicon underpins the CMX context-memory tier for inference KV-cache. → the inference memory hierarchy in Chapter 9.7; DPU security offload in Chapter 11.6.

Three hardware forks under the data-path software fix the per-node economics, and each maps onto a tier.

TLC vs QLC trades endurance and write bandwidth against density. TLC (3 bits/cell) has higher write endurance (DWPD) and better sustained write performance, which suits the scratch and hot/checkpoint tiers that absorb bursty, write-heavy traffic. QLC (4 bits/cell) packs ~33% more capacity per die at lower $/TB, but with materially lower endurance and weaker write throughput. That is tolerable for the read-mostly capacity, data-lake and context tiers. The mistake is picking by NAND category instead of by measurement. Compare the drive's rated sustained write bandwidth and its endurance budget, in absolute TB written per day, against what the tier will do, including write amplification and degraded-mode rebuild. Shuffle spill and small random writes fail that test on QLC; large sequential checkpoint drains onto a wide enough enterprise-QLC pool can pass it. Match the medium to the measured write pattern, and default to TLC wherever the proposed QLC design cannot clear both limits. To track those limits in service, require the OCP Datacenter NVMe SSD Specification (v2.7, 8 January 2026) in the RFQ: its physical-media-units-written counter lets you compute write amplification, and it mandates latency-monitor and telemetry logs.

PCIe 5.0 vs 6.0 sets the per-drive ceiling. A PCIe 5.0 x4 enterprise NVMe drive tops out near 14 GB/s sequential read. PCIe 6.0 doubles the lane rate, and the first mass-production Gen6 enterprise drives (Micron's 9650, in volume in 2026) reach roughly 28 GB/s read and 14 GB/s write at around 5.5M IOPS. Samsung's PM1763 entered mass production in July 2026 at 4 to 16 TB, rated up to 28.4 GB/s read and 21.9 GB/s write on the 16 TB model (Samsung), which gives Gen6 designs another drive source to qualify. The consequence is drive count, and drive count reshapes the storage shelf. A fixed per-node bandwidth target needs half as many Gen6 drives as Gen5 only if the drives you buy deliver twice the usable bandwidth and PCIe, NIC, endurance and capacity do not set a higher minimum count. A 96-drive GPU-initiated server needs its fully populated PCIe and cooling layout tested; the drive count alone does not make Gen6 necessary.

E1.S vs E3.S is the form-factor fork. E1.S (the ruler) is the dense, hot-swap format favored inside compute servers. E3.S is the higher-power, higher-capacity format common in dedicated storage nodes. E1.L, the long ruler, targets high-density 1U capacity servers; KIOXIA's first E1.L QLC drive, the LD4, is sampling to select customers at 15.36 and 30.72 TB (KIOXIA, October 2026). Liquid cooling now spans E1.S and E3.S: Micron's 9650 ships in both form factors with E1.S as the liquid-cooling target, Wiwynn's showcase uses 96 liquid-cooled E3.S drives, and KIOXIA's cold-plate CM9 E3.S drives have been sampling since October 2026. The cooling decision therefore follows the chassis, not the form factor. At Gen6 power and density, the storage shelf inherits a cooling decision of its own.

The last fork is topology: every byte from the networked tiers crosses a fabric, and which fabric is a co-design decision with the back-end compute network of Part 8. There are three placements, with three sets of consequences.

A dedicated storage rail (separate switches and NICs for storage) gives the cleanest isolation. Checkpoint-write incast and dataset-read bursts never collide with the all-reduce collectives the training job lives or dies on. Choose it when a coexistence test shows that shared resources cannot protect both the storage and the collective deadlines. NVIDIA's DGX B300 SuperPOD reference uses this placement, with a storage fabric (InfiniBand or RoCE Ethernet) separate from the compute fabric; that is one vendor's design choice, not a goodput ranking that holds for every workload. The extra NICs, switch ports, optics and power buy network separation, but shared PCIe, host memory, storage controllers or power still need a failure check. Count those additions in Chapter 8.5 rather than charging isolation as a nominal allowance.

Converging storage onto the back-end compute fabric reuses the non-blocking InfiniBand/RoCE network you already paid for and saves the dedicated rail. A checkpoint write from 16k GPUs is a synchronized incast event, though. Convergence saves ports only if the shared NIC, PCIe path, queues and target keep both workloads on deadline through that burst and through the required failure. Separate traffic classes and tuned congestion control protect allocation but create no physical capacity. Choose convergence when the concurrent test below passes; otherwise buy independent resources or reduce admitted traffic. → surviving-capacity inventory in Chapter 8.5 and congestion controls in Chapter 8.6.

A high-performance front-end/storage fabric can carry the hot path when it is provisioned for it. NVIDIA's DGX GB200 SuperPOD reference works this way: storage and in-band management share one physical Ethernet fabric, with storage on its own VXLAN segment and two of each BlueField-3's four ports reserved for it. The out-of-band management network carries BMC and recovery control. Keep it under its own name, because lumping it in with the storage fabric hides their different capacity and failure requirements.

On the rail, the transport is the final sub-fork. NVMe-oF over RDMA (RoCE or InfiniBand) and NVMe/TCP differ in latency, CPU use, congestion control and operations, but neither protocol supplies a portable tier result. Measure your target and device at the workload's I/O size, queue depth, concurrency, cache state, CPU path, network and hop count, including tail latency under failure and checkpoint incast. Choose RDMA or TCP from that end-to-end test and the throughput or latency target that governs the tier, not from the label 'GPU hot tier'. Between the two RDMA flavors, the InfiniBand-vs-RoCE decision is the one Part 8 makes for the compute fabric, and the storage version stings the same way: untuned RoCE under checkpoint incast is where tail latency blows out. Converged Ethernet has caught up where the NIC, target and congestion recipe are validated together, and Spectrum-X-class fabrics are sold on exactly that promise for GDS and KV offload. Whichever transport you pick, confirm that the vendor supports it on your NICs and target and that its recovery path passes the tests in Chapter 8.4. That means NVMe/RDMA over InfiniBand or RoCE, or NVMe/TCP across an IP path you have tested. An Ethernet NIC alone proves neither RDMA support nor NVMe target compatibility. Carry the choice into the inventory of Chapter 8.5 and the congestion testing of Chapter 8.6.

A surviving Ethernet link is only the first step in storage recovery. NVMe over Fabrics keeps its own state above the wire. Discovery finds the reachable subsystems, a fabric connection associates the host with a controller, and the host then sets up its admin and I/O queues and checks the namespace and its access state before issuing work. An NVMe submission/completion queue pair is protocol state carried by an RDMA queue pair or a TCP connection, and the two have different identifiers and fail on different clocks. A re-established TCP stream does not by itself restore the NVMe controller association, and a live RDMA queue pair does not by itself establish namespace access. Record the host identity, the target subsystem NQN, the transport addresses and the namespace identifiers, and pin the NVM Express base and transport specification revisions you support separately from the NIC's Ethernet speed or the SSD's PCIe generation.

Trace a read end to end, because recovery means the application got correct data, not that the link came back up. Along the way the host selects an accessible namespace path, submits the command, moves the data over the transport, sees a successful completion and hands the buffer over. A write completion must also meet the persistence guarantee you configured, because ordinary completion and power-loss durability are not the same thing. A file checkpoint adds flush and publication steps beyond one NVMe command, covered in Chapter 9.4. Record where buffers are staged if the direct path fails, so a reconnect does not silently change CPU load, memory demand or delivery time.

Multipath must keep the same namespace on deadline. A second target address is not a replica. Check that each path reaches the intended namespace through its own NIC, switch, controller and power feed. Linux native NVMe multipath selects paths by Asymmetric Namespace Access (ANA) state, and optimized, non-optimized and inaccessible paths are not equal candidates. Choose the policy your kernel and target support, and confirm it with nvme list-subsys and the namespace's ANA state.

Three parties own the recovery clocks. The client owns command timeout and reconnect, the target owns controller state and recovery, and the application owns the request or checkpoint deadline. Controller-loss timeout, reconnect delay, detection and any queue drain must nest inside that deadline. The nvme connect options set them, and each must match the target's supported recovery policy. Infinite retries can keep a command pending while the training or serving SLO is missed. Accept the path by breaking it with reads and writes in flight, and keep command status, timeouts, ANA transitions and data checksums from every test:

• Kill the active path. Passes if a surviving path to the same namespace completes correct I/O before the application deadline.
• Kill its controller. Passes if a surviving controller does the same; two addresses on one controller fail here.
• Remove every valid path. Passes if the client reports the failure within a bounded time instead of retrying forever.
• Restore the path. Passes if failback neither reorders application publication nor overloads the survivor.

Coexistence is tested at the shared physical bottleneck. Run dataset reads, checkpoint drain, metadata operations and repair alongside the actual collective job. Identify the common NIC, PCIe uplink, switch queue or target service. A separate VLAN or priority cannot create capacity, and separate network switches still leave shared host memory or PCIe contention. Compare per-job delivery, commit delay, metadata tails and collective step time with matched solo runs, using the workload budgets from Chapter 9.2 and Chapter 9.8. Convergence saves ports only while every foreground workload survives the required failure and repair traffic.

MLPerf Storage exercises real storage with defined workload generators (MLCommons' benchmark code); its training harness emulates accelerator computation rather than running the proposed model on that many physical GPUs. Version matters. In v3.0 (September 2026) the emulated accelerator changes from H100 to B200 or MI355, an S3 layer arrives, and two inference workloads appear: KV-cache and vector database. Only compare results within one version. Record benchmark version, workload, access API, client count, cache state and the emulated accelerator profile with every result. Neither simulated accelerator utilization nor an isolated storage score proves real collectives, decode, cuFile delivery or model-state restart. Use the published workload first for comparison, then replay the real loader, model and fault sequence on the installed stack. The first test characterizes storage; the second validates the application.

The recurring mis-builds in this layer all come from optimizing the file system's headline number while ignoring the path:

• Buying GDS-capable hardware and running buffered I/O. A supported NIC and file system mean nothing if the data loader issues ordinary read() calls and silently falls back to the bounce buffer. You pay for the bypass and get bounce-buffer latency. Validate the path, not the bill of materials.
• QLC under the checkpoint stream. Write-heavy, bursty checkpoint traffic on a cheap capacity tier can burn through its endurance or hit a sustained-write cliff just as the checkpoint must drain. TLC is a candidate for write absorption and QLC for read-heavy capacity. A QLC pool gets checkpoint duty only when its rated host writes, full-drive sustained service and degraded state pass the endurance and drain deadlines.
• Converging storage onto the collective fabric without traffic isolation. Saving a dedicated rail by dropping 16k-GPU checkpoint incast onto the same InfiniBand that carries all-reduce converts a checkpoint into a training stall. The capex you saved is dwarfed by the goodput you lost.
• Treating the storage shelf as a thermal afterthought at Gen6. Putting a 96-drive Gen6 flash server into an air-cooled storage row without checking the populated chassis's power, inlet and airflow limits can strand the density behind a cooling wall. If the chassis needs liquid cooling, check flow and service access too.

Choose the medium and transport that meet your capacity, host-write endurance, payload-delivery and recovery targets together. Keep a tested staged path when its operational simplicity outweighs a measured direct-DMA gain, and adopt offload when it frees the resource that limits useful work. The purchase depends on the populated chassis and the surviving connection passing those tests. A faster drive or wire that leaves the command, controller or application stalled adds cost without feeding the GPU.