Sizing, Data Gravity & Resilience

You now have the pieces: the parallel filesystem (Chapter 9.2), the NVMe data path (Chapter 9.3), the checkpoint math (Chapter 9.4), the loader (Chapter 9.5), the object tier (Chapter 9.6) and the inference KV hierarchy (Chapter 9.7). This chapter assembles them into a sized, sited, survivable subsystem. The hard part is that storage fails silently. A mis-sized fabric throws an error. A mis-sized storage tier starves the GPUs, and the loss shows up as a utilization number a few points low, indistinguishable at a glance from a hundred other causes.

Four decisions structure what follows. Sizing sets the storage:compute ratio and the bandwidth budget that keep the accelerators fed. Network co-design keeps the checkpoint incast off the training collectives, the placement that most affects storage goodput. Data gravity asks whether you move the corpus to the compute or the compute to the corpus, and the egress economics behind that choice govern multi-site strategy. Resilience and TCO cover durability, multi-tenancy QoS, security, and the benchmarking-and-acceptance regime that turns a vendor claim into a contractual commitment.

Chapter 9.1 sets the rule this section applies: run its personality test and name each tier's governing axis before you buy. A hot tier feeding packed sequential shards or absorbing a checkpoint drain is bandwidth-sized, and the governing number is GB/s per GPU, not petabytes. A tier serving a directory of billions of small files is metadata-sized, and one serving KV fetches is latency-sized. Almost every AI capacity tier is capacity-sized: it holds the corpus and the checkpoint history, and the governing number is cost per TB. Here the hot tier is sized from the feed rate and the capacity tier from retained bytes.

The hot-tier budget starts from a per-GPU read target, and the reference architectures (RAs) give tiers, not one floor. NVIDIA's DGX B300 Spectrum-X Ethernet RA builds in 64-node, 512-GPU scalable units (SUs). Its Standard tier offers 80 GB/s of read and 40 GB/s of write per SU, roughly 0.16 GB/s of read per GPU. That tier counts on the compute nodes' own memory to cache most datasets. Its Enhanced tier, for multiple concurrent multimodal training jobs, is 250 / 124 GB/s per SU, about 0.49 GB/s/GPU. NVIDIA's XDR InfiniBand B300 RA publishes the same per-SU table for a 72-node, 576-GPU SU, which works out to about 0.43 GB/s/GPU for Enhanced and 0.14 for Standard. Name the RA before you divide.

The DGX H200 guidelines separate two levels. For Good, Better and Best, a single 8-GPU client needs read rates of 4, 8 and 40 GB/s, which is 0.5, 1 and 5 GB/s per GPU. The shared tier must deliver 15, 40 and 125 GB/s to a 32-node SU at once, about 0.06 to 0.49 GB/s/GPU. Don't put single-client and SU-aggregate figures on one axis. Computer-vision training gets its own line in both generations: about 4 GB/s of read per GPU. The H200 guidance ties it to high-resolution images; the B300 RAs tie it to datasets over 30 TB. Other workloads need read and write budgets you measure yourself.

NVIDIA says outright that its tiers assume a mix of workloads and that you should characterize your own first. So derive the read requirement from what the loader consumes: samples or tokens per GPU-second × stored bytes per sample, adjusted for cache hit rate and concurrent jobs, multiplied by GPU count. Derive the write floor from serialized checkpoint bytes and the drain window you will accept. NVIDIA's published read/write pair for the tier you compare against is a cross-check on that derivation, not a replacement for it. Use the exact pair rather than rounding it into a universal write-is-half-of-read rule. That is why the SuperPOD RAs quote bandwidth per scalable unit, not per array. At eight SUs the B300 Enhanced tier reaches 2,000 / 992 GB/s and Standard reaches 640 / 320. You size the tier to the matching SU count, not to a capacity quote.

Capacity sizing is the other axis, and a handful of ratios govern it. The corpus (raw plus tokenized plus versioned), checkpoint retention (committed generations and in-flight space times the unique serialized state from Chapter 9.4) and a working-set multiplier for shuffled access together set the petabytes. A code/text run and a frontier multimodal run can need very different corpus footprints at the same GPU count. Record corpus versions, replicas and checkpoint retention before buying single- or double-digit petabytes just because a fleet has a thousand GPUs. Size the hot tier to bandwidth and the cold tier to corpus, and don't let a capacity datasheet push you into buying the hot tier by the petabyte.

Follow two concurrent jobs through cold reads, overlapping saves, restore and host loss. Size the complete bid from the governing constraint. Here endurance sets the drive count, so the case buys extra capacity.

Select ten hosts and 120 drives on endurance. Require 100 GB/s reads and 40 GB/s protected logical writes simultaneously with one host absent and repair active, rounding the service floors upward. Nine surviving endpoints offer 9 × 40 = 360 GB/s per direction. Testing must show how controllers, coding and repair consume that ceiling.

The BOM has 128 compute nodes: 256 compute + 20 storage + four metadata ports = 280 endpoint links. Compute placement 43/43/42, storage 5/5/0 and metadata 1/1/0 use 49/49/42 endpoint ports plus six uplinks on each plane's leaves. All fit 64 ports. The two planes need six leaves, six spines and 36 leaf-to-spine links, and each spine uses six ports. Total cables: 280 + 36 = 316; cable-end terminations: 632. All-pluggable links need 632 modules, and each DAC/AOC replaces its two module positions. Chapter 8.5 covers media, lengths, spares and port modes.

Each leaf's six 200 Gb/s uplinks supply 150 GB/s line rate per direction. Five storage hosts expose 100 GB/s per plane, leaving 50 GB/s before protocol and cross-leaf repair. Placing all 32 vision nodes and ten text nodes on leaf three demands (256 × 500 × 0.40)/1,000 + (80 × 20,000 × 4)/10⁹ ≈ 51 GB/s. Test encoded traffic against these cuts. This case covers one host loss; rack, site and plane-loss service need separate testing.

Flip: surviving endurance permits 995.328/(1.5 × 1.1) ≈ 600 TB/day application writes, versus the assumed 509.6. Above that crossover, add hosts or buy drives with a higher rated endurance. The eight-host bid needs 840.84/(84 × 30.72) ≈ 0.33 DWPD, above 0.30. Extra namespace servers cannot repair a media-wear deficit. Complete the bid with metadata failover, firmware/power-loss behavior, GDS/client versions, licenses/support, OOB ports, racks, power, cooling and installation spares.

The storage subsystem does not live on its own wire. It shares the fabric that carries the training collectives, or you must deliberately keep it from sharing. In a training cluster the most damaging storage mistakes are placement errors, not sizing errors. A checkpoint is a synchronized burst: thousands of ranks drain (write) or read at the same instant (Chapter 9.4). That is a textbook incast, with many senders converging on one receiver and transient congestion at its switch ports. Put that incast on the same fabric as your all-reduce, on the same cadence as your checkpoint interval, and you get a periodic collision that shows up as goodput loss correlated with checkpoint timing. The checkpoint succeeds every time. It just steals bandwidth from the collectives whenever it runs.

The placement decision has three options, and it is a co-design problem spanning storage and network rather than either alone (the fabric side is engineered in Chapter 8.5). A dedicated storage rail, a physically separate set of NICs and switches for storage traffic, means a checkpoint storm never crosses the back-end switches, though the snapshot copy still competes inside each node. Converging storage onto the back-end fabric with strict QoS is cheaper in NICs and switches, but you now rely on PFC/ECN priority classes and queue separation to keep the incast from starving the all-gather, and you must prove it holds under load. Sharing with no isolation works until the first concurrent checkpoint-and-collective collision, and then step time degrades mysteriously and intermittently.

Data has gravity: the larger and more active a dataset, the harder and costlier it is to move, and the more it pulls services and compute toward it. For an AI program this becomes a hard economic constraint at petabyte scale, because the cloud bill for moving data, egress, is asymmetric by design. Ingress is free; egress is metered. The hyperscalers price internet egress at roughly $0.087–0.11 per GB at the first paid tier (Azure ~$0.087/GB, AWS ~$0.09/GB, GCP Premium $0.12/GiB, about $0.112/GB; list price, first tier; negotiated rates and the deeper volume tiers are lower). Cross-region and cross-AZ transfer add their own per-GB charges on top.

At petabyte scale these are not rounding errors. Moving 1 PB out at the ~$0.09/GB first-tier rate costs on the order of ~$90,000, and even at AWS's published volume tiers it is ~$54,000. A planned bulk move should be priced on a private path instead: AWS charges $0.02/GB out over Direct Connect from a US region to a US location (~$20,000 per PB), plus the port at $22.50/hour for 100 Gb/s dedicated, which moves a petabyte in about a day at line rate.

The share of the bill depends on the ratio of bytes out to bytes stored. One zero-egress storage vendor's scenario, 20 TB stored with 15 TB a month sent out, puts egress at 74% (AWS), 78% (Azure) and 79% (GCP) of the bill. At AWS first-tier list prices ($0.023/GB-month for S3 Standard, $0.09/GB out), the monthly egress bill passes the storage bill once more than about a quarter of the stored bytes leave each month (0.023 / 0.09 ≈ 0.26). Build the actual bill from stored GB-month by class, requests, retrieval bytes, each billed transfer leg, destination storage, verification and repeat reads after cache misses. AWS S3 pricing separates these components, and each candidate venue needs the same schedule on the same date.

That asymmetry forces the central question: move the data to the compute, or move the compute to the data? Moving data to compute is the default, and it is fine while the data is small or born where the GPUs are. Once the corpus is large, born in one place (a customer's region, a regulated jurisdiction, an on-prem lake) and accessed repeatedly, gravity flips the economics. Bringing the accelerators to the data beats repeatedly paying egress to stream it to a distant cluster. That is why frontier operators co-locate the prep supercomputer (Chapter 9.9) with the corpus, why regulated workloads pin compute to the jurisdiction where the data must legally remain (Chapter 10.10), and why multi-site strategy follows gravity rather than the other way around.

The egress structure also creates a deliberate lock-in: because leaving costs money the incumbent keeps, gravity is a moat. The 2026 wrinkle is regulatory, and it has three pieces that get conflated. Provider exit programs came first, and they are voluntary and conditional: AWS has waived egress for customers leaving the platform since March 2024, and the other majors run comparable programs you have to qualify for and apply to. The EU Data Act's outright ban on switching charges, egress included, applies only from 12 January 2027. Until then providers may still levy cost-based switching fees. A third piece covers running workloads rather than exits. For EU customers running workloads across several providers in parallel, the EU Data Act (Art. 34) lets a provider charge for egress only to recover its actual cost. Since September 2025 Google Cloud has billed eligible EU and UK multicloud traffic at zero through its opt-in Data Transfer Essentials, which covers only one organization's in-parallel workloads. Microsoft offers an at-cost credit through a validated support request, and AWS lets EU customers request reduced rates.

The first two pieces dent the moat only for a qualifying exit, and the third only for multicloud pipelines. None of the three touches the day-to-day cross-region streaming inside one provider that dominates an active AI pipeline. So design the data's birthplace and residence deliberately. Land the corpus where the compute will live, replicate selectively rather than stream repeatedly, and treat every cross-region copy as a recurring egress liability, not a one-time cost.

Compare streaming and a local replica over one horizon. Streaming pays for charged cache-miss bytes on every pass, plus both endpoint services. A replica pays for the initial verified move, changed-byte synchronization, protected capacity and operations. Initial move cost divided by positive recurring net savings gives the crossover horizon. Transfer bytes divided by sustained path payload, plus verification and cutover, gives the move deadline. Choose replication only if both tests pass; low reuse or high standing cost favors streaming. Data rights and recovery can exclude either option first.

Resilience in an AI storage subsystem is a different problem from resilience in enterprise IT, because the workload values a different thing. Traditional storage optimizes durability and availability: never lose a byte, never be unreachable. An AI training storage tier optimizes goodput. It keeps the GPUs fed and lets a failure cost as few GPU-hours as possible (Chapter 12.2). The two diverge sharply on the hot tier. A scratch/checkpoint hot tier does not need eleven-nines durability, because its contents are reproducible from the last durable checkpoint and the corpus. Heavy erasure coding and cross-site replication there buy durability the workload does not value, at the cost of the write bandwidth it does. The capacity tier is the inverse. The corpus and the canonical checkpoint history are the irreplaceable assets, and that is where durability spend belongs (Chapter 9.6).

Multi-tenancy and QoS come next, and this is where a shared storage tier earns or loses its keep. A storage fabric serving many tenants or many jobs is a noisy-neighbor problem in slow motion. One tenant's metadata storm or checkpoint burst can starve another's ingestion reads, and the victim sees it as unexplained GPU idle. The controls are tenant-level bandwidth and IOPS quotas, metadata-rate fairness, and priority classes that protect the latency-sensitive (KV, ingestion) from the bursty (checkpoint). This is the storage-tier mirror of the compute-side sharing spectrum in Chapter 10.3 and the congestion engineering in Chapter 8.6. The NVMe 2.4 specification (August 2026) lets a drive controller enforce its own bandwidth and IOPS caps. Those caps could back tenant quotas on shared drives, including NVMe-oF ones. No shipping drive is confirmed to support this yet, so put it in the RFP as a requirement to confirm, not an assumption. Avoid designing around Predictable Latency Mode or NVM Sets: NVM Express plans to obsolete both in 2027.

Security rounds it out: encryption at rest and in flight, tenant isolation strong enough that one tenant cannot read another's corpus or checkpoints, and, for confidential workloads, attestation-gated access consistent with the TEE model in Chapter 11.5. Weights and training data are among the most valuable and most regulated assets in the building, and the storage tier is where they sit at rest.

Storage is a small line on the bill of materials and a large lever on the outcome, which is why it gets under-funded. The capex is real but bounded. An all-flash parallel filesystem sized to a large cluster's per-GPU feed must be compared with the accelerator capex it keeps productive, including protected media, host service, fabric, software/support and installed power/cooling, all priced for the same workload. The cost that dominates is the GPUs the storage strands. A hot tier one notch too small to keep a fleet fed, or a checkpoint path that perturbs collectives a few percent, costs goodput across the entire accelerator fleet. On a power-bound, depreciating fleet, every point of goodput is a proportional fraction of the most expensive asset in the building. What you are pricing is the array plus the GPU-hours a wrong array burns.

The recurring costs split by tier and by venue. On-prem, the hot tier is dominated by flash media and the parallel-FS software/support, and the capacity tier by $/TB and the power and space of dense flash or remaining HDD. In the cloud, egress and cross-region transfer can govern a repeatedly remote-read workload, as the gravity section showed, so a cloud storage TCO that ignores data movement is fiction. Price three things together: the media, the bandwidth (and the fabric it rides), and the movement. Optimize any one in isolation and the other two punish you.

Run the two-job workload and retain manifests, versions, topology and time series. Require cold loader rates, simultaneous 60 s drains and 5 s commits, then repeat with one host absent and repair active at the 100/40 GB/s gates. In each matched repetition, loaded p99 collective step time must be ≤1.05× baseline. If it is not, choose more isolation, admission control or capacity.

Restore both jobs within Chapter 9.4's state-age, 180 s recovery and correct-next-step limits. Separately fail a namespace service and a storage path against the Chapter 9.2/Chapter 9.3 budgets. Test missing or corrupt generations, noisy neighbors and unauthorized reads, and verify daily host writes and reserve. Don't sign the purchase until these results and the installed power/cooling schedule are in. Chapter 13.9 covers how to run commissioning.

Four trends are reshaping the sizing-and-resilience calculus, and the consolidated roadmap (Chapter 16.2), not this chapter, treats each of them. GPU- and DPU-initiated I/O moves the data path off the host CPU entirely. The accelerator (or a BlueField-class DPU) issues storage I/O directly, which removes a CPU bottleneck that has capped the per-GPU feed and shifts where the storage rail terminates (Chapter 9.3). All-flash everywhere, with QLC-backed object and capacity tiers displacing HDD, changes capacity-tier TCO from a $/TB-and-spindle calculation to a $/TB-and-watt one and narrows the bandwidth gap between hot and cold tiers. The 2026 NAND shortage has paused the displacement on price (see Chapter 9.6). File/object convergence erodes the hard line between the parallel-FS hot tier and the object capacity tier, letting a single namespace span both and simplifying placement. And deeper CXL tiering (Chapter 9.7) inserts a memory-class tier between DRAM and NVMe that reshapes the inference KV hierarchy and, increasingly, the training scratch tier. None of these repeals the chapter's logic: you still size to a feed rate, isolate the incast, and pay for gravity. They move the numbers, not the decisions.

Choose the bid whose surviving configuration passes service, capacity, metadata and wear together, then site it using the complete movement bill and recovery boundary. The worked BOM buys more drives because endurance governs. Another workload can instead buy uplinks, namespace services or a different cache policy. Accepting a paper total without the mixed-load and failure tests spends the same money and leaves the program's useful output unproven.