Storage & Data

Storage decides whether an accelerator computes or idles. Four I/O personalities compete for the same flash and fabric and can exhaust different resources, so a tier tuned for one can starve the others and leave GPUs idle.

A parallel file system keeps GPUs saturated only if its metadata plane survives the small-file storm and its data plane supplies the bytes. If your corpus holds billions of small files, test with them: sequential bandwidth alone cannot prove the system fits.

Whether a GPU stays fed can turn on per-request latency before aggregate bandwidth. A supported direct path moves bytes from flash into HBM without a host-memory bounce buffer, but your measured workload and the path that actually survives decide whether the bypass pays.

Checkpointing is the control knob for training goodput. The interval, tier and write bandwidth you choose set how many GPU-hours each failure erases, and at frontier scale interruptions come every few hours.

The data loader sits inside every training step. Get its format, sharding or CPU budget wrong and the GPUs stall, burning depreciation while they wait to be fed.

Object storage holds the whole corpus, every checkpoint lineage and every shipped model. What it costs you depends less on the vendor than on whether you build it as a flash-fronted serving layer the GPUs read from.

Inference turned the KV-cache into a first-class storage problem. Its bytes now spill past HBM, and where they land (DRAM, CXL, NVMe or Ethernet-attached flash) sets your tokens per second and your cost per token.

Size the hot tier by per-GPU bandwidth when sequential delivery binds, then check metadata, retained bytes and endurance. Put the corpus where the compute can reach it, and control checkpoint incast. Miss a budget in the surviving system and idle accelerators absorb the cost.

Before any GPU sees a token, a second cluster must dedupe, filter, decontaminate and tokenize trillions of tokens. Undersize it and you starve the training fleet or waste GPU-hours on string processing.