Data Ingestion, Preprocessing & the Data-Loader Path

The earlier chapters of Part 9 cover storage that sits beside the training loop: the parallel file system that holds the corpus (Chapter 9.2), the NVMe tier and GPUDirect path that move bytes fast (Chapter 9.3), and the checkpoint stream that protects against failure (Chapter 9.4). This chapter covers the path that sits inside the loop. The GPU cannot start step N until the batch for step N is decoded, augmented, collated and resident in device memory. If the pipeline cannot sustain that rate, the accelerator stalls and burns depreciation while it waits.

The loader path turns almost every wrong choice into GPU idle time, and idle GPUs mean wasted megawatts and stranded depreciation. We trace the runtime pipeline stage by stage, then the format fork (sequential vs columnar vs raw), the placement fork (stream vs stage vs cache) and the offload fork (CPU vs GPU vs pre-decoded). We end on the two mistakes people make most often: under-provisioning CPU and DRAM relative to GPUs, and treating a multimodal pipeline as the text pipeline with pictures bolted on. Offline corpus construction (dedup, filtering, tokenization at corpus scale) is a different machine and lives in Chapter 9.9. The legal regime that governs what may enter the corpus is Chapter 10.10.

A training data loader is a producer-consumer pipeline with the GPU as the consumer. Every sample passes through six stages on its way into a batch: (1) fetch a shard or object from the storage tier; (2) decode the container and the payload (JPEG/WebP, audio, video frames or token records); (3) transform/augment (resize, crop, normalize, tokenize, pad); (4) shuffle across a buffer so consecutive batches are decorrelated; (5) collate samples into a tensor batch; and (6) transfer the batch host-to-device over PCIe/NVLink. Stages 1–5 run on the CPU host unless you explicitly offload them. Stage 6 crosses the bus into HBM.

The governing constraint is a throughput inequality. Let the GPUs consume batches at rate C (samples/s) and the loader produce them at rate P. In an ideal, steady-state overlapped pipeline, if P ≥ C and prefetch absorbs the measured jitter, the GPU need not wait for the loader. If P < C, the average data-starved fraction is (1 − P/C), and measured per-step waits also include tails beyond that model. No amount of fabric bandwidth or HBM capacity recovers the loss, because you are bottlenecked on the slowest of stages 1–6. Keeping P ≥ C is the whole engineering problem of the loader path. C climbs with each GPU generation, the corpus grows into the petabytes, and per-sample decode cost, especially for images and video, refuses to fall.

A slow loader appears as idle GPUs rather than as missing bandwidth, so MLPerf Storage's training results are framed around accelerator utilization (AU), not raw bandwidth. Real reads must feed an emulated accelerator-consumption schedule while meeting the validity threshold in the MLPerf Storage rules. v3.0 (September 2026) requires 90% AU for Unet3D and 85% for RetinaNet. The August 2025 v2.0 round used 90% for 3D U-Net and ResNet-50 and 70% for CosmoFlow. v3.0 also lets training and checkpointing run over an S3 object interface as well as POSIX, so a buyer can compare object-direct ingestion with a file tier under the same AU rule. An S3 result and a POSIX result are different access paths, so compare like with like.

The benchmark result is a maximum emulated accelerator count under those rules. It is not a promise about the proposed model's MFU or training goodput. A storage tier that misses the real loader's consumption schedule still starves the most expensive asset in the data center. Keep the benchmark's workload, version and cache rules intact, then test batch wait, accepted steps and restart behavior with your real loader. → Chapter 9.3 for the measurement boundary; Chapter 9.8 for installed acceptance.

The first hard-to-reverse decision is the on-disk format. It is written across petabytes, and re-encoding a multi-PB corpus is a multi-day, multi-thousand-core job in its own right (→ Chapter 9.9). Three families dominate, and they trade the same three things against each other: sequential read efficiency, random-access / column-projection ability, and metadata/small-file pressure on the storage tier.

Sequential shards (WebDataset in POSIX tar, and MosaicML MDS) are the default for LLM and large multimodal training. The corpus is packed into shards of typically 50 MB–1 GB, each holding thousands of samples in record order. MDS defaults to ~67 MB, and Databricks/Mosaic guidance says 50–100 MB shards work well across modalities. The win is that the loader issues large sequential reads instead of millions of tiny random ones. That is the difference between a parallel file system running near its streaming bandwidth and the same file system collapsing under metadata-server load from a small-file storm. The small-file problem from Chapter 9.2 is exactly what sharding exists to defeat. The cost is that random access within a shard is awkward, and you cannot cheaply read one column of a multi-field record without decoding the whole sample.

Columnar Apache Parquet is the default for data-lake-native, tabular and structured data. Parquet stores data by column with embedded compression and statistics, so a loader can project just the fields it needs and push down filters. It also integrates natively with the lakehouse engines that produced the data (→ Chapter 9.6). That suits text/token corpora and structured multimodal metadata. It is less natural for raw image/video bytes, and naive row-group sizing can reintroduce small-read patterns. Parquet is where the corpus lives during curation, and many shops convert hot training splits from Parquet into WebDataset/MDS shards for the run itself.

Raw files (one image or clip per object) are the path of least resistance and the most common scaling mistake. Take a directory of 2 billion JPEGs as an illustrative corpus. It is trivial to produce and punishing to train from when the namespace service is undersized. Reading each sample can require several metadata operations plus a tiny random read. That saturates the file system's metadata plane long before its bandwidth, and it makes object-store request-rate limits the binding constraint. Raw files are fine for small datasets and prototyping. At corpus scale, that metadata load makes a fast storage tier perform like a slow one. Pack the corpus into shards once it caps your read rate; keep raw files only if the job needs cheap random access to individual samples.

Sharding solves the read-pattern problem but creates a statistics problem. If you read shards in order and emit samples in shard order, every batch comes from a narrow slice of the corpus, gradients are correlated and training quality suffers. The fix is to shuffle, but you cannot hold a 50 TB corpus in memory for a global shuffle. The standard technique is a two-level shuffle. First shuffle the order of shards globally, which is cheap because there are only thousands of them. Then keep a shuffle buffer of K samples in host DRAM and draw each batch at random from it as new samples stream in.

Buffers of a few thousand to tens of thousands of samples are common starting points. Buffer size trades host memory against decorrelation, and how the corpus is ordered by source and class decides whether a small buffer hurts training quality. A bounded shuffle is not a uniform permutation of the whole corpus, so test sample mixing and model quality at the buffer size you pick.

The choice with the largest downstream cost is determinism under elastic scale. A 10,000-GPU run will be interrupted: Meta's Llama 3 405B saw an unplanned interruption roughly every three hours over a 54-day window on up to 16,384 H100s (Meta, Llama 3 paper, 2024). On restart you want the run to resume mid-epoch, in seconds, without re-reading from the top, and you want the sample order identical whether it restarts on 16,384 GPUs or 12,288. Streaming loaders built for this (MosaicML StreamingDataset is the canonical example) make the shuffle elastically deterministic. Sample order is a function of the saved seed, shuffle algorithm and consumed-sample position, so supported elastic resumption can replay the same ordered samples to debug a loss spike.

A loader without this turns every failure into either a full-epoch replay (egress and idle-GPU cost) or a silently non-deterministic data order (irreproducible training). The checkpoint/restart math this rides on is in Chapter 9.4.

Prove elastic resumption with an uninterrupted control run and an interrupted run from the same immutable dataset. Store the consumed global sample cursor alongside model state, not the count of samples already fetched into queues. On restart, restore it together with the loader state and corpus manifest, and keep the batching policy and library release unchanged. Keep the global batch size constant and divisible by the restart device count, and hold the canonical-node partition, seed, shuffle algorithm and transform policy fixed. Reject a restart count that cannot divide the global batch before you launch it. Restart on a supported device count and compare the next ordered global sample IDs and augmentation seeds. Prefetched but unconsumed records must neither disappear nor count as completed. The Mosaic Streaming requirements define which device counts and settings are supported. Exact sample order does not promise bit-identical floating-point training, so test that separately against a tolerance.

The failure that strands the most compute is invisible until the GPUs are installed. Stages 2–5 (decode, augment, shuffle, collate) run on the CPU host by default, and they do not get faster when you buy faster GPUs. As accelerator throughput climbs generation over generation (H100 to B200 to GB300 to Vera Rubin), the consumption rate C climbs with it. The host CPU's production rate P stays flat unless you provision for it. A pipeline that was balanced on the previous GPU generation becomes CPU-starved on the next. NVIDIA's DALI documentation states the root cause directly: "These data processing pipelines, which are currently executed on the CPU, have become a bottleneck…"

The lever is the GPU:CPU-core and GPU:host-DRAM ratio, decided when you spec the server, and it depends on the workload in ways that catch people out. Rack-scale systems fix the ratio for you, so look it up before you pick the platform. NVIDIA's spec pages (October 2026) show how far apart the platforms sit. The DGX B200 pairs each of its eight GPUs with 14 x86 cores and 256 GB of DRAM on the host side: 112 cores and 2 TB in all, with memory configurable to 4 TB. On a GB300 NVL72 it is 36 Arm cores with about 236 GB of LPDDR5X; across all 72 GPUs, 2,592 cores and 17 TB. On a Vera Rubin NVL72 it is 44 cores with up to about 750 GB; across the rack, 3,168 cores and up to 54 TB. Core counts do not translate one-for-one into decode throughput across x86 and Arm. Training processes, communication threads and checkpoint staging claim cores and memory first, and on Grace and Vera so does any GPU use of the coherent CPU memory. If the loader needs more than what is left, the extra has to come from GPU offload, offline pre-processing or separate CPU preprocessing nodes.

A pure-text LLM pipeline is cheap per sample. Tokens are small, decode is light and augmentation is trivial, so a modest core count per GPU and a few dozen prefetch workers keep P ahead of C, if profiling confirms that count keeps pace without exhausting host memory. An image pipeline, and far more so a video pipeline, is the opposite. Media decode cost depends on codec, resolution, frame count and pipeline, so benchmark CPU seconds and decoded bytes per sample for your own workload. The chassis that comfortably feeds eight GPUs on text will starve them on video. Under-provision cores and you cap goodput at whatever the CPU can produce. Over-provision and you pay for idle cores, which are cheap insurance next to idle GPUs.

Host DRAM is the second half of the ratio, and it serves three masters at once: the shuffle buffer, the prefetch queue (you want several batches staged ahead so a slow read never reaches the GPU) and, if you stage rather than stream, the node-local dataset cache. Under-size DRAM and the shuffle buffer shrinks (worse decorrelation), the prefetch depth shrinks (jitter reaches the GPU), or the cache thrashes back to the storage tier. A training node can therefore need substantial host DRAM alongside its HBM. The loader's shuffle and prefetch buffers must fit while page cache, checkpoint staging, optimizer/offload state and the operating system also use memory. Budget and test those reservations together, because the DRAM is not all available to the loader.

The node demands 8 × 500 = 4,000 samples/s. Stored reads are 4,000 × 0.40 MB = 1.6 GB/s, and decoded delivery is 4,000 × 2.0 MB = 8.0 GB/s. CPU demand is 4,000 × 0.0020 = 8.0 core-seconds/s, so ceil(8.0/0.70) selects 12 worker cores, with training and services already reserved outside this budget. A 0.50 s gap needs 2,000 prefetched decoded samples, or 4.0 GB. Compressed shuffle uses 10,000 × 0.40 MB = 4.0 GB, so loader buffers total 4.0 + 4.0 + 1.0 = 9.0 GB.

Add the process allowance and checkpoint staging and you get 9.0 + 6.0 + 40 = 55 GB, which fits the 64 GB allocation. Choose compressed shuffle, and test those 12 cores with the full pipeline. Flip: decoded shuffle needs 20 GB, which raises concurrent use to 4.0 + 20 + 1.0 + 6.0 + 40 = 71 GB. That fails. The crossover for the 10,000-sample shuffle is (64 − 4 − 1 − 6 − 40) GB / 10,000 = 1.3 MB/sample. Above it, keep shuffle compressed, reduce the tested buffer depth or add memory. More storage bandwidth does not repair the memory deficit. DALI's CPU/mixed/GPU operator model offers an offload alternative to profile, not a speedup for these assumed rates. Chapter 9.8 sizes the storage tier from per-node demand like this.

When the CPU cannot keep up, there are three main escapes. They trade CPU cost, GPU cost and storage footprint against each other.

Scale out the CPU. Add cores, prefetch workers and nodes. This is the brute-force path, and it works until you hit the chassis core ceiling or the point where adding CPU sockets to feed GPUs costs more than the alternatives. Its real virtue is that the data path stays simple and the format stays unchanged.

Offload decode and augmentation to the GPU (NVIDIA DALI). DALI can move JPEG and video decode and augmentation onto the GPU, through nvJPEG-based image decoders and NVDEC where the device and codec support them, alongside selected CPU and mixed operators. It runs its own execution engine to overlap I/O with compute. NVIDIA describes it as addressing "the problem of the CPU bottleneck by offloading data preprocessing to the GPU." This is the dominant answer for image and video pipelines where decode is the wall. The tradeoff is real: you spend GPU cycles and a slice of HBM bandwidth on preprocessing instead of on the model. That pays off when the CPU alternative would otherwise leave the GPU idle, and it is a poor trade when the GPU is already compute-bound.

Pre-process the corpus once into a training-optimized format (FFCV, or transforms baked into MDS). FFCV does the resize and re-encode work a single time, offline, writes samples to one page-aligned file, and reads them back with caching and asynchronous threads. Some CPU work per sample remains: JPEG decode if samples stay compressed, plus JIT-compiled augmentations. That is far less than a stock loader does, but it is not zero, and the authors chose to keep it on the CPU so the GPU spends its cycles on training. Storing samples raw removes decode at a much larger storage footprint. The authors report ResNet-50 reaching 75% accuracy on ImageNet in about 20 minutes on one machine (Leclerc et al., CVPR 2023). Pre-processing is the fastest path for fixed-preprocessing workloads.

The cost is paid in two coins. The storage footprint is larger or differently shaped, since decoded data is bigger than compressed. And you lose flexibility: augmentations baked in at encode time cannot vary per epoch, so heavy randomized augmentation either stays on the runtime path or must be designed into the format.

Blackwell (B200, B300, GB200, GB300) adds a fourth option, narrower than the three above: the hardware Decompression Engine, which handles decompression and nothing else. It sits in the copy engine and you reach it through nvCOMP. Snappy, LZ4 and Deflate/Gzip data is decompressed as it crosses PCIe or C2C, at up to 600 GB/s by NVIDIA's figure, and no SMs are used. It cannot decode images or video, and NVIDIA lists no Zstandard support, so pick the shard codec accordingly. For host-to-device copies, the buffers have to be allocated from a pool carrying the hardware-decompress usage flag. With any other allocation, nvCOMP's default backend falls back to SM kernels instead, so check a profile to see whether the engine is really in use.

A DALI operator is not itself proof of GPUDirect Storage. For CPU decode, host staging can be the intended path. For GPU-ready records, choose a loader and storage client that issue supported cuFile operations, and repeat the process-counter checks in Chapter 9.3. Keep dtype, shape, sample identity and data checksum fixed between direct and staged runs. Pick the integration that reduces real batch wait without changing the training input or overcommitting HBM.

Independent of format and offload is the placement question: where do the bytes sit relative to the GPU when the run starts? There are three postures. The fork sets your startup latency, your egress bill and your exposure to storage tail latency.

Stream from object storage. The loader pulls shards directly from S3/GCS/on-prem object as it needs them and holds only the shuffle and prefetch buffers locally. MosaicML StreamingDataset and WebDataset-over-S3 are built for this, and AWS publishes loader best practices for the same pattern: parallel connections, sharding and prefetch to keep GPUs fed. The win is that you skip a full-corpus staging step, and the data can be larger than any local disk. First-shard fetch and decoder startup still determine time to the first batch. The exposure is object-store tail latency and request-rate limits, since a single slow GET that reaches the GPU is a stall. There is also recurring egress cost when compute and storage sit on a billed cross-zone or cross-region route, or when the service charges for retrieval (the data-gravity economics of Chapter 9.8). Deep prefetch and many parallel connections hide the tail.

Stage to a node-local NVMe or parallel-FS cache, then train. Copy the working set onto fast local storage up front and read from there for the run. This removes per-step dependence on the object store and its tail latency, and it amortizes egress over many epochs. It is the right call for a corpus read many times, and Meta's Research SuperCluster fronted training with a multi-tens-of-PB flash cache. The cost is the staging delay before the first step and the local capacity ceiling. If the working set exceeds local NVMe, you cannot stage it whole and must fall back to streaming or a hybrid.

Cache hot, stream cold (the hybrid that wins in practice). Most real pipelines stream from object storage but interpose a node-local or rack-local cache of recently used shards. The first epoch pays object-store cost and later epochs are served from flash. It is the placement version of the storage hierarchy that runs through all of Part 9, and it explains why the parallel-FS and NVMe tiers of Chapter 9.2 and Chapter 9.3 sit between object storage and the GPU.

A text pipeline is close to uniform. Every sample is a token sequence of bounded size, decode is trivial and the batch is a clean rectangle. A multimodal pipeline (interleaved image-text, audio or video) breaks all three assumptions at once, and treating it as "the text pipeline plus pictures" is a common design mistake.

First, per-sample cost variance explodes. A caption is bytes. The image it describes is a megabyte of JPEG that must be decoded and resized, and the video clip is hundreds of frames through a hardware decoder. Within one batch, decode cost varies with codec, resolution and frame count, so the slowest sample sets the batch latency, and a few heavy video samples can stall an otherwise-fed pipeline. Uneven decode cost is the strongest case for GPU-offloaded decode (DALI's hardware NVDEC path) and for length-aware or cost-aware batching, which groups similar-cost samples so the straggler problem stays bounded.

Second, modalities arrive at different rates and sizes. The shard often packs heterogeneous records, and the collate step must pad and mask across variable-length sequences and variable-resolution tensors. That work is itself CPU-heavy and a frequent hidden bottleneck. Third, alignment and sampling logic lives in the loader: which frames of a video pair with which text span, and how to sample within long clips. It must be deterministic for resumption to work.

The engineering consequence is that multimodal runs need a higher CPU:GPU ratio, more aggressive GPU offload and cost-aware batching. They also expose loader weaknesses that a text-only run never would. Decode cost also makes preparing a multimodal corpus offline far heavier than preparing text: images and video must be decoded, frames extracted and everything re-encoded into uniform shards. Do that work in the offline supercomputer of Chapter 9.9, not on the runtime path.

The loader must let the program know, and prove, exactly which data went into a given model. Three capabilities make that possible, and they are increasingly built into the format and loader rather than bolted on afterward.

Versioning pins a corpus to an immutable, content-addressed snapshot. "Model X was trained on dataset version Y" then becomes a precise, reproducible statement, and re-running training reads byte-identical inputs. Lineage records the transformation graph from raw source through dedup, filtering and tokenization (→ Chapter 9.9) to the shards the loader consumed, so any sample can be traced back to its origin. Governance enforces what may enter the corpus and proves what did: license and consent status, PII handling, opt-outs and regional restrictions.

At frontier scale, training-data provenance is a first-class liability surface. Where copyright, privacy or data-residency obligations apply, answering "what was in the corpus?" with evidence is part of meeting them. Set provenance and retention from the applicable regime and the program's reproducibility requirements. That regime, and the compliance architecture it demands, is the subject of Chapter 10.10. The loader's job is to make the answer cheap to produce by carrying version and lineage metadata in the shards it streams.

Choose format, worker count and buffering from the measured modality, then pick CPU, GPU or offline decoding by the resource it frees. A loader passes when it supplies the required samples through jitter and concurrent checkpoint staging, and resumes the saved sample order after a failure. A faster format chosen without those memory and correctness budgets can swap visible I/O wait for silent data repetition or host-memory exhaustion.