The Data-Prep Supercomputer: Offline Data Processing

A frontier training program runs two supercomputers, and only one of them is GPU-dominant. The first, the one this Part has been about, keeps thousands of accelerators saturated with a clean, shuffled, tokenized stream. The second runs before it, often weeks before, and manufactures that stream. It ingests raw web crawls and licensed corpora, strips boilerplate, deduplicates at corpus scale, filters for quality and safety, scrubs out anything that overlaps your evaluations, and tokenizes the survivors into the exact binary format the loader will memory-map. This is the data-prep supercomputer, and it is the most consistently undersized, under-instrumented and under-respected machine in the building.

It gets disrespected because it looks like ETL, and every organization thinks it already knows ETL. But web-scale prep for a frontier run is not a nightly Spark job. It is a multi-petabyte, trillion-token batch pipeline whose stages have opposite hardware appetites. Some are embarrassingly parallel and CPU-bound, some are metadata-storm small-file workloads, some are GPU-accelerated classifier sweeps, and some are all-to-all shuffles that look like a network benchmark. Size it as one thing and you bottleneck on whichever stage you got wrong. This chapter walks the pipeline stage by stage, then sizes the cluster that runs it, whose power, network and storage profile is deliberately unlike the GPU fleet next door.

The case for taking prep seriously is a goodput case, and it cuts two ways. First, prep is on the critical path to the run. If the corpus takes six weeks to prepare and your GPU cluster is energized and idle for four of them, you have paid frontier-cluster lease or depreciation for a month of string processing. Prep wall-clock comes straight out of the training schedule, and the GPUs cannot start a training release until its authorized, versioned input is ready. Incremental corpus releases need their own quality gates. Second, the quality of prep sets the ceiling on what the GPUs can achieve. The open-data era (FineWeb, RefinedWeb, DataComp-LM, Dolma) taught that careful filtering and deduplication beat raw scale at a fixed token budget: a smaller, cleaner corpus trains a better model than a larger, dirtier one. Nemotron-CC (NVIDIA, 2024) adds the limit: filters that discard about 90% of a crawl leave too few unique tokens for a 15T-token run, and its 6.3T-token set matched DCLM on MMLU with about four times the unique real tokens. Size the filter to the token horizon you plan to train on, not to the highest short-run score. Every duplicate you fail to remove is a GPU-hour spent memorizing boilerplate, and every leaked benchmark sample is a leaderboard number you have to caveat.

The decision is rarely whether to do prep at all; it is how much engineering to invest, and on what hardware. The expensive failure is the one nobody plans. Two weeks before launch you discover that the pipeline cannot tokenize the licensed corpus fast enough on your CPU pool, and you either delay the run or, far worse, borrow GPU nodes to brute-force tokenization. That is an expensive way to run a Rust regex engine if the exact tokenizer and normalization pipeline still execute on the CPU while the accelerators sit idle.

A web-scale prep pipeline is a directed sequence of stages, each with a distinct hardware personality and a distinct fork. The canonical order is text extraction, language ID, quality/heuristic filtering, exact dedup, fuzzy dedup, classifier filtering, decontamination, tokenization, then shuffle/shard. It is not arbitrary: you filter cheaply before you filter expensively, and you dedup before you tokenize so you never tokenize a duplicate. The table maps each stage to what binds it, so you can see where the prep cluster's bottleneck actually lives.

The web mirrors, syndicates and templates itself relentlessly. In RefinedWeb (2023), exact and fuzzy dedup removed about half of the documents that had already passed language and quality filtering, about 12% of the original Common Crawl documents. Removing those duplicates is one of the largest levers on token efficiency, alongside quality filtering, because a model wastes capacity memorizing anything it sees too many times, and over-represented duplicates skew the distribution. But dedup is also a deletion from the derived training set. Retained source IDs and an authorized, immutable input make that exclusion reproducible and reversible, and the aggressiveness knob (how similar counts as "the same") is the most consequential single number in the pipeline.

There are three tiers, and they compose. Exact dedup hashes whole documents or lines and drops identical copies. It is cheap, CPU-bound and embarrassingly parallel, and it catches the trivial cases. Fuzzy / near-duplicate dedup is the real work. Compute a MinHash signature per document (a compact sketch of its n-gram set), then use Locality-Sensitive Hashing (LSH) to bucket documents whose signatures collide in any band. You then compare only plausibly similar pairs instead of facing the quadratic all-pairs explosion.

The banding configuration sets a probability curve, not a hard Jaccard cutoff. FineWeb used word 5-grams and 14 bands of eight MinHash values; its method and per-snapshot comparison are documented by the authors in 2024. At similarity s, the candidate probability is 1 − (1 − s⁸)¹⁴; at s = 0.75 it is about 77%. Record the normalization, candidate handling and survivor policy with the corpus they produce. Semantic dedup goes further: embed every document, cluster in embedding space, and drop near-neighbors that are paraphrases or translations a MinHash would miss. The cost is an embedding-model inference pass over the entire corpus.

The hardware fork inside dedup carries the biggest 2026 swing. Fuzzy dedup at trillion-token scale is an all-to-all shuffle of billions of signatures. On a CPU cluster it can end up the long pole, so time it instead of assuming. GPU-accelerated dedup (NVIDIA's NeMo Curator on RAPIDS, and the FED-class GPU dedup frameworks) collapses that time. In NVIDIA's own test, NeMo Curator v25.07 took 0.5 h on 32 H100 GPUs to deduplicate 1.96 trillion tokens, a RedPajama-v2-scale corpus. Treat that as a vendor result against unnamed baselines, and re-measure with your own corpus and dedup policy. Curator's dedup workflows need GPUs and run on a Ray backend. A prep pool with no GPUs needs a CPU engine instead, such as datatrove, which Hugging Face used to build FineWeb's 2024 release: 15T tokens from 96 Common Crawl snapshots.

NVIDIA’s NeMo Curator deduplication stages put GPU implementations on the shortlist. Compare signature generation, bucketing, shuffle and survivor selection separately, at a fixed corpus, normalization and dedup policy. Keep end-to-end wall-clock, peak memory, spill and accepted output with the CPU/GPU comparison, because a fast signature kernel does not time the complete corpus release. This is the rare case where the prep cluster legitimately wants a few GPUs, not to train but to keep the dedup stage from being the long pole. You either pay for a large CPU pool to grind the shuffle, or borrow a small GPU partition to do dedup in hours and free the CPUs for tokenization.

After dedup, you decide what is worth keeping. The fork is heuristic filtering versus classifier-based curation, and most serious pipelines use both: heuristics first because they are cheap and interpretable, classifiers where a measured quality gain justifies their cost.

Heuristic filters are the C4/Gopher/FineWeb lineage. They drop documents by language-ID confidence, mean line length, fraction of alphabetic characters, repetition ratios, boilerplate or blocklisted terms, and perplexity under a reference model. They are cheap on CPU and fully auditable, and every threshold is a knob you can explain to a regulator or a model-behavior reviewer. Classifier filters score each document with a trained model: a fastText or transformer quality classifier (often trained to recognize "educational" or instruction-like text, as in FineWeb-Edu and the DataComp-LM curation), plus safety classifiers for toxicity, CSAM signals and policy categories. These can capture learned quality signals beyond a hand-written regex. But corpus-scale cost depends on the CPU/GPU implementation you select, and their decisions can be harder to interpret than explicit rules. Their false-positive rate silently shapes the model: a quality classifier that down-weights a dialect, a domain or a non-English register is making a model-behavior decision, not a data-cleaning step.

Fresh crawls also carry a lot of machine-generated text. Pangram Labs sells an AI-text detector, and in a September 2026 preprint with the University of Maryland it ran that detector on web data that had already passed FineWeb-style quality filtering. The detector labeled 27.5% of June 2026 tokens and 31.1% of August tokens as AI-generated. Those are detector labels, not confirmed authorship. Across 800 small models of up to about 1B parameters, mixing such text into a large human-text budget pushed loss on human-written validation data up almost at once. It helped only data-starved runs, and only early on. Budget compute for a detector pass next to your other classifiers, store each document's score so the policy can be re-cut later, and hold out a human-only validation set alongside the mixed one.

For the technical stratum, A rejects 8/1,000 = 0.80% and passes. B rejects 35/1,000 = 3.5% and fails. On the point estimate the crossover is 20 eligible rejections; 21 fails even if the pooled global score improves. A 1,000-record sample is noisy, though. To show the true rate is below 2.0% at 95% confidence (exact binomial, one-sided), the sample may hold at most 12 rejections; A's eight put the upper bound at about 1.4%, so A passes that test too. Between 13 and 20 rejections, a candidate meets the sample limit yet cannot prove the population rate sits below 2.0%; settle with the data owner, before scoring, which of the two tests gates the release. Before you release a real corpus, run the same gate on every required stratum, test the retained corpus for quality and eligibility, and report where the adjudicators disagreed.

Compare dedup scope and filter thresholds at a fixed token budget, tokenizer and evaluation protocol, and inspect source and language retention as well as model outcomes. Preserve excluded IDs and reason codes under authorized retention so you can regenerate the corpus after an approved policy change. FineWeb's controlled curation comparisons supply the experimental method. Chapter 10.10 owns policy, and this chapter owns its execution.

Decontamination protects the meaning of every benchmark you will ever report. If your pretraining corpus contains the test items from GSM8K, MMLU, HumanEval or whatever you benchmark on, your model can memorize the answers and your reported scores are fiction. Benchmark items, MMLU among them, can land in a training corpus verbatim or reworded, and every leaked item lets a model score by recall instead of by generalizing. Test both kinds of overlap against the versioned evaluation registry before you interpret a score. A clean mirror shows the size of the damage: on GSM1k, a fresh human-written mirror of GSM8K, accuracy fell by up to 8 points for the most affected model families, while frontier models showed little drop. The fork is how aggressively you scrub, and against which eval sets.

The standard mechanism is n-gram / substring overlap: index every eval set you care about, then drop or flag any training document that shares a long enough contiguous span (commonly an 8- to 13-gram match) with a test item. It is CPU-bound, heavy on index lookups and conceptually simple. The catch is that it is necessary but not sufficient. Paraphrase, translation and reformatting slip straight past a literal n-gram match, so a corpus can pass decontamination and still be semantically contaminated. What works is procedural rather than algorithmic. Maintain a frozen, versioned registry of every benchmark and decontaminate against the entire registry before training. When you add an eval after training, scan the frozen training corpus referenced by its manifest for overlap. If overlap exists, mark the original score contaminated and use a clean mirror or held-out replacement, then add the eval to the registry and remove overlaps before the next training run. Get it wrong and the cost is a credibility event: an outside party reproduces your eval on a clean split and your numbers collapse.

Tokenization is where the prep cluster's hardware profile reveals itself, and where the most common sizing surprise lives. BPE applies ranked pair merges; Unigram selects the best path through a scored segmentation lattice. Widely deployed production tokenizers are often CPU-oriented and parallelize across documents, but GPU BPE implementations can also parallelize merge work within a document. Benchmark the exact tokenizer, normalization and regex path, document-size distribution, batch size, and available CPU and GPU implementations. Those measurements, not an assumed CPU-only limit, size the tokenization fleet.

At frontier-corpus scale, tokenization is a large compute pass whose wall-clock depends on measured end-to-end throughput. The decision is whether to size a dedicated CPU pool, use a GPU implementation, or lean on a managed distributed framework (Spark, Ray, Dask, or NeMo Curator's streaming executor that overlaps CPU and GPU stages). Choose by throughput, cost and pipeline compatibility on the real corpus.

The prep supercomputer is sized against the opposite constraints from the GPU fleet. The headline is that CPU-bound extraction and tokenization can make prep CPU-dense, RAM-heavy and storage-bandwidth-bound. GPU dedup or classification changes that mix, so size each stage's compute, memory, storage and network before you call the hall power-light. A CPU-heavy prep rack can fit an air-cooled row where a dense training GPU rack needs liquid cooling, though its populated CPU/GPU mix still sets the real power and cooling limits. Its scarce resources are CPU cores, DRAM capacity (dedup hash tables and shuffle buffers are memory-hungry), fast local scratch (NVMe for intermediate shuffle spill), and sustained sequential read bandwidth from the object/capacity tier where the raw corpus lives (→ Chapter 9.6). The network it stresses is the data-center fabric for all-to-all shuffles, not a non-blocking GPU back-end.

Two structural decisions matter: dedicated or shared, and co-located or remote. A dedicated prep cluster (or a large pool of general CPU compute) decouples prep wall-clock from GPU availability, so you prepare the next corpus while the current run trains. The cost is standing up and powering a second cluster. Sharing idle GPU-node CPUs is capital-efficient, but it couples your prep schedule to GPU idleness and wastes accelerators. On location, prep wants to run where the data already is. Raw corpora are multi-petabyte, and data gravity makes them economically immovable: at 2026 list prices, moving 1 PB out to the internet costs about $90k at the first-tier rate and about $54k at AWS's published volume tiers, and negotiated rates are lower. Prep is therefore the textbook case of move compute to data, the same gravity logic developed in Chapter 9.8.

The stage table below shows which resource sets each duration. Its rates are the assumptions above, and a faster isolated kernel changes the release date only if it shortens the limiting term.

The same prep failures recur, because each comes from treating prep as ETL rather than as a sized supercomputer:

• Tokenizing without implementation benchmarks. Teams size a CPU pool or borrow GPU nodes before benchmarking the exact normalization, regex and tokenizer pipeline. The wrong hardware mix leaves either CPU cores or accelerators idle.
• Treating a post-training scan as a repair. When a new eval overlaps the frozen training manifest, mark the original score contaminated and use a clean mirror or held-out replacement. Add the eval to the registry and remove overlaps before the next training run.
• Treating dedup threshold and classifier operating point as hygiene, not hyperparameters. Teams pick a similarity threshold and a quality-classifier cutoff because they sounded reasonable, never ablate them, and discover a quality regression or a silent capability gap after a frontier run. Global dedup that deletes the good recurring web is the canonical case.
• Sizing the prep cluster with GPU-fleet instincts. Liquid cooling, a non-blocking back-end and accelerator-first design, bought before the prep stages are measured, can overspend on idle GPUs while starving the cores, DRAM, scratch and read bandwidth the CPU-bound stages need. Derive each resource from the complete pipeline budget. → Chapter 9.8.

Choose the prep resource mix by the stage that controls delivery, and the curation policy by accepted output quality at a fixed training budget. Keep corpus release behind both gates: finishing early with an untraceable or over-filtered dataset does not feed the intended run. Under-sizing memory or scratch delays the program, and accepting an untested exclusion policy spends the GPU budget on a different corpus than the one the program meant to train.