Cluster-Scale Benchmarking, Reference Training & Storage/Scheduler Validation

By the time a cluster reaches this chapter, it has been tested in pieces. The power chain showed its failure modes under load (Chapter 13.6), the fabric was commissioned link by link with point-to-point bandwidth and PTP time-sync gates (Chapter 13.7), and every node finished the signed burn-in campaign, diagnostics and silent-data-corruption screen, sized from test intensity, fleet exposure and the measured defect-discovery stopping rule (Chapter 13.8). All of it passed. None of it shows that the machine you bought, a single tightly coupled supercomputer, works. This chapter covers cluster-scale validation: the unit under test is the whole system, and the question changes from "does each part meet spec" to "does the assembled system deliver the goodput the contract assumes."

Validation runs in three widening rings, and a ring does not open until the one before it clears. Ring one is synthetic collective benchmarking, NCCL and OSU at scale. It isolates the fabric and the communication libraries from application noise and gives you a hard, repeatable busbw number to gate on. Ring two is subsystem acceptance: storage (checkpoint write, data-loader read, metadata) and the scheduler or orchestrator (gang placement, topology awareness, preemption), each against its own bandwidth and correctness bar. Ring three is a reference (proxy) training run that combines all of it: a real model, real optimizer state and real checkpoints on real storage, scheduled the way production will schedule it, and run long enough to surface stragglers, thermal drift and slow leaks that a five-minute benchmark never sees. For training, ring three supplies measured goodput for the SLA layer you committed to. For inference, it supplies correct request throughput and tail latency for the agreed traffic mix. The earlier correctness, safety, isolation and recovery gates stay independently binding.

The fabric was already proven point to point in Chapter 13.7: each accepted path meets its physical and payload gates, every cable reconciles to the map, and FEC and counter data sit inside their profile. That proves the wires. It does not prove the collective, which is what training runs on every step: all-reduce, all-gather, reduce-scatter and broadcast across the full job. A fabric that is flawless link by link can still collapse on an all-reduce because of one mis-seated transceiver three hops away, a routing imbalance that congests a rail, an adaptive-routing misconfiguration, or a SHARP/NVLS offload that failed to engage. The collective benchmark is the only test that exercises the fabric the way the workload will.

The standard tool is the NCCL tests suite (all_reduce_perf, all_gather_perf and relatives), run as a multi-node MPI or SLURM job across the full cluster, sweeping message sizes from a few KB to multiple GB. Gate on bus bandwidth (busbw), not algorithm bandwidth, because busbw normalizes for the collective's data-movement factor. Comparing it with a hardware ceiling still requires the same operation, rank count and offload convention. Read busbw two ways. The large-message plateau tells you whether the fabric reaches its bandwidth ceiling. The small-message latency floor tells you whether the collective's fixed cost stays bounded, and that fixed cost governs strong-scaling efficiency.

One provider ran a 72-rank all-reduce sweep inside a healthy GB200 NVL72 domain and measured ~806 GB/s busbw at 2 GiB, rising to ~928 GB/s at 32 GiB (Crusoe's GB200 NVL72 NCCL validation guide, 2026). Only the biggest buffers lift busbw above 900 GB/s/GPU, the NVLink5 unidirectional rate. In nccl-tests, busbw is algbw × 2(n-1)/n (the ring convention), so a figure above the per-GPU one-way NVLink rate implies NVLink SHARP is reducing inside the switch. The source does not state the NVLS setting, so that part is this guide's inference. The scale-out, multi-rack number over InfiniBand or Spectrum-X is a separate gate, derived from the NICs, active rails, topology and oversubscription, protocol efficiency, message-size plateau and collective data-movement model.

Two companion suites round out ring one. OSU Micro-Benchmarks (osu_bw, osu_latency, osu_allreduce) give a vendor-neutral, MPI-level cross-check on NCCL's numbers, useful because they take a different code path. If the two disagree, compare transport, message size, rank placement and routing before assigning the fault. The suites take different paths through the same fabric, so a healthy NCCL sweep does not clear the fabric and a slow OSU run does not convict the collective library. MLPerf Training (and, for storage, MLPerf Storage, below) provides an industry-comparable, full-workload reference if the operator wants a number that benchmarks against published results rather than only against the cluster's own reference. The current round is MLPerf Training v6.0 (June 2026), the first with mixture-of-experts benchmarks, including DeepSeek-V3 671B; CoreWeave trained it in ~2 minutes on 8,192 GB300 NVL72 GPUs. MLPerf is heavyweight to run correctly, and whether it pays for itself depends on the operator. For a neocloud selling capacity, a clean MLPerf submission is a marketing-grade external attestation that an internal NCCL number is not.

NVIDIA's Exemplar Cloud program (announced May 2025) is a third, vendor-side comparable. A provider's cluster runs NVIDIA's benchmark recipes and must reach at least 95% of NVIDIA reference-architecture throughput to pass. Configuration alone, NVIDIA says, routinely leaves partner deployments 8–12% short of the matching reference architecture. Typical culprits are Grace-VM command-queue virtualization, C-state and NUMA settings, NCCL_IB_QPS_PER_CONNECTION, and a topology file that was never mounted into the container. Its worked examples span a wider range. A GB200 MoE job in a VM trailed by 12–14%. A 512-GPU GB300/ConnectX-8 cluster ran 31% slower per iteration. On B200 VMs, an NCCL topology file that never made it into the container cost 13–53%. Part of the 31% comparison reflects a different NCCL version. That case also saw AllGather busbw climb from ~28 to ~61 GB/s after QPS went from 1 to 4. It is a vendor reference-architecture gate, not a substitute for ring three's soak of the owner's workload or for the goodput SLA. Neoclouds now collect the Exemplar stamp as a go-live trophy: IREN's Horizon 1 was Microsoft-accepted and GB300 Exemplar-validated in the same week (Aug 2026). → external-rating context in Chapter 12.2.

Open vendor tooling can now script much of rings one and three. NVIDIA open-sourced its Cluster Readiness Engine in September 2026 under an Apache-2.0 license, and announced v0.7.0 at the OCP summit in October 2026. It packages topology-aware NCCL sweeps, level-4 DCGM diagnostics and a NeMo reference pretraining job. To find stragglers it keeps splitting the failing groups, which is the ring-bisection hunt above done by machine. It only reports and ships no pass thresholds, so the floors remain the owner's contract terms. Cordoning, quarantine and return to service remain the operator platform's job, so the acceptance package should say who makes that call. For NVL72-class domains, DCGM 4.7 brings multi-node NVLink checks (nvloom, mnnvbandwidth) plus a health watch on the IMEX domain. Record the DCGM version and the IMEX state alongside the busbw sweep. On Kubernetes, also log the pinned software recipe, for example an NVIDIA AI Cluster Runtime (AICR) recipe version. A recipe's signed evidence comes from the hardware NVIDIA tested it on, not from this cluster. AICR can also validate the running cluster against the recipe and record signed evidence of that result, and that record is the one that belongs in the acceptance package.

The suites above are NVIDIA-path tools. The table below carries the same device, workload and recovery evidence onto AMD Instinct, Google Cloud TPU and AWS Trainium, where the diagnostic suite differs and, on a managed service, the provider rather than the customer owns physical acceptance.

Storage is where cluster acceptance most often goes wrong, because operators test it as one thing when it is three. For training those are write, read and metadata, each with its own failure mode and bandwidth bar; an inference cluster adds a fourth path, KV cache and retrieval, covered below. → architecture in Chapter 9.5 and Chapter 9.6; checkpoint math in Chapter 9.4.

The write path is checkpointing, and it gates failure recovery. When a synchronous training job checkpoints, every rank drains optimizer and model state to the parallel filesystem in a burst, and the whole job stalls until the slowest writer finishes (unless asynchronous or multi-tier checkpointing overlaps it). The acceptance question is sustained aggregate write under a realistic concurrent burst, not peak write bandwidth, because that sets how long the job is frozen every checkpoint interval. VAST's survey of 40 production runs inferred model sizes from an assumed ~14 bytes per parameter of checkpoint state. That total is what standard mixed-precision Adam training writes: 2 bytes of BF16 weights, a 4-byte FP32 master copy and two 4-byte FP32 Adam moments. Frameworks that quantize optimizer state to 8 bits, or that keep no separate BF16 weight copy, write fewer bytes, so size from the tensors your own framework serializes. Size the write path from quantities kept separate: checkpoint interval, serialized bytes, the synchronous pause every rank takes, the asynchronous drain that follows, the point of durable completion, restore time after a failure, and the training overhead the contract permits. A write subsystem that benchmarks beautifully on a single stream but collapses when 128 nodes write at once fails the test that matters. MLCommons gave MLPerf Storage v2.0 (Aug 2025) a dedicated checkpoint benchmark because failures come often at scale: its illustrative model of a 100,000-accelerator cluster at full utilization puts them roughly 30 minutes apart. At that rate, checkpoint write and restore bandwidth decides whether each failure costs minutes of lost work or hours.

The read path is the data loader, and it gates steady-state goodput. If the loader cannot feed the GPUs, they idle: invisible to a fabric benchmark, fatal to MFU. The bar is a sustained per-GPU read throughput target met at full node count (not a four-node extrapolation), validated by replaying the actual loader or by MLPerf Storage's training workloads. The checkpoint write and restore budgets from Chapter 9.4 are separate gates: an aggregate read number or a write-to-read ratio hides hot-spotting and cannot set the recovery deadline.

The metadata path is the silent killer. Lots-of-small-files (LOSF) workloads, such as image datasets and sharded tokenized corpora, stress metadata operations and small-file IOPS rather than bandwidth. A parallel filesystem that delivers terabytes per second of sequential read can still fall over on a million-file-per-second metadata storm. Put mdtest and a representative LOSF replay in acceptance, because LOSF degrades scaling in a way no large-block fio run will reveal.

Inference adds a fourth path: KV cache and retrieval. A cluster that will serve inference needs its KV-cache offload tested for read and write latency and throughput at the target concurrency. Retrieval workloads need vector-query latency at a stated recall. Chapter 9.7 covers the tiering. The next round, MLPerf Storage v3.0 (Sept 2026), brought KV-cache and vector-database workloads plus an S3 access layer. Test through whichever protocol production will use, POSIX or S3, not the one that scores best.

Checkpoint example: trace and policy choice. A snapshots at t = 0, releases at 2 s, drains at 42 s and commits at 45 s. Drain throughput is 80/40 = 2.0 GB/s; multiplying by eight ranks would count the state twice. Under the initial 120 s period, B snapshots at 120, releases at 122 and commits at 165 s. A loss at 130 s rejects uncommitted B and restores A: age 130 − 0 = 130 s passes. But a loss at 164.9 s still restores A, now 164.9 s old: FAIL. A favorable event does not accept the policy.

Apply 9.4's bounds: period S + 2 + 40 + 3 ≤150 s requires S ≤105 s, and 2/S ≤0.020 requires S ≥100 s. Choose S = 100 s: B snapshots at 100, releases at 102, drains at 142 and commits at 145 s. At a loss at 130 s, B is still uncommitted and A remains recoverable outside the failed node. The worst pre-commit age approaches 145 s, within 150 s, while the pause fraction is exactly 2.0%. The 45 s completion lag is below the period, so drains do not backlog. This spends the whole pause allowance for 5 s of age margin, so faster capture is required if measurement or interference raises the pause. A 105 s period uses all the age margin; 106 s fails age, while 99 s fails pause.

Restore every shard from A and validate hashes, optimizer, RNG and loader state, and the next step. Restart = 5 + 10 + 20 + 5 = 40 s ≤45 s; reload = 80/20 = 4.0 GB/s. A 25 s reload reaches 45 s; 26 s makes 46 s and fails. Only the chosen policy's simulated timing passes. Releasing the installed cluster still waits on tests of durable placement and manifests, kill/requeue timestamps, correctness and sustained interference, and on the open installed-fabric checks in 13.7. Storage owns durability; scheduling owns placement; the workload owns correctness. On corrupt state, quarantine the output and restore the known-good generation. Replay with concurrent rebuild and collective traffic, and restore the baseline. PyTorch DCP distinguishes staging from asynchronous completion; obtain the actual store's proof of atomic publication and its failure domains.

The scheduler is the subsystem most likely to be hand-waved at acceptance ("SLURM works, ship it"), and the one most likely to strand the fabric you just spent a chapter validating. On rack-scale, NVLink-domain hardware, a scheduler that places a job's ranks without respecting topology will scatter a tightly coupled job across NVLink domains. Collectives that should have stayed on the 130 TB/s in-rack fabric then spill onto the comparatively narrow scale-out fabric. The benchmark busbw is fine; the delivered bandwidth to a real job is not, because the scheduler fragmented the domain. Topology-aware, gang-scheduled placement is therefore an acceptance criterion, not a tuning knob deferred to operations.

The 2026 choice forks along a familiar line. SLURM dominates dedicated training clusters (the 70% trade-press estimate has no defined population denominator). It brings parallel-job allocation, optional time-sliced gang scheduling (test it separately), fair-share and QoS priority, preemption and, critically for Blackwell, block scheduling that allocates whole NVLink domains through a topology.yaml so coherent-memory jobs land aligned. Kubernetes (with KAI Scheduler, Run:ai, Volcano or the SLURM-on-K8s bridge) brings multi-tenancy, fractional GPUs and a cloud-native control plane. The cost is that you need a gang scheduler. The default kube-scheduler admits pods one at a time and can start half a gang and deadlock, and the native Workload/PodGroup gang scheduling added in Kubernetes 1.35 reached beta in 1.37 (August 2026) but is still off by default. Production clusters therefore use KAI, Volcano, Kueue or Run:ai, and your acceptance plan should state which mechanism it validated. Which control plane to validate against is itself an acceptance decision, and for a multi-tenant neocloud the answer is increasingly both, converged. → the build-out treatment is in Chapter 10.1 and Chapter 10.2; here the bar is correctness, not architecture.

This is the acceptance test the whole chapter builds toward, because it is the only one that runs the machine as a machine. A reference training run uses a real model of representative scale (a GPT- or Llama-class transformer, often a deliberately sized proxy rather than a frontier model). It advances through the canary and partial-fleet promotion gates of Chapter 13.10. Only after full-fleet promotion does its sustained acceptance window begin, across the full cluster, through the production scheduler, checkpointing real optimizer state to production storage. It does not run to convergence. It runs to measure goodput and surface the slow failures that short benchmarks cannot see: thermal drift as the room heats, a single GPU that clocks down after six hours, a memory leak in the loader, a checkpoint that slows as the filesystem fills, a straggler that appears only under sustained collective pressure.

The run produces a defensible performance number: ML Productivity Goodput, computed with Chapter 14.1’s ledger over the declared wall-clock window and Chapter 12.2’s metric boundaries. The SLA rests on goodput rather than on busbw, storage GB/s or a scheduler dashboard showing green, because only goodput shows how training time and compute combine across those layers. With the reference workload pinned, the run can measure every layer (scheduling, runtime and program, the last as MFU) and report full ML Productivity Goodput. When the customer owns the model code, a provider usually contracts only for Provider Goodput (scheduling × runtime) and reports the MFU layer for context, as Chapter 12.2 sets out. Correctness, isolation, recovery and safety keep separate pass/fail gates. The acceptance gate is whether sustained goodput over the reference window clears the contractual floor, and whether the residual badput breaks down into causes you understand and can attribute.

Without attribution, a missed SLA becomes a dispute among the scheduler, storage and platform teams. Assign every acceptance shortfall with 14.1's useful-output ledger and 12.2's metric boundaries. Scheduling owns admission; reliability and storage own interruption and restore; the platform owns kernel and fabric performance. MFU, the program layer, sits with whichever party owns the model code, and the contract has to name that party. Record the workload, placement, precision, window and each badput event against the contractual floor, so the remedy follows the boundary that failed.

Two published reference points show why the run needs both an injected failure and a named goodput layer. SemiAnalysis reported a typical MTBF of roughly seven days for one 512-H100 cluster at a top-tier operator (October 2024). If interruptions arrived at that rate as a Poisson process, a 48-hour soak on a cluster that size would see none about three times in four (e^(−2/7) ≈ 0.75), so recovery has to be proven with the injected node failure, not by waiting for a real one. Meta's Llama 3 report (Table 4, July 2024) gives 41% BF16 MFU for 405B pre-training on 16,384 H100s at 8K sequence length. On a run like that, ML Productivity Goodput is less than half of Provider Goodput, which is why the contract must name its layer. The MTBF belongs to one cluster and is no per-GPU rate; the MFU belongs to one run and is neither a Hopper band nor a GB300 gate. Set the acceptance window and the goodput floor from the accepted fleet's own interruption history and model configuration.

Cluster-scale validation delivers a signed acceptance package: the contractual baseline and the seed of the day-2 reliability program. It contains the full-scale NCCL/OSU busbw sweep with the straggler-hunt log and the final node-exclusion list; the storage results for the write, read and metadata paths with the per-GPU targets met; the scheduler validation, including the preempt-checkpoint-resume trace; and the reference training run's sustained goodput curve with its badput breakdown, or the inference run's offered and accepted traffic, errors, and TTFT, inter-token and completion latency tails. It also captures the fabric and node baseline, the busbw and per-node performance fingerprints, so day-2 operations can detect drift against a known-good reference instead of guessing. → the baseline handoff into operations and the operational-readiness gate live in Chapter 13.10; the failure-rate data this seeds feeds Chapter 14.3.

Inference example: trace, action and flip. Reconcile 9,950 + 50 = 10,000 offered requests; completion fraction = 9,950 / 10,000 = 99.5%, above 99.0%. The uncertainty-bounded tails are 191 ms, 48 ms and 7.801 s (report 7.80 s), within their separate gates. The normal-service simulation passes. Release still waits on client and server traces, and on replica-loss and restore tests at the same traffic, with bounded retries, correct output, restored queue and cache state, and tenant isolation. Service owns request accounting; platform owns replica and device events; operations witnesses restoration. Incorrect output or failed isolation aborts the canary; restore the last validated serving build. A 199 ms indicated TTFT reaches the 200 ms bound; at 200 ms indicated, the 201 ms upper bound fails even with 99.5% completion. Then reduce admitted concurrency or repair the measured prefill bottleneck and repeat the same offered trace; do not hide rejected requests by changing the denominator. Hand the service profile to Chapter 13.10. The method follows MLCommons' separation of workload scenarios, correctness and latency constraints, with these owner-assumed gates kept distinct.

The recurring cluster-acceptance failures share one root cause: validating components instead of the system.

• Summing green checkmarks. Every node passed burn-in, every link passed point-to-point, storage passed single-stream fio, the scheduler started a hello-world job, and the cluster is declared accepted without a single full-scale collective or a reference training run. The failures that appear only at the system level (straggler-on-collective, storage-rebuild-versus-restart, scheduler-fragmenting-domains) are the ones that bite in production.
• Peak bandwidth instead of sustained-under-burst. Accepting storage on a hero single-stream number, then watching the checkpoint freeze the whole job for minutes when 128 nodes write at once. Validate the write path under a realistic concurrent burst, not best-case streaming.
• A goodput SLA with no badput decomposition. Committing to a goodput floor without the three-layer attribution model leaves no way to assign a miss to scheduling, runtime or program causes, and the SLA becomes unenforceable.
• Throwing away the baseline. Treating acceptance as a one-time gate and capturing no instrumented full-scale reference leaves day-2 operations with nothing known-good to detect drift against. → Chapter 14.3.

Choose the workload profile whose correctness, durable recovery, isolation and sustained performance all pass through the production stack. A training goodput average can hide a torn checkpoint, and a fast inference tail can hide dropped requests. Holding those failures costs qualification time; releasing them transfers the cost to customer progress and trust.