Network Fundamentals & AI Traffic Characterization

In a single-GPU world the network is an afterthought: feed data in, ship results out. In a frontier AI cluster the interconnect is a first-class part of the computer, and for the largest synchronous jobs it gates everything else. A modern training run is one program spread across tens of thousands of accelerators that must agree, repeatedly and synchronously, on the value of a shared set of gradients. That agreement is a collective communication: an all-reduce, all-gather or reduce-scatter. Until it completes, every GPU that has finished its share sits idle at a barrier, waiting for the slowest participant. So the network does not carry bytes between the computations; it sequences the computation itself. Misjudge it and the expensive accelerators run at a fraction of their FLOPS, burning wall-clock while they wait on each other.

The rest of Part 8 builds on the primitives set here. First comes the three-network model (scale-up, scale-out, scale-across) and why its boundaries are moving. Next are the collective primitives and how each parallelism strategy maps onto a network tier. Then the tyranny of the tail and the BSP barrier, which turn a one-in-a-thousand slow link into a fleet-wide stall, and the gap between raw link rate and delivered goodput, which decides whether a fabric justifies its capital. Last are the two traffic profiles, training versus inference, and job-completion-time as the north-star metric that reframes every downstream networking decision.

An AI cluster is not one network. It is a hierarchy of at least three, each with its own bandwidth, latency, reach and cost per bit, and the most consequential design decision is where you draw the boundaries between them. The hierarchy spans roughly three orders of magnitude in per-GPU bandwidth and four in latency. Traffic that lands on the wrong tier is either ruinously expensive or ruinously slow.

Scale-up is the memory-semantic fabric inside a tightly coupled domain. Historically that was a node. Today it includes 72-GPU NVL72 racks (GB200, GB300 and, shipping since June 2026, Vera Rubin), and NVIDIA’s March 2026 forecast adds a 576-GPU Rubin Ultra domain across eight racks. Each kind of domain (node, NVL72 rack, Rubin Ultra domain) has its own topology and software model. It is the fastest and most expensive tier. NVLink 5 on GB200 and GB300 delivers ~1.8 TB/s per GPU bidirectional (~0.9 TB/s per direction), so the rack carries ~130 TB/s of aggregate injection bandwidth. Vera Rubin’s NVLink 6 raises that to 3.0 TB/s per GPU and 216 TB/s per rack. Supported peer load/store operations let a GPU read or write another GPU’s memory without a host-staged message, though the platform’s ordering rules still apply and remote access does not imply local latency or cache coherence. Tensor-parallel and expert-parallel traffic belongs here, because those collectives are bandwidth-hungry on every layer. → Chapter 8.2.

Scale-out is the packet-switched fabric that stitches domains into a cluster of tens of thousands of GPUs. It is roughly an order of magnitude slower per GPU: GB300’s 800 Gb/s ConnectX-8 (100 GB/s per direction) is ~1/9th of NVLink 5’s ~0.9 TB/s per direction, and GB200’s 400 Gb/s ConnectX-7 (50 GB/s) is ~1/18th, before overhead. The semantics are message passing (RDMA over InfiniBand or RoCE/Ethernet). This is the natural home for data-parallel gradient all-reduce and pipeline-parallel point-to-point traffic. → Chapter 8.4 and Chapter 8.5.

Scale-across is the newest tier and a creature of the power-bound era: the WAN/DCI fabric that couples multiple campuses when no single site can be energized at the scale a frontier run demands. Crossing a site boundary does not select the algorithm. Measured RTT, bandwidth and the job’s communication pattern do. A tested cross-campus or intra-metro path can preserve synchronous methods, while longer regional paths move the design toward hierarchical or relaxed-synchrony methods, and toward compression when the bandwidth budget demands it. Power scarcity can push a frontier lab toward multiple campuses when one site cannot supply the required compute, but the protected-route budget must still admit each workload. The grid constraint does not make synchronous traffic fit a WAN. → Chapter 8.8; the power-bound rationale is in Chapter 16.1.

Around these three sits a fourth network that every deployment has and no fabric budget should forget: the front-end network, the CPU-side Ethernet that carries data ingestion, checkpoint traffic, storage access, user/API serving and cluster services. Meta describes its training clusters as running two independent networks, a front end for ingestion, checkpointing and logging and a back end for training. NVIDIA’s reference architectures split the same functions into distinct storage and in-band management fabrics beside the compute fabric (Meta, SIGCOMM 2024; NVIDIA reference architectures, 2025). The front end is ordinary Ethernet in kind but not in size, and sizing it is a live decision with an order of magnitude inside it. Its rate follows concurrent ingestion, checkpoint save/restore, KV movement and service traffic, together with their deadlines. Count the NIC ports actually attached to storage and the uplinks shared with management; dividing a tray aggregate by GPU count does not establish workload demand. A reference configuration is useful only with its host connections, storage-device connections and oversubscription boundary intact. The NVIDIA GB200 reference architecture describes those as a separate fabric. Front-end sizing lives with Chapter 9.2 and Chapter 9.3, and management recovery with Chapter 8.7.

Distributed training is built from a small vocabulary of collective communication primitives, and almost all the traffic in a cluster is one of them. All-reduce sums a tensor across all ranks and returns the result to all; it is the workhorse of data-parallel gradient synchronization. All-gather and reduce-scatter, the two halves a ring all-reduce is composed of, move sharded weights and activations under tensor and sharded-data parallelism. All-to-all is the signature of mixture-of-experts: each token is routed to its selected experts, and the byte matrix can be sparse and skewed, stressing a destination or a cut rather than loading every path uniformly. Point-to-point sends move activations forward and gradients backward across a pipeline stage.

The engineering content is the mapping: which parallelism dimension generates which collective, and which tier can carry its exposed exchanges within budget. Get the mapping right and each collective lands on a fabric sized for it. Get it wrong and a bandwidth-bound collective lands on a tier that cannot feed it, and the job stalls. The four parallelism dimensions are nested so that they map onto the bandwidth hierarchy.

The table is a placement hypothesis to test. Meta’s Llama 3 405B run gives it a published anchor. Meta ordered its parallelism dimensions [TP, CP, PP, DP], innermost to outermost, to suit the network. Tensor parallelism (TP=8) stays inside the NVLink server, while pipeline parallelism (PP=16) and FSDP data parallelism run outermost, across the multi-hop fabric. Holding TP and PP constant, Meta measured BF16 MFU of 43% on 8,192 H100s and 41% on 16,384. Its 131k-token-sequence layout added 16-way context parallelism and reached 38% (Meta, July 2024). Those drops mix communication, batch and sequence effects, so they show a direction, not a network-only cost.

The design move is to keep the highest-frequency rows (TP, EP) inside the scale-up domain, where the local NVLink fabric’s bandwidth can keep the collective off the critical path when the chosen schedule overlaps its completion. The lower-frequency rows (PP, DP) can spill onto the cheaper scale-out fabric when their measured timing and overlap tolerate lower bandwidth or oversubscription. Domain size therefore dominates fabric design: each GPU added to the domain can keep another chunk of TP/EP traffic off the slower fabric when the runtime places that state and exchange locally. The same logic reshaped fabric design for mixture-of-experts inference. Wide expert parallelism benefits when its exposed dispatch/combine traffic stays on NVLink, but enough expert capacity and a tested hierarchical NVLink/RDMA schedule can justify crossing domains. → MoE inference and EP degree in Chapter 8.2.

One fact separates AI networking from every other kind of data-center networking: a synchronous training step is a bulk-synchronous-parallel (BSP) computation, and a BSP barrier runs at the speed of its slowest participant. When ten thousand GPUs all-reduce a gradient, the collective is not done when the median link finishes. It is done when the last one does. Average bandwidth sets how long a bandwidth-bound collective takes, but the tail decides when the barrier actually closes. A fabric with superb median latency and a fat 99.9th-percentile tail can deliver worse training throughput than a fabric with mediocre median latency and a tight tail. The tyranny of the tail inverts the intuition every web-scale network engineer brings to the problem.

The traffic pattern depends on the collective algorithm. Parameter-server aggregation and some tree/reduction fan-ins produce synchronized many-to-one incast, while ring all-reduce produces synchronized neighbor flows with a long dependency chain, so its slowest transfer stretches the collective tail. The algorithm’s traffic matrix determines whether switch buffers, flow control (PFC) or packet loss drive that tail. One slow link, one congested switch or one mis-tuned congestion-control loop can push the collective past its budget of tens to hundreds of milliseconds. Over a months-long run, that overrun recurs on every step.

Meta’s production RoCE fabric shows how large the effect gets. When jobs landed on partly allocated racks, static path pinning piled traffic onto a few rack-switch uplinks, and the congestion degraded training performance by more than 30% at worst. Spreading flows with enhanced ECMP and more queue pairs improved AllReduce by up to 40% over baseline ECMP (Meta, August 2024). Those gains belong to Meta’s topology and collectives; the job’s own step trace decides whether removing a delay recovers enough accelerator time to justify the fabric change. What transfers between clusters is the mechanism: barrier-coupled work turns a fabric defect into fleet-wide idle.

An 800 Gb/s port does not deliver 800 Gb/s of useful gradient synchronization. Protocol overhead, congestion, retries and barrier wait stand between the rate on the invoice and the training run’s wall-clock, and a faster port justifies its price when useful communication advances the job sooner. Six terms need separating. MAC link rate is the Ethernet service rate, distinct from coded SerDes signaling. Payload throughput is useful delivered bytes divided by transfer time in a named direction; payload efficiency divides that rate by the corresponding MAC byte rate. Collective algorithm bandwidth is the message size divided by operation time; NCCL bus-bandwidth normalization depends on operation and ranks and is not switch telemetry. MFU divides useful model FLOPs by elapsed seconds times the participating accelerators’ rated peak FLOPs/s at the chosen precision. Exposed communication is dependency delay after useful overlap, divided by step time when reported as a fraction. Job goodput is the useful work delivered within the job’s quality and service limits; as a fraction, it is the share of wall-clock time that advances the job across the checkpoint/recovery window you define. Unlike MFU, it says nothing about peak FLOPS. The operations tally lives in Chapter 14.1.

The protocol choice is the cleanest illustration. InfiniBand’s effective throughput and latency belong to the tested adapter, switch, collective and message size, and a switch-hop latency cannot stand in for application completion. Untuned RoCEv2 over Ethernet can show materially higher and more variable latency with a long tail when congestion or reordering triggers the NIC’s retry behavior. Go-back-N receivers repeat traffic that a receiver with supported selective retry or out-of-order placement can avoid. NVIDIA’s 2024-10-28 Colossus announcement reports roughly 95% data throughput for Spectrum-X while training Grok; Chapter 8.4 weighs that vendor figure against its baseline. Meta’s 2024-08-05 production account describes the topology, routing and congestion tuning that made RoCE viable for its distributed training. The lesson is that tuning and topology justify their cost when they shorten the same job’s exposed communication, and a faster link justifies its cost when that job is still bandwidth-limited. The NCCL-tests definitions keep collective comparisons on the same denominator. → transport qualification in Chapter 8.4.

For n=64, each rank sends W=2(n−1)S/n=2×63/64×4 GiB=7.875 GiB=8,455,716,864 bytes. Healthy communication is W/(45×10⁹ bytes/s); step time is 800 ms + max(0, W/(45×10⁹)×1,000−150 ms), or 840 ms at 10 ms precision: pass. One lost uplink leaves R=45×31/32 GB/s, and the same expression gives 840 ms at 10 ms precision: pass. The unrounded times differ although the displayed results coincide. Keep TP local and admit DP only after the built fabric sustains the assumed tail service; this step alone does not justify a faster NIC.

The flip is growth on a fixed surviving cut. With n/2 senders, R(n)=31×45×2/n GB/s and communication takes 2(n−1)×4×2³⁰/(2×31×45×10⁹) seconds. That must be no more than (1,100−800+150) ms. The crossover is about 147 ranks: the largest balanced even group that passes is 146, and 148 fails. Extra ranks need extra leaf ports, so this sensitivity does not enlarge the original two leaves. Fewer bytes or more overlap moves the crossover outward, and so does a faster NIC. On HGX B300 or GB300 NVL72, where ConnectX-8 gives each GPU 800 Gb/s, a fabric built to match halves the communication time: DP hides inside the 150 ms overlap (800 ms healthy and degraded), and the crossover moves to about 293 ranks. The NCCL-tests method supplies normalization; Chapter 8.5 tests spine loss and counts expansion.

Training and inference often produce different traffic, but the label does not select the fabric. Synchronous collectives are bursty and bisection-hungry, so measured step-time and MFU targets often justify high bisection. Many local inference requests permit upper-tier multiplexing, while distributed MoE, KV movement and prefill/decode disaggregation can require substantial scale-out capacity. Published fabrics span 1:1, 2:1–3:1 and reported 7:1 examples. Chapter 8.5 turns those observations into a per-tier ratio through its locality, exposure and failure-headroom tests; this chapter supplies the traffic characterization those tests read.

The 2025–2026 twist is that inference traffic is no longer trivial. Reasoning models emit long decode sequences, mixture-of-experts routing creates real all-to-all traffic, and prefill/decode disaggregation moves the KV cache across the fabric. Inference, once a single-node afterthought, now exercises the scale-up domain hard. Its characterization still differs from training (request-SLO-bound, with synchronization inside distributed kernels), but the gap is narrowing, and a fabric scoped for last-generation single-node inference can be undersized for MoE reasoning serving. → inference fabric demands in Chapter 1.3; MoE all-to-all in Chapter 8.2.

The transfer allowance is 250−180=70 ms. Eight requests carry 8×1 GiB=8,589,934,592 bytes. Across racks, completion is 180 ms +8,589,934,592/(45×10⁹)×1,000 ms, about 370 ms: fail. With local service of 300 GB/s it is about 210 ms: pass, rounded to 10 ms. Choose local prefill/decode placement, accepting its memory and scheduling constraints to avoid the request-tail miss.

Batch size reverses the choice: B×2³⁰/(45×10⁹) ≤0.070 gives B≤2.93. Two concurrent transfers give about 230 ms: pass; three give about 250 ms but fail the unrounded limit. Choose cross-rack placement at two transfers or fewer, or acquire more service capacity. Hand Chapter 9.7 the KV representation, bytes/request, burst, ownership-transfer event and time allowance; it owns KV sizing. The Dynamo disaggregated-serving design supplies the mechanism. Validate the joint request tail in Chapter 13.7.

Every networking decision in Part 8 ultimately answers to one metric, and it is not bandwidth, latency or port count. It is job-completion-time (JCT) for training, and its inference twin, tokens-per-second-per-dollar at the SLO. JCT is the wall-clock from job start to a trained model, and it folds in everything the spec sheets hide: the tail, the congestion, the goodput, the failures and restarts. A fabric is good if and only if it lowers JCT per dollar of capital and per watt of power. A fabric can have best-in-class link rate and best-in-class median latency and still lose on JCT because its tail or its reliability is poor.

Reframing networking around JCT changes what you optimize. You stop chasing peak bisection bandwidth as a vanity number and start chasing the combination that minimizes wall-clock: a tight tail, high goodput, and fast recovery from the inevitable link and switch failures (which the Llama-3 run attributed ~8.4% of interruptions to). That reorders the priorities of the whole Part. Congestion control, in-network reduction, adaptive routing and topology matter because they move JCT, not because they look impressive in a bake-off. The reliability chapter makes the same shift from availability to goodput for the facility: the question is never ‘is the link up?’ but ‘how much useful training did the fabric deliver this week?’ → goodput vs availability in Chapter 12.2; checkpoint/restart math behind failure recovery in Chapter 9.4.

Choose the tier boundary from the exchanges that remain exposed, then carry healthy and degraded deadlines into the fabric order. Keep a group local when crossing fails that budget; split it when memory capacity demands the crossing and the measured schedule can afford it. Choosing on port rate alone buys capacity the job may never use while leaving its slowest dependency unpaid for.