Congestion Control, Load Balancing & In-Network Compute

Chapter 8.5 sized a topology that, on paper, is non-blocking — enough bisection bandwidth that every GPU can talk to every other GPU at line rate. This chapter is about why that paper number is a lie until you solve three problems that the topology alone does not. AI traffic is the pathological worst case for a packet network: a handful of enormous, long-lived elephant flows per host, all synchronized to the same collective, all hitting the wire at the same microsecond, all converging on the same set of destinations during an all-reduce. There are no mice to fill the gaps and no statistical multiplexing to smooth the bursts. The fabric is either carrying a coordinated stampede or it is idle. That is the regime in which congestion control, load balancing, and in-network compute stop being tuning knobs and become the difference between the bandwidth you paid for and the bandwidth you get.

This chapter works through the lossless-vs-lossy fork (and the PFC pathologies that haunt the lossless path), the DCQCN parameter space that you must tune or buy pre-tuned, the load-balancing fork between flow-hashed ECMP and adaptive/per-packet spray (and the reordering bill the latter sends to the NIC), and in-network compute — SHARP and friends — the rare lever that improves goodput and cuts power at the same time. We close on the telemetry that makes all of it observable, because a congestion problem you cannot see is a goodput problem you cannot fix. → fundamentals and the goodput framing in Chapter 8.1; the protocol/transport choices these mechanisms ride on in Chapter 8.4.

Datacenter congestion control was designed for the web-scale workload: millions of short flows, bursty but uncorrelated, where TCP's loss-and-backoff and a little ECN keep buffers shallow and tails short. AI training inverts every one of those assumptions. A single all-reduce on a 100k-GPU cluster is one logical operation decomposed into thousands of simultaneous, multi-gigabyte point-to-point transfers that all start on the same barrier and must all finish before the next step can begin. The collective runs at the speed of its slowest flow — this is the tail-latency tyranny of Chapter 8.1 made concrete at the packet layer. One congested link, one hash collision, one PFC pause that ripples the wrong way, and the straggler it creates stalls every GPU in the job on the next bulk-synchronous barrier.

That structure produces three distinct failure modes, and the rest of the chapter is organized around defending against each. Buffer pressure and loss from incast — many senders, one receiver, classic during the reduce phase — which the congestion-control loop must absorb. Path imbalance — a few elephant flows colliding on the same physical link while parallel links sit idle — which load balancing must spread. And endpoint overhead — every byte of the reduction traversing the GPU, the PCIe bus, the NIC, and back — which in-network compute can eliminate. Each has its own fork, and each fork has a downstream cost in goodput, power, or lock-in.

RDMA — the zero-copy, kernel-bypass transport that makes GPU-to-GPU networking fast — was born assuming a lossless fabric. InfiniBand delivers losslessness natively with credit-based flow control: a sender never transmits unless the receiver has advertised buffer credits, so packets are never dropped for lack of room. Ethernet has no such mechanism in the base standard, so RoCEv2 borrows one: Priority Flow Control (PFC), a per-priority PAUSE frame that tells the upstream link to stop sending before a buffer overflows. PFC works — and it is the source of the nastiest pathologies in AI networking.

The first is head-of-line blocking. PFC pauses an entire traffic class, not a flow. When a downstream buffer fills, the PAUSE stops every flow in that priority on that link, including flows whose destinations are perfectly idle — innocent bystanders, the "victim flows." The second is the PFC pause storm and, in the worst case, deadlock: because PAUSE is hop-by-hop back-pressure, a congestion hotspot propagates upstream toward the sources, and in a topology with a cyclic buffer dependency (which fat-trees can form under certain failure or routing conditions) the pauses can form a permanent standstill where no switch can drain because every switch is waiting on another. A deadlocked fabric does not degrade — it stops. The defense is a PFC watchdog: a timer on every queue that, if a port stays paused beyond a threshold, drops the stuck traffic and logs it rather than letting the deadlock persist — trading a localized packet loss for fabric-wide survival.

This is why the lossless-vs-lossy question is the master fork of the chapter. Stay lossless and you inherit PFC's pathologies and the operational burden of keeping them caged. Meta's production RoCE fabric famously ran PFC-only at 400G on its back-end, leaning on careful topology and path management rather than a tuned end-to-end loop — a deliberate bet that simplicity beat the DCQCN tuning treadmill at their scale. The Ultra Ethernet Consortium takes the opposite bet: design RDMA to tolerate loss. UEC 1.0 (published June 11, 2025) pairs packet spraying with packet trimming — a congested switch drops the payload but forwards a truncated header so the endpoint learns instantly which packet was lost and retransmits just that one — turning loss from a catastrophe into a fast, surgical signal. → transport semantics and the RoCE in-order penalty in Chapter 8.4; the deep-buffer-vs-shallow switch architecture that shapes how much PFC you need in Chapter 8.3.

DCQCN — Data Center Quantized Congestion Notification — is the end-to-end loop that keeps a RoCE fabric out of PFC pause and away from loss. The mechanism is elegant: a switch experiencing early congestion marks packets with ECN (sets the Congestion Experienced bit) before its buffer is full; the receiving NIC notices the mark and fires a Congestion Notification Packet (CNP) back to the sender; the sending NIC quantizes its injection rate down in response, then probes back up as the congestion clears. Done right, DCQCN holds buffer occupancy in a sweet spot — full enough to keep links busy, empty enough to never trigger PFC — and PFC becomes the backstop it was meant to be rather than the daily driver.

Done wrong, it is a disaster, and "wrong" is the default. DCQCN exposes a large, coupled parameter space — the ECN marking thresholds (Kmin and Kmax: when to start marking, when to mark every packet), the marking probability slope, the CNP generation cadence, the rate-increase and rate-decrease step sizes and timers, all interacting with the per-queue PFC thresholds and headroom. Set the ECN threshold too high and you mark too late, the buffer overruns, PFC fires, and you are back in pause-storm territory. Set it too low and you throttle senders that were not actually congested, leaving bandwidth on the table and inflating job completion time. The parameters are also topology-, speed-, and workload-dependent: a tuning that is perfect at 400G on a two-tier Clos is wrong at 800G on a three-tier one. This is the single largest reason an untuned RoCE fabric runs at a fraction of its rated throughput, while a tuned one closes 80–90% of the gap to InfiniBand.

A fat-tree gives you many equal-cost paths between any two endpoints. The question is how you spread traffic across them, and AI traffic makes the naive answer fail badly. The default is ECMP — Equal-Cost Multi-Path — which hashes each flow's 5-tuple to pick one of the available uplinks. For web traffic with thousands of small flows, the hash spreads load beautifully by the law of large numbers. For AI traffic with a handful of elephant flows per host, the law of large numbers does not apply: two fat flows can hash to the same physical link and collide, saturating it while a parallel link sits half-empty. The collision is not transient — these flows live for the whole training run — so the straggler it creates is permanent until something reshuffles the hash. On a synchronous collective, that one congested link taxes every step. This is the structural reason flow-level ECMP under-delivers on AI fabrics: there are too few flows to hash well, and they last too long to absorb a bad hash.

Two answers exist, and they trade simplicity against the cost of fixing in-order delivery. Adaptive routing keeps the flow concept but lets the switch re-pick the egress port dynamically based on real-time queue depth, steering a flow away from a congested link onto a quieter one — InfiniBand has done this for years, and Ethernet silicon increasingly supports it. Packet spraying goes further: it abandons per-flow pinning entirely and sprays the packets of a single flow across every available path, achieving near-perfect link utilization. The cost is that packets now arrive out of order, and classic RoCE's go-back-N retransmission treats out-of-order as loss — the in-order penalty that has historically made spray a non-starter on RoCE. The whole point of Ultra Ethernet's transport is to break that penalty: UET sprays at the switch and reorders at the NIC, making out-of-order delivery a feature rather than a fault, and exposing per-packet multipathing that was previously locked inside proprietary fabrics. → the in-order penalty and how each transport handles it in Chapter 8.4; the topology that defines how many paths exist to spread across in Chapter 8.5.

The previous two sections fight congestion by managing the traffic. In-network compute attacks the problem from the other side: send less traffic in the first place by doing the collective's arithmetic inside the switch. In a conventional all-reduce, every GPU's gradient buffer is shuffled across the network in a ring or tree, summed at each hop on the endpoints, and the result scattered back — the data crosses the fabric multiple times, and every reduction step burns GPU streaming-multiprocessor cycles and PCIe/NIC bandwidth that could have been training. SHARP — NVIDIA's Scalable Hierarchical Aggregation and Reduction Protocol — moves the reduction into the switch ASIC: GPUs send their data up an aggregation tree, the switches sum it in-network as it passes, and only the single reduced result comes back down. The data crosses the fabric roughly once instead of many times.

The consequences are unusually clean for an infrastructure decision because they cut in the right direction on multiple axes at once. SHARP roughly doubles effective all-reduce bandwidth versus a non-SHARP configuration, because the aggregation happens in transit rather than via repeated endpoint exchanges. It frees GPU SMs that were busy doing reduction arithmetic, handing them back to compute. And it cuts the bytes on the wire, which means less congestion to control and less power burned moving data — a rare lever that improves goodput and reduces energy per step simultaneously. SHARPv4 on the Quantum-X800 generation pushes 14.4 TFLOPS of in-network compute (a 9x jump over the prior generation) and supports FP8, All-Reduce, and MPI_Alltoall offload, integrated into NCCL 2.27 so frameworks get it transparently. The same idea lives inside the scale-up domain as NVLink-SHARP, doing in-switch reduction across the NVL72/NVL576 NVLink fabric → canonical scale-up treatment in Chapter 8.2.

The catch is the catch you expect: in-network compute is, today, deeply vendor-coupled. SHARP is an InfiniBand/Quantum (and NVLink) feature; using it commits you to that fabric and its switch radix, topology, and aggregation-tree constraints. It also imposes structure — the reduction tree must map onto the physical topology, which interacts with how you laid out scalable units in Chapter 8.5. The open-standards answer is younger: collective offload is on the roadmap for Ultra Ethernet and the broader merchant ecosystem, but as of 2026 the production, double-the-bandwidth, frees-the-SMs version of in-network reduction is a single-vendor capability. The decision is therefore a goodput-vs-lock-in trade made at fabric-selection time, not later — you cannot bolt SHARP onto a fabric that was not built for it.

Every mechanism above produces a counter, and the difference between a fabric that runs at 95% goodput and one that mysteriously runs at 70% is almost always whether the operations team is watching the right ones. The congestion-control loop emits ECN-marked packet counts and CNP rates — a rising CNP rate on a link is the earliest signal that DCQCN is being asked to throttle, and a leading indicator of a hotspot before it becomes a PFC pause. PFC itself emits pause-frame counts and cumulative pause duration per priority per port — any sustained pause time is a red flag, and pause storms are visible here long before they deadlock. The transport emits retransmit, RNR-NAK, and out-of-order counters that distinguish a real loss problem from a load-balancing reordering problem. And the switches expose per-queue buffer occupancy and microburst histograms that show whether you are riding the DCQCN sweet spot or skating the edge of overflow.

The hard part is correlation, not collection. A straggler flow shows up as elevated JCT at the framework, a congested queue at one switch, a CNP spike at one NIC, and a pause counter three hops upstream — and tying those together to the one link that needs attention requires time-aligned telemetry across the whole fabric, which is exactly why precise time synchronization (PTP/IEEE-1588, → Chapter 8.7) is a prerequisite for congestion diagnosis, not a separate concern. The operational practice of streaming these counters, baselining them, and alerting on deviation belongs to the observability stack: fleet observability and GPU/network health in Chapter 10.6, and DCIM-grade facility-plus-fabric telemetry correlation in Chapter 14.2. The design rule for this chapter: instrument the counters before first light, because the first large training run is the worst time to discover you have no visibility into why a step is slow.