Congestion Control, Load Balancing & In-Network Compute

Chapter 8.5 sized a topology that is non-blocking on paper: enough bisection bandwidth for every GPU to talk to every other GPU at line rate. That paper number is a lie until you solve three problems the topology alone does not. AI traffic is the pathological worst case for a packet network: a handful of enormous, long-lived elephant flows per host, synchronized to the same collective, hitting the wire in the same microsecond and converging on the same destinations during an all-reduce. A synchronized collective can hit the fabric as a coordinated stampede followed by an idle gap, so do not count on background mice to smooth the burst. Mixed inference and storage traffic add their own demand to the same queue and admission budget. At that point congestion control, load balancing and in-network compute stop being tuning knobs. They decide the difference between the bandwidth you paid for and the bandwidth you get.

The chapter covers the lossless-versus-lossy fork and the PFC pathologies that haunt the lossless path; the DCQCN parameter space you must tune or buy pre-tuned; and the load-balancing fork between flow-hashed ECMP and adaptive or per-packet spray, which sends a reordering bill to the NIC. It then turns to in-network compute (SHARP and its relatives), a lever that can improve goodput and energy per step when an eligible reduction removes critical endpoint work and traffic at lower whole-system energy cost. It ends on the telemetry that makes all of this observable, because a congestion problem you cannot see is a goodput problem you cannot fix. → fundamentals and the goodput framing in Chapter 8.1; the protocol and transport choices these mechanisms ride on in Chapter 8.4.

Data-center congestion control was designed for the web-scale workload: millions of short flows, bursty but uncorrelated, where TCP's loss-and-backoff and a little ECN keep buffers shallow and tails short. AI training inverts every one of those assumptions. A single all-reduce on a 100k-GPU cluster is one logical operation split into thousands of simultaneous, multi-gigabyte point-to-point transfers. They all start on the same barrier, and all must finish before the next step can begin. The collective runs at the speed of its slowest flow, the tail-latency tyranny of Chapter 8.1 made concrete at the packet layer. One congested link, one hash collision or one PFC pause that ripples the wrong way creates a straggler, and the straggler stalls every GPU in the job at the next bulk-synchronous barrier.

Three failure modes follow, and the rest of the chapter defends against each in turn. Buffer pressure and loss come from the traffic pattern of the algorithm: many-to-one incast in parameter-server and tree fan-ins, or synchronized neighbor flows in ring all-reduce. The congestion-control loop has to absorb them. Path imbalance means a few elephant flows collide on one physical link while parallel links sit idle, and load balancing must spread them. Endpoint overhead is reduction arithmetic on the GPU plus bytes moving through memory, PCIe and the NIC. Eligible in-network operations can reduce it, although the host attachment and library path determine which transfers and control work remain. Each failure mode carries its own downstream cost in goodput, power or lock-in.

RDMA, the zero-copy, kernel-bypass transport that makes GPU-to-GPU networking fast, was born assuming a lossless fabric. InfiniBand delivers losslessness natively with credit-based flow control: a sender never transmits unless the receiver has advertised buffer credits, so packets are never dropped for lack of room. A lossless RoCEv2 deployment adds a separate Ethernet mechanism, Priority Flow Control (PFC): a per-priority PAUSE frame that tells the upstream link to stop sending before a buffer overflows. PFC works, and it is the source of the nastiest pathologies in AI networking.

The first is head-of-line blocking. PFC pauses an entire traffic class, not a flow. When a downstream buffer fills, the PAUSE stops every flow in that priority on that link, including flows whose destinations are perfectly idle. These innocent bystanders are the "victim flows." The second is the PFC pause storm and, in the worst case, deadlock. PAUSE is hop-by-hop back-pressure, so a congestion hotspot propagates upstream toward the sources. In a topology with a cyclic buffer dependency (which fat-trees can form under certain failure or routing conditions) the pauses can lock into a permanent standstill, with every switch waiting on another and none able to drain. A deadlocked fabric does not degrade; it stops. The defense is a PFC watchdog: a timer on every queue that, if a port stays paused beyond a threshold, drops the stuck traffic and logs it rather than letting the deadlock persist. That trades a localized packet loss for fabric-wide survival.

So the loss model is a design decision, made alongside loss recovery and receiver resources. Meta’s 2024 production account describes its PFC-based 400G back-end, which ran with end-to-end congestion control disabled in that reported deployment; its controlled topology and traffic are part of that result. Ultra Ethernet specifies congestion signaling and loss recovery, with trimming where your switch and transport configuration support it. The switch discards the payload but forwards a shortened packet that identifies what was lost, so supported selective recovery can retry the missing data rather than everything after it in the window. Feedback still travels a real route and can itself be delayed. Choose the transport’s supported combination, then test the fault response instead of inferring it from the words lossless or lossy. → operation and completion semantics in Chapter 8.4; switch buffer resources in Chapter 8.3.

DCQCN (Data Center Quantized Congestion Notification) is the end-to-end loop that keeps a RoCE fabric out of PFC pause and away from loss. The mechanism is elegant. A switch experiencing early congestion marks packets with ECN (it sets the Congestion Experienced bit) before its buffer is full. The receiving NIC notices the mark and fires a Congestion Notification Packet (CNP) back to the sender. The sending NIC quantizes its injection rate down in response, then probes back up as the congestion clears. Done right, DCQCN holds buffer occupancy in a sweet spot, full enough to keep links busy and empty enough never to trigger PFC. PFC then becomes the backstop it was meant to be rather than the daily driver.

Done wrong, the same feedback can oscillate between excess queueing and underused links. DCQCN exposes a large, coupled parameter space: the ECN marking thresholds (Kmin and Kmax, when to start marking and when to mark every packet), the marking probability slope, the CNP generation cadence, and the rate-increase and rate-decrease step sizes and timers. All of them interact with the per-queue PFC thresholds and headroom. Set the ECN threshold too high and you mark too late, the buffer overruns, PFC fires, and you are back in pause-storm territory. Set it too low and you throttle senders that were not actually congested, leaving bandwidth on the table and inflating job completion time. The parameters also depend on topology, speed and workload. A move from 400G on a two-tier Clos to 800G on a three-tier one changes in-flight bytes and feedback delay, so the old thresholds need retesting. An untuned RoCE fabric can leave rated capacity unused, but a quieter queue obtained by over-throttling senders is not recovered throughput. Measure the job’s step tail, receiver drain and queue trajectory together.

First locate the service constraint. An interior egress queue rising while alternative paths sit idle calls for a path-distribution remedy. Every path converging on a full destination link calls for admission control or more destination capacity. A quiet wire with depleted posted receives calls for receive-resource repair, and adequate receives with stalled memory consumption point to the consumer. Meta’s receiver-driven admission account motivates this separation. In UET, NSCC uses network congestion signals to regulate injection, while RCCC uses receiver-granted credits to control arrivals. They address different resource boundaries, so protect the returning control traffic and test lost or stale feedback. Direct switch-to-source signaling needs a supported source response and a failure policy.

IEEE’s switch-to-source signaling work has no published standard. P802.1Qdw (source flow control) never produced a draft, and its withdrawal was requested in March 2026. P802.1EJ, Backward Notification, replaces it as a stand-alone project, PAR-approved in September 2026, with a broader scope: congestion management, path rebalancing and failure recovery. The PAR schedules a ballot draft for 2028 and completion for 2029.

The in-band path moved faster. The CSIG individual Internet-Draft (draft-ravi-ippm-csig-01, 2024-02-02) expired, but Ultra Ethernet 1.1 (12 October 2026) adopts the mechanism as Congestion Signaling (CSIG). Each switch updates a 4- or 8-byte tag with the path’s extreme value for one signal, such as the least available bandwidth or the largest queueing delay, and the receiver reflects it to the sender in an extended acknowledgement. The sender learns the state of its bottleneck hop instead of ECN’s single bit for the whole path, so it can climb back into freed capacity faster. Version 1.1 also adds programmable congestion management, which lets operators load alternative congestion-control and load-balancing algorithms into the endpoint NIC beside the NSCC and RCCC baselines. CSIG is optional and works end to end only when every switch on the path supports it and the fabric enables it, so ask which switch ASICs, NOS releases and NIC firmware implement the 1.1 tag before you design around it.

Worked decision A: incast backlog and pause headroom

Four inputs offer 4 × 400 = 1,600 Gb/s. With the output draining, excess arrival is (1,600 − 400) × 10⁹ / 8 = 150 × 10⁹ B/s. Backlog after the assumed delay is 150 × 10⁹ B/s × 2 × 10⁻⁶ s = about 0.3 MB, below the 0.40 MB usable allowance. The crossover is 0.40 × 10⁶ B / (150 × 10⁹ B/s), about 2.7 µs. The unrounded threshold is 2⅔ µs; 3 µs produces 0.45 MB and fails. Sustained incast still needs admission.

Ingress PFC headroom is a separate budget. Each paused input needs 400 × 10⁹ bit/s × 2 × 10⁻⁶ s / 8 + 2 × 9,216 B + 32,000 B = about 0.15 MB. The unrounded allocation check is 150,432 B ≤ 160,000 B, and four simultaneous pauses require 601,728 B ≤ 640,000 B. At 3 µs, one input needs 200,432 B and fails. Raise each port to 800 Gb/s, as a single-port 800G NIC would, and keep the 2 µs: both budgets fail, with backlog at about 0.6 MB and 250,432 B of headroom needed per ingress. A GB300’s two 400G planes are each a 400G port, so they stay in the case above. Do not deduct egress drain from ingress headroom, and do not double-count shared storage.

Accept buffering only when allocation and measured delay pass both budgets; otherwise reduce admission or buy more usable buffer. Verify cell rounding, pool competition, pause/resume and watchdog recovery. The Cumulus buffer and RoCE documentation supplies the configuration method. Chapter 8.5 owns the surviving-fabric tally and Chapter 13.7 runs the acceptance test.

If you build a RoCE fabric from merchant switch silicon and standard NICs, you own the DCQCN parameter space. That means a standing investment in network engineers who can read ECN/CNP counters, run incast tests and re-tune every time you change speed, scale or topology. The alternative is to buy a fabric where the vendor has done that work and validated it. NVIDIA Spectrum-X bundles switches and SuperNICs with a supported congestion and adaptive-routing profile, and NVIDIA’s October 2024 Colossus report describes roughly 95% data throughput for that deployment. A single escalation path can save integration work, but it does not replace workload acceptance. InfiniBand uses credit flow rather than DCQCN. The decision is a classic build-versus-buy on the network. Self-tuned merchant RoCE can trade a lower switch and NIC bid for a recurring tuning liability carried by your engineering team. An integrated fabric turns only its contracted support duties into a line item. Compare acquisition, NIC, switch, optics and firmware support, and failure-recovery cost at equal service. Pick based on whether you have the network-engineering bench to keep a merchant RoCE fabric tuned through every speed, scale and topology change.

Worked decision B: tune, remove a spine, then recover

Export the effective NIC and switch settings and the bidirectional counters. The versioned Cumulus profile is the class-map reference, not authority for these proposed thresholds. Verify that background traffic cannot occupy the headroom already reserved. With the assumed first-event alpha, the sender rate becomes 400 Gb/s × (1 − 1/16) = 375 Gb/s. Four such senders still offer 1,500 Gb/s to the destination’s 400 Gb/s service, and one decrease cannot fix a sustained incast. Continue feedback and admission until the aggregate arrival rate fits service, or the queue keeps growing.

Replay the same traffic seed and rank placement. Collect healthy tails, queue high-watermarks, ECN/CNP counts and pause duration. Remove one spine, timestamp the withdrawal, verify Chapter 8.5’s surviving ports, and measure both the transition and the converged behavior. Pass requires recovery within 1 s, converged p99 within 1,100 ms, healthy p99 within 900 ms, no corruption and no persistent watchdog-driven pause cycle. Report transition interruption separately. Inject receiver starvation and delayed CNP separately.

Hold deployment until the hardware trace is in. Tune one control family at a time with a rollback snapshot; if marking comes too late, lower the egress thresholds before consuming ingress headroom. If tails rise while queues stay empty, restore the prior rate-recovery profile. The decision flips to accept only when the specified profile passes every bound, and a converged p99 above 1,100 ms reverses it even if port utilization looks balanced. Chapter 13.7 records installed acceptance.

A fat-tree gives you many equal-cost paths between any two endpoints. The question is how to spread traffic across them, and AI traffic makes the naive answer fail badly. The default is ECMP (Equal-Cost Multi-Path), which hashes each flow's 5-tuple to pick one of the available uplinks. With thousands of small web flows, the hash spreads load beautifully by the law of large numbers. With a handful of elephant flows per host, that law does not apply: two fat flows can hash to the same physical link and collide, saturating it while a parallel link sits half-empty. A collision can persist across the critical collective while those flows keep their hashes, so the straggler remains until a flow ends or the path mapping changes. On a synchronous collective, that one congested link taxes every step. Flow-level ECMP under-delivers on AI fabrics for a structural reason: there are too few flows to hash well, and they last too long to absorb a bad hash.

Two answers exist, and they trade simplicity against the cost of fixing in-order delivery. Adaptive routing lets supported InfiniBand and Ethernet switches steer traffic away from a congested egress toward a quieter path, using the congestion information they actually see. The movement unit can be a flow, flowlet or packet. Local egress occupancy cannot reveal every downstream bottleneck, and permitted movement must match the receiver’s ordering requirements. On merchant Ethernet silicon this has become utilization-aware dynamic load balancing: the switch assigns each new flow, or flowlet, to the least-loaded member link instead of a blind hash. A flowlet is a burst separated by an idle gap long enough to re-path without reordering. RDMA-aware variants recognize individual queue pairs and re-place them when congestion appears.

These switch-side mechanisms can recover part of ECMP’s collision tax without replacing the transport, which is attractive in a multi-vendor fabric, provided their packet movement stays inside the receiver’s ordering requirements. What they cannot do is let one elephant use several paths at once. That still takes per-packet spray and endpoint reordering. Packet spraying goes further: it abandons per-flow pinning entirely and sprays the packets of a single flow across every available path, so one flow can exploit spare capacity on several paths. The cost is that packets now arrive out of order, and classic RoCE's go-back-N retransmission treats out-of-order as loss. This in-order penalty has historically made spray a non-starter on RoCE. Breaking it is the point of Ultra Ethernet's transport. A UET implementation combines packet-level multipath selection with NIC-side placement, reordering and completion behavior supported by the delivery mode you choose. Out-of-order delivery becomes a feature rather than a fault, and per-packet multipathing, once locked inside proprietary fabrics, becomes available. → the in-order penalty and how each transport handles it in Chapter 8.4; the topology that defines how many paths exist to spread across in Chapter 8.5.

The previous two sections fight congestion by managing the traffic. In-network compute attacks the problem from the other side: send less traffic in the first place by doing the collective's arithmetic inside the switch. In a conventional all-reduce, every GPU's gradient buffer is shuffled across the network in a ring or tree, summed at each hop on the endpoints, and scattered back. The data crosses the fabric multiple times, and every reduction step burns GPU streaming-multiprocessor cycles and PCIe/NIC bandwidth that could have been training. SHARP, NVIDIA's Scalable Hierarchical Aggregation and Reduction Protocol, moves the reduction into the switch ASIC. GPUs send their data up an aggregation tree, the switches sum it in-network as it passes, and only the single reduced result comes back down. Switch reduction can replace repeated endpoint exchanges with aggregation along a tree; count endpoint-facing bytes separately from traffic summed over every fabric hop.

SHARPv4 on the Quantum-X800 generation provides 14.4 TFLOPS of in-network compute, nine times the prior generation, with FP8 support (NVIDIA, March 2024). SHARP and NVLink-SHARP move eligible reduction arithmetic into the switch and free GPU SMs for model work, so compare them with the best eligible endpoint algorithm. For n ranks each reducing S bytes, a bandwidth-oriented ring sends 2(n−1)S/n bytes per rank and receives the same amount. An ideal switch reduction receives S from each rank and returns S to it, so its endpoint-facing ratio to the ring is n/[2(n−1)]. That ratio describes bytes. Reducing bytes can save congestion and energy, but execution time and whole-system power still need measurement. Headers, tree depth, reduction resources, library launch and overlap all still cost time. Freeing GPU arithmetic helps only if that work sits on the critical path, so a job dominated by another collective or by local compute will not inherit a byte-count speedup. NCCL-tests normalization supplies the measurement convention; keep algorithm bandwidth, bus bandwidth and the actual job deadline separate.

Switch reduction is not NVIDIA-only. Broadcom announced in July 2025 that it was shipping Tomahawk Ultra, a scale-up Ethernet switch chip that runs AllReduce, Broadcast and AllGather inside the switch and that Broadcom calls endpoint-agnostic. That is a vendor claim, and no independent collective benchmark has been published. UALink Common 2.0, announced in April 2026, specifies in-network compute, but UALink switch silicon only starts reaching labs late in 2026.

Ultra Ethernet 1.1 (October 2026) adds optional in-network collectives (INC) to scale-out Ethernet: the switch performs barrier, broadcast, allreduce and reduce. An INC manager sets up each communicator, and the switch combines every endpoint’s contribution before returning the result. UEC says INC can reduce the network bandwidth a large reduction needs by as much as a factor of two, the value the ratio n/[2(n−1)] above approaches as n grows. The 1.1 scope is tight. Every endpoint attaches directly to one logical switch (a vendor may build it from several), and INC traffic must run on a lossless class, PFC or credit-based flow control, over UET’s reliable ordered delivery. Choosing INC therefore reopens the lossless-class headroom and deadlock tests from earlier in this chapter.

SHARP needs supported NVIDIA switch, endpoint and library resources. You cannot bolt that reduction onto a fabric that lacks the engines or aggregation topology. Whatever the vendor, the acceptance record names the operation, dtype and accumulation precision, message-size range, group count and concurrency, switch aggregation resources, NIC and library releases, and proof that the library selected offload. Exhaust a reduction resource and remove an aggregation path: record whether the call retries, falls back to the endpoint algorithm or fails the communicator, and whether the fallback still meets its deadline. NVIDIA’s SHARP account supports the mechanism. Chapter 8.2 covers platform eligibility and the UALink family record that separates specification scope from purchasable support; the physical cut is in Chapter 8.5. Choose offload when the measured critical phase and fallback justify the integration and vendor coupling, and reverse the choice when resource exhaustion exposes a deadline the endpoint implementation cannot meet.

Every mechanism above produces signals, and a fabric can leave expensive links idle while the operations team watches the wrong counter. Use queue, retry, pause and receiver data at a boundary and time resolution that distinguish competing causes. The congestion-control loop emits ECN-marked packet counts and CNP rates. A rising CNP rate on a link is the earliest signal that DCQCN is being asked to throttle, and a leading indicator of a hotspot before it becomes a PFC pause. PFC itself emits pause-frame counts and cumulative pause duration per priority per port. Any sustained pause time is a red flag, and pause storms show up here long before they deadlock. The transport emits retransmit, RNR-NAK, and out-of-order counters that distinguish a real loss problem from a load-balancing reordering problem. The switches expose per-queue buffer occupancy and microburst histograms that show whether you are riding the DCQCN sweet spot or skating the edge of overflow.

The hard part is correlation, not collection. A straggler flow shows up as elevated JCT at the framework, a congested queue at one switch, a CNP spike at one NIC and a pause counter three hops upstream. On the GPU side, NVIDIA’s NCCL Inspector plugin logs bus bandwidth and duration for every collective, so the same straggler also appears as one slow collective on every rank that waits for it. Tying those to the one link that needs attention takes time-aligned telemetry across the whole fabric. Precise time synchronization (PTP/IEEE-1588, → Chapter 8.7) sets the uncertainty of cross-device event ordering, though it cannot reconstruct a burst that was never captured. Streaming these counters, baselining them and alerting on deviation belong to the observability stack: fleet observability and GPU/network health in Chapter 10.6, and DCIM-grade facility-plus-fabric telemetry correlation in Chapter 14.2. Instrument the counters before first light. The first large training run is the worst time to discover you have no visibility into why a step is slow.

Join job and rank → NIC/QP → priority and queue → physical port/lane → route member. Check whether a congested egress comes first, followed by ECN marks, receiver CNP and a sender-rate decline. Concurrent growth on every route into the same destination points to receiver admission instead. Re-run with the consumer deliberately slowed: rising receive-resource stalls while the wire is quiet falsify the path-congestion diagnosis. Record the clock origin and uncertainty, sample period, histogram window, resets, wraparound and missing or coalesced samples. Polling can reveal trends, but bursts shorter than the polling interval need queue high-watermarks and triggered hardware captures. Streaming does not create finer source timestamps. gNMI transports updates, IOAM defines packet-carried fields, and implementation-specific INT/IFA modes need their own schemas. A UEC 1.1 CSIG tag carries one aggregated value per signal, the worst hop on the path, not a full per-hop trace. Budget added packet bytes, sampling rate and collector traffic, and repeat the load test with capture disabled to expose observer effects. Hand the paired traces and acceptance bounds to Chapter 13.7.

Ultra Ethernet’s June 2025 public specification gave the merchant Ethernet ecosystem a common way to describe packet-level multipath, endpoint delivery and reordering, congestion signaling and supported loss recovery. That is a concrete alternative to buying only an integrated Spectrum-X or InfiniBand stack. The current release (below) keeps network signals separate from receiver credits and adds optional CSIG and in-network collectives, both described above. Choose a version-matched endpoint, switch profile and delivery mode, then test the same operations, traffic and faults. The multi-vendor bet pays when a second implementation delivers all of it; publishing a mechanism does not measure job goodput.

MRC adds a distinct open transport path. Its May 2026 production report describes an OCP RoCEv2 extension with packet spray, out-of-order placement and selective retransmission, used by OpenAI and Microsoft. That is a named deployment record, not a substitute for UET’s broader scope, and MRC 1.0’s Write-oriented operation boundary is detailed in Chapter 8.4.

Meta’s MetaRoCE (announced August 2026) is a third open path. It drops PFC, sprays each connection over multiple paths, and combines ECN-based sender control with rate hints from the receiver. Meta’s own controlled test on a 64-node AMD GPU cluster running RCCL collectives showed about 86% throughput at 1% packet loss. At the October 2026 OCP Summit Meta announced open-source implementations; the first hardware implementation runs on AMD Pensando NICs.

Spray, trimming and offload trade endpoint state, switch support, integration effort and recovery behavior. Keep the standards trigger in Chapter 16.2 and retest when the version or supported operation changes.

Choose the control that bounds the measured bottleneck and survives the lost path: movement for spare interior capacity, admission for a destination constraint, receiver repair for exhausted resources, and offload for eligible critical work. Choosing a mechanism from a port-utilization graph alone can move the queue while leaving the same GPUs waiting.