Scale-Out Fabric: Protocols, Standards & Transport

A scale-out fabric has one job: carry the collective traffic of a distributed training or inference job (all-reduce, all-gather, reduce-scatter, all-to-all) across the back-end network that links scale-up domains, without becoming the bottleneck that idles the most expensive silicon in the building. The accelerators are bought and the power is contracted. What remains variable is how much of every step goes to computing and how much to waiting on the network. That share depends on the transport implementation, the paths it can use, and whether the endpoints can complete the operations the job requires.

The decision presents as five options: InfiniBand, RoCEv2 in whichever lossless or lossy mode you select, NVIDIA Spectrum-X, Ultra Ethernet (UEC) and, since May 2026, the OpenAI-led MRC. It is really one question asked twice. First: do you run a purpose-built lossless transport (InfiniBand) or Ethernet? If Ethernet: does a RoCEv2 NIC's documented ordering, retry and PFC behavior meet the job's deadlines, or does a proven multipath implementation (Spectrum-X, UET or MRC, each at its current published specification) justify its integration cost by spreading traffic across paths while preserving every required operation and completion rule? Each answer cascades into a switch-vendor list, an operational skill base and a congestion-control parameter space. It also brings the part nobody prices at scoping time: a multi-year lock-in to whoever owns the firmware your goodput silently depends on.

For most of a decade the scale-out question had two default answers: InfiniBand for frontier training and RoCEv2 on Ethernet for everyone cost-sensitive. Two forces broke that binary. Hyperscalers refused to single-source their largest capex line on one vendor's proprietary fabric, and AI-tuned Ethernet arrived claiming InfiniBand-class effective throughput on a merchant-silicon supply chain. The result is a genuine five-way fork. The right answer depends on cluster scale, tenancy model, in-house operational depth, and how much vendor lock-in you will tolerate against how much goodput you are willing to leave on the table.

InfiniBand (NVIDIA Quantum) is the incumbent for tightly coupled training. It is lossless by construction: credit-based flow control means a sender never transmits a packet the receiver has no buffer for, so the fabric does not drop frames under congestion the way Ethernet does. It carries native RDMA, adaptive routing, and in-network reduction through SHARP (collective offload). The cost is a single-vendor supply chain (NVIDIA end to end: switch, NIC, cable, subnet manager), an operational discipline most Ethernet teams do not have, and a price premium. → In-network compute and SHARP are engineered in Chapter 8.6; the switch and NIC silicon in Chapter 8.3.

RoCEv2 (RDMA over Converged Ethernet v2) puts RDMA semantics on a routed UDP/IP Ethernet underlay. It promises the commodity economics and multi-vendor supply of Ethernet with RDMA's CPU-bypass data path. The catch is that classic RoCE inherited the go-back-N assumption of an ordered, lossless link: a gap can trigger retransmission of everything from the missing packet onward. PFC-based losslessness is one supported mitigation; a documented lossy RoCE mode with appropriate endpoint recovery is another. That engineering is the hard part, and its failure modes (head-of-line blocking, PFC deadlock, victim flows) recur throughout this chapter. Meta runs production RoCE at 24k-to-100k-GPU scale, which proves it works, but Meta's published account shows how much fabric engineering that took.

NVIDIA Spectrum-X is Ethernet that behaves like a purpose-built AI fabric: a Spectrum switch plus a BlueField or ConnectX SuperNIC, version-matched, doing adaptive routing, per-packet spraying, NIC-side reordering and programmatic congestion control. Together they use multiple paths without the classic in-order penalty, subject to the endpoint configurations NVIDIA supports. Reported deployments to date use Spectrum-4 switches such as the SN5600. The newer Spectrum-6 switches and ConnectX-9 SuperNICs belong to the Vera Rubin generation. NVIDIA announced them in production on May 31, 2026, but no at-scale deployment has been published. That is separate from the Vera Rubin NVL72 racks themselves, which have shipped since June 2026 (Chapter 8.3). The best-known Spectrum-4 deployment is xAI's Colossus, where NVIDIA reports about 95% data throughput against the ~60% it says standard Ethernet delivers under flow collisions. The cost is that this is Ethernet on the wire but still effectively an NVIDIA end-to-end story. The switch and the SuperNIC come from one vendor, so the lock-in is real even though the protocol is nominally open.

Ultra Ethernet (UEC) is the industry's open answer: a full transport stack (UET) specified by the Ultra Ethernet Consortium, whose initial public 1.0 specification was released on 2025-06-11. It is designed from scratch for AI and HPC: packet spray across multipath, out-of-order delivery decoupled from message ordering with reassembly in the NIC, two congestion-control algorithms (sender-side NSCC and receiver-credit RCCC), packet trimming for fast loss signaling, and native RDMA. The promise is a common transport for a multi-vendor supply chain, with AMD, Broadcom, Cisco, Arista, HPE, Intel, Meta and Microsoft among the UEC participants. The procurement cost is testing the current version across your actual endpoints and switches: a compliance declaration does not prove that two suppliers complete the same operations or recover from the same failure together. → The standards trajectory is consolidated in Chapter 16.2.

MRC (Multipath Reliable Connection) is the newest branch and the pragmatist's open path: an OCP specification (Release 1.0, dated March 2026) publicly released in May 2026 by OpenAI with AMD, Broadcom, Intel, Microsoft and NVIDIA. Unusually for a day-one standard, it was already production-proven by the consortium paper's account (2026): it ran on OpenAI's and Microsoft's largest training clusters before publication, including a 50k-GPU production pretraining job at OpenAI and a documented 75k-GPU pretraining job. Where UET replaces the RoCE stack outright, MRC extends RoCEv2 with the minimum machinery AI training needs. The data plane narrows to RDMA Write and Write-Immediate. Every packet carries an entropy value, so a single connection sprays across many paths, or whole planes. Reliability moves to selective acknowledgment, with packet trimming as an optional fast loss signal where switches support it, and the fabric runs lossy with PFC off. Optionally, SRv6 source routing replaces dynamic path selection outright: hosts stamp the path onto the packet and the switches forward from static tables. The bet is UET's core performance ideas at a fraction of its scope: simpler to implement on existing RoCE-class NICs and merchant switch silicon, but narrower, a training-first transport rather than a general HPC and storage stack. Vendor support landed fast on both ends of the wire (ConnectX-8, AMD Pollara/Vulcano and Broadcom Thor Ultra among NICs; Tomahawk-class, Spectrum-X and P4-programmable silicon among switches). Its multiplanar consequence, eight 100G planes from one 800G NIC reaching a 131,072-GPU two-tier fabric on 51.2 Tb/s switches (radix 512 at 100G), is engineered in Chapter 8.5.

The fork is still widening. Meta announced MetaRoCE on 2026-08-24 and open-source implementations at the OCP Global Summit on 2026-10-12. It is a clean-sheet RDMA transport rather than a RoCEv2 extension: no PFC, multipath packet spraying, the existing verbs API kept, and a first implementation on AMD Pensando NICs. Meta calls it production-proven, without deployment figures, and reports about 86% throughput at 1% packet loss on a 64-node AMD GPU cluster running RCCL collectives. Google's Falcon is older: OCP has hosted it openly since 2023, and it first shipped on Intel IPUs. NVIDIA says future ConnectX-10 SuperNICs will support it. Chapter 8.6 compares how their congestion and reliability mechanisms differ. Put three things in the purchase contract: the transport, the version of its specification and the NIC firmware release. Treat every pairing of RoCE, UET, MRC, MetaRoCE and Falcon as wire-incompatible until you have tested it.

Two independent axes determine behavior under the five product names: does the fabric drop packets under congestion (lossy) or refuse to (lossless)? and must packets arrive in order, or can the endpoint reassemble an out-of-order stream? Every transport is a point in that 2x2. The modern AI-Ethernet entrants exist because the historically dominant corner, lossless and ordered, turned out to be a trap at scale.

Lossless means the fabric applies back-pressure to prevent buffer overflow rather than dropping frames. InfiniBand does this natively with credits. Ethernet retrofits it with PFC (IEEE 802.1Qbb): when an ingress buffer fills, the switch sends a PAUSE upstream for that traffic class, stopping the sender. PFC is a blunt, per-class, per-link hammer. It pauses all flows in the class on that link, not just the one causing congestion. That is head-of-line blocking: an innocent flow sharing the link is paused for a congested neighbor it has nothing to do with. Worse, PAUSE propagates hop by hop upstream, and in a multi-tier Clos with a cyclic buffer dependency it can deadlock the fabric entirely, a failure that needs watchdogs to detect and break. → The PFC/ECN/DCQCN parameter space and its pathologies are the subject of Chapter 8.6.

Ordered delivery is the second trap. Classic RoCE assumes a packet stream arrives in sequence and treats an out-of-order arrival as a loss. That triggers go-back-N retransmission: the receiver discards everything after the gap and the sender retransmits from the lost packet forward. This is the RoCE in-order penalty. It is why classic RoCE cannot freely spray a flow across multiple equal-cost paths: if two paths have different latency, the resulting reordering looks like loss and triggers cascading retransmits. So vanilla RoCE pins each flow to a single path (ECMP hashing on the flow tuple). A single hot link can then bottleneck a flow while parallel links sit idle, which is exactly the all-to-all collision pattern that collapses untuned RoCE goodput.

Spectrum-X, UEC, and MRC all attack the same root cause: they decouple wire order from message order. The NIC sprays packets of a single flow across many paths, accepts them out of order, and places accepted packets and tracks message completion according to the operation’s ordering rules. A delay, a bit of congestion, or a single dropped packet on one path no longer poisons the whole flow. Selective retransmission replaces go-back-N and can reduce retransmission to the missing packet rather than repeating the later window, though feedback, retry and outstanding dependencies still determine the completion delay. UEC adds packet trimming: instead of dropping a congested packet outright, the switch truncates it to its header and forwards the stub, so the receiver learns of the loss immediately and signals a fast, surgical retransmit. This decoupling is why the protocol war is really a war over who owns the NIC that does the reordering.

First compare required operations with implemented operations. A and B are eligible for qualification because their assumed device declarations include Read, Write and Write-with-Immediate. C is excluded before any throughput comparison: the MRC 1.0 specification dated 2026-03-21, §5.1 and §6.2.2, excludes Reads, Sends and atomics, and a faster NIC cannot supply a missing verb. The flip is the workload’s Read requirement. If every remote fetch is safely converted to an owner-pushed Write with a tested notification and ownership protocol, C becomes eligible too. That conversion costs application work and receiver-resource management; it is not a transparent transport swap.

Then run the completion trace. In the successful transfer, register and authorize the destination, post the receiver resource, issue the Write-with-Immediate, accept all three packets, and release the consumer only after the required receiver completion and memory synchronization. A sender completion alone does not prove that an application on the remote accelerator has consumed the bytes. If the middle packet is lost, the selected reliable implementation retries under its documented rules, and incomplete data must not become a successful consumer event. A duplicated packet must not produce a second logical notification or repeat a non-idempotent application action. After retry exhaustion, require the documented completion error, stop dependent consumers, invalidate uncertain state and restart the application protocol from an agreed boundary.

The RDMA programming manual supplies the RC verbs context. Nothing is measured here, so no winner is named. Selection waits until each eligible candidate has passed through identical operation, visibility, loss, duplicate and failed-completion checks: A and B, plus C if its Reads were converted. Chapter 8.1’s workload comparison then ranks whichever pass. Pick the passing implementation whose support and substitution cost you can carry; Chapter 13.7 records installed acceptance.

The number that drives the AI-Ethernet movement is the gap between tuned and untuned RoCE effective throughput. ECMP flow-pinning can collide elephant flows on one path while others idle, and congestion, drops and a classic go-back-N implementation can then amplify completion time. NVIDIA's 2024-10-28 figures put Spectrum-X on xAI's Colossus at roughly 95% data throughput, against 60% for an unnamed “standard Ethernet” baseline. That is one vendor-reported deployment, not a test against a named competitor or a value for tuned RoCE, so quantify the penalty on your own endpoints, collective algorithm, topology and traffic matrix. The gap decides real money: at 100,000-GPU scale the network is the second-largest line after the accelerators themselves. What that difference costs a buyer on the same job is the accelerator time lost when exposed transfers miss the completion target.

The consequence for the decision: RoCE's cost advantage is real only if you have the team to capture it. The merchant-silicon switch is cheaper than InfiniBand, but the tuning labor, the validation rigs and the on-call fabric expertise are not. Spectrum-X sells that tuning as an integrated product, while UEC standardizes related mechanisms for multi-vendor implementations. Neither removes the need to validate delivered goodput on the target fabric. The operator can reuse a proven congestion design, but still owns endpoint settings, fault diagnosis and release qualification. You are, in effect, choosing whether to buy the goodput as a capability (Spectrum-X/UEC) or build it as a competency (hardened RoCE).

The strategic shape of Ethernet in 2026 is a choice between integrated platforms and open implementation paths. InfiniBand couples its transport with supported SHARP offload for tightly coupled runs, Spectrum-X gives NVIDIA an Ethernet platform, and UEC gives the merchant ecosystem a specification to implement. The initial UEC 1.0 release on 2025-06-11 is the historical inflection. It describes a coordinated stack across physical, link, transport, software, storage and management, giving NIC and switch suppliers common interfaces to build against. The sourcing bet pays off when a second implementation meets the same operation and failure requirements, and the buyer can then weigh its integration cost against the incumbent's price and support obligation.

Three developments make that choice concrete. First, the current specification is 1.1, released 2026-10-12, succeeding 1.0.3 of 2026-07-16. 1.1 is more than a point release. Version 1.0.x specified the libfabric API; 1.1 adds a verbs mapping, so UET devices appear through the RDMA verbs stack. They use a new reliable-unconnected QP type, not RC. 1.1 also adds In-Network Collectives (INC), where the switch performs barrier, broadcast, allreduce and reduce for endpoints attached directly to one logical switch on a lossless fabric (→ Chapter 8.6). Both additions are optional. So put two questions to every NIC and switch supplier: which API does its UET stack expose to your collective library, and has it declared and tested INC? Being in the specification does not make a feature a shipping product.

Second, implementations already differ. Broadcom's 800G Thor Ultra announcement of 2025-10-14 claims UEC feature compliance, while AMD's August 2025 Pollara 400 presentation describes a distinct 400G implementation, so neither NIC rates nor compliance labels are interchangeable. Third, MRC supplies another open path: its May 2026 production report describes a RoCEv2 extension with packet spray, out-of-order placement and selective retransmission in the reported system. The two open transports address different operation and integration boundaries. UET covers a broader operation set, while MRC 1.0 excludes Reads, Sends and atomics, so a workload that needs one of those pays for application changes before MRC becomes eligible. A buyer choosing UET pays for version-matched qualification, because the UEC compliance process uses self-attestation, not a guarantee that arbitrary devices interoperate.

For the decision-maker, this trajectory argues for explicit optionality management. If you must deploy at frontier scale on a deadline that favors an already supported integrated fabric, buy that combination after workload acceptance and capture the completion-time gain it actually demonstrates. But build the cluster’s physical layer (cabling, optics, structured plant) for the channel, lane, connector and module requirements of each credible Ethernet or InfiniBand replacement, so you can re-decide the transport at refresh after qualification. The physical investment is the irreversible part. The protocol running over it is comparatively reversible if you did not hard-wire your topology to one vendor's switch radix. → The full subsystem roadmap, including UEC milestones and the 800G→1.6T→3.2T optics ladder, is consolidated in Chapter 16.2; the physical-layer choices that preserve or destroy that optionality are in Chapter 8.9 and Chapter 8.10.

Freeze the UET specification version, software API and delivery mode together with the NIC, switch and firmware set. The 1.1 specification (2026-10-12) lists four transport profiles. Unified Local is new in 1.1 and aimed at scale-up; AI Base, AI Full and HPC carry over from 1.0.x. The same document's compliance chapter, like the older compliance readme, still uses the names AI Base, AI Extended and HPC. Compliance is still a vendor's self-attestation. §8.1.2 requires each declaration to give the spec version and section it meets, the profile and its optional features, and the checklist version used in testing. Line up artifacts by edition, profile and feature row instead of assuming profile names match. Mark Reads, Writes, Sends, atomics, matching, ordering, retry limits, security and receiver-resource limits as required, supported or absent for the actual endpoint. An optional mechanism on a data sheet is not enabled merely because both devices use Ethernet.

For coexistence with RoCE or ordinary traffic, document classification from endpoint marking through switch queue and pool, MTU/FEC, routing entropy, trimming behavior where enabled, congestion signals, and which traffic can be paused. Test load and failure with the coexisting traffic present, including receiver-resource exhaustion and a peer that lacks an optional feature. Separate reachability from interoperation: packets reaching an endpoint do not prove that its transport recognizes the operation or completes it correctly. Migration needs explicit owners for the old and new application paths, and you cannot assume wire compatibility between RoCE, UET and MRC. Chapter 8.6 owns congestion and reduction tuning, and Chapter 13.7 owns the installed acceptance record.

Everything above assumes a single-tenant supercomputer. Once the fabric serves multiple customers (any GPU neocloud, any cloud GPU service), the transport choice collides with an isolation requirement, and you cannot decide the two separately. A multi-tenant back-end fabric must stop one tenant from reaching or reading another tenant’s traffic and limit the congestion or attacks it can impose. That means enforcing reachability, memory authority, confidentiality and promised performance isolation as separate controls. It must do so on a high-bandwidth fabric whose completion and loss-recovery requirements constrain how buffering and policing can isolate tenants.

The mechanisms differ by transport. InfiniBand isolates with partition keys (PKeys) that the subnet manager defines and adapter and switch hardware enforce: a tenant's nodes share a PKey and cannot communicate across partition boundaries. Ethernet fabrics overlay tenancy with VXLAN/EVPN (network virtualization that gives each tenant an isolated L2/L3 segment over a shared underlay). One enforcement location outside a tenant-controlled host is the DPU: a BlueField/Pensando-class DPU at the host edge terminates the tenant's VPC, enforces microsegmentation and encryption, and rate-limits per tenant. Isolation is then enforced in hardware at the boundary of every server rather than trusted to the fabric core. DPU-enforced VPC isolation and VLAN/VXLAN segmentation solve different parts of tenancy, so the isolation test must prove which policy, DMA and memory controls survive tenant-root compromise. An overlay identifier or hardware device alone does not establish that boundary.

For protocol choice, multi-tenancy can favor Ethernet + DPU when the DPU enforces the required VPC, DMA and memory boundary outside tenant control. InfiniBand PKeys give partitioning, but the DPU-enforced VPC model (full network virtualization, per-tenant encryption, line-rate microsegmentation) maps more naturally onto an Ethernet fabric with programmable DPUs at the edge. A neocloud that must rent isolated slices to mutually distrusting tenants on shared GPUs is therefore pulled toward the Spectrum-X / UEC + DPU side of the fork, where the isolation primitives and the security tooling are richer, even before goodput enters the conversation. Here the fabric decision and the security decision merge: the same DPU that enforces the tenant boundary is the zero-trust enforcement point. → The tenant security boundary is engineered in Chapter 11.6, and zero-trust microsegmentation in Chapter 11.7; the DPU silicon itself in Chapter 8.3.

The choice resolves along four rules, in order.

(1) Scale and coupling. InfiniBand is still the most mature integrated fabric when the job is tightly coupled frontier training and your team already knows HPC fabrics, provided the supported switch/NIC/library combination is the one you buy. It brings credit-based losslessness, SHARP in-network reduction and a single support boundary. Ethernet is now a frontier answer too. The MRC consortium paper (2026) reports OpenAI and Microsoft running their largest training clusters on MRC over Ethernet. Disclosures from 2024 put xAI's Colossus on Spectrum-X and Meta on RoCE, while big GB300 clusters still ship on Quantum-X800 InfiniBand. Spending also favors Ethernet. Dell'Oro reported on 2026-09-03 that Ethernet extended its lead in AI back-end switch sales in 2Q 2026, the first quarter in which back-end switch sales passed front-end. Celestica, NVIDIA, Arista and Cisco were the top four Ethernet vendors. For a 3–5 year bet that means a deeper pool of Ethernet fabric engineers and more than one switch source to bid against each other, though Dell'Oro expects supply rather than demand to limit the market for at least one to two more years. Market share does not rank goodput, so your default is still whichever combination passes your own acceptance test.

(2) Supply-chain strategy. If single-sourcing your largest network capex on one vendor is strategically unacceptable, you are on the Ethernet side of the fork regardless of goodput. The only questions are which Ethernet, and whether the second supplier’s implementation actually interoperates.

(3) Time vs maturity. If you need near-InfiniBand goodput on Ethernet today, Spectrum-X is the finished product and you accept the NVIDIA switch+SuperNIC pairing. If you can absorb integration risk to own a multi-vendor future, UEC is the bet. If the workload is training-shaped and you want open multipath on current RoCE-class silicon, MRC is the narrower open bet, already carrying frontier production traffic, though a required Read verb excludes MRC 1.0.

(4) Tenancy. Hard multi-tenant isolation pulls toward Ethernet + DPU, where VPC virtualization and microsegmentation are native and the memory, confidentiality and performance controls that must survive tenant compromise live.

Whichever rule selects, the tested combination must meet the same healthy and degraded workload deadline, recover from loss and duplicates without releasing an incomplete result, and name who owns the pager and firmware lifecycle.

Two anti-patterns recur. The first is buying link rate instead of goodput: specifying a faster port and getting a slower cluster because the transport collapses under collisions. The fix is to benchmark delivered all-reduce bandwidth, not read the port label. The second is adopting RoCE without the team to tune it: capturing Ethernet's switch discount while forfeiting its goodput, so the “cheaper” fabric misses its communication deadline and idles the accelerators it was supposed to feed. Both come from deciding on the spec sheet rather than on delivered goodput and total operational cost. Price the fabric, validate it, and staff it against the goodput it has to deliver. → The topology and oversubscription decisions that sit on top of this transport choice are in Chapter 8.5; the traffic characterization that motivates all of it in Chapter 8.1; the scale-up domain this fabric stitches together in Chapter 8.2.