Network Fabric Commissioning & Validation

By the time the network team arrives, the hall has load-banked power, flushed and proven cooling, and an integrated-systems test that pulled the plug and watched the building survive (Chapter 13.6). It does not yet have a fabric anyone can trust. The cables are run, the optics seated, the switches racked and powered. Somewhere among tens of thousands of links sit marginal transceivers, a handful of rails cross-patched into the wrong leaf, a few ports negotiating the wrong FEC, and one boundary clock nobody configured. None of them announce themselves. They surface later as a 4,000-GPU job running at 70% of the bandwidth it should, a collective that intermittently times out, or an incident whose telemetry timeline is too blurry to read.

Fabric commissioning drags those defects into the light before the cluster is declared ready. Each acceptance gate (the BER threshold, the topology-verification method, the synthetic-versus-real-workload seam, the time-sync gate) carries a specific downstream cost when it is set wrong. Defects are cheapest to find at the lowest layer that can reveal them. A marginal optic found at layer 1 is a five-minute swap by a technician already on the floor. The same optic found after training starts costs days of wall-clock time, a forensic hunt across thousands of streams, and a straggler that throttles a synchronous job to the speed of its weakest link. Commission the fabric bottom-up (physical, then topology, then point-to-point, then timing) and a population of latent defects becomes a finite punch list. Skip it, and you ship that punch list to operations with the GPUs already spinning.

Fabric commissioning runs strictly bottom-up, because debugging cost sets the order. A layer can be trusted only once the layer beneath it is clean, since a lower-layer defect masquerades as a defect at every layer above. A marginal optic (layer 1) shows up as packet loss (layer 2/3), then as collapsed RDMA bandwidth (transport), then as a slow all-reduce (collective), then as a straggler (the job). Start the diagnosis at the top and you spend days chasing a software ghost that is really a $200 transceiver. So validate the bottom first, and refuse to run any higher-layer test over a link that has not passed the one below it.

The sequence has six steps: (1) physical-layer acceptance, starting from the installer's certified passive plant, with every link clean, every optic in spec and every port stable under soak; (2) topology validation, with every link landing exactly where the cabling map says, on the right rail and tier; (3) point-to-point bandwidth and latency, where synthetic RDMA traffic confirms each path delivers its design bandwidth and latency; (4) congestion and QoS verification, with PFC/ECN/QoS behaving as configured under deliberate stress; (5) scale-up (NVLink) validation, proving the intra-node and intra-rack domain distinct from the scale-out fabric; and (6) the timing plane, with PTP accuracy gated as an acceptance criterion. Cluster-scale collective benchmarking sits one layer above this chapter and needs real GPUs: NCCL all-reduce sweeps, OSU/MLPerf, and the bisection-bandwidth acceptance number that becomes a contractual SLA. It lives in Chapter 13.9. The seam is deliberate. This chapter proves the fabric is sound with synthetic traffic; the next proves the cluster is fast with real workloads.

The first gate is the physical layer, where the largest population of defects lives. Modern AI fabrics run PAM4 SerDes at 100–200 Gb/s per lane (800G and 1.6T links). That signal regime is so tight that every link needs forward error correction to work at all. The raw, uncorrected channel, the pre-FEC BER, is error-prone. FEC (RS-544 KP4 on Ethernet, the equivalent on InfiniBand) corrects it to a post-FEC BER that is meant to be effectively error-free. Read both numbers, because they answer different questions. Post-FEC BER says whether the link is currently delivering clean frames. Pre-FEC BER says how much margin FEC is burning to do it, and a link that runs clean today on the last few dB of FEC headroom will start dropping frames the first warm afternoon. At 200G per lane, some optics in the IEEE P802.3dj draft add an inner FEC inside the module on top of the RS(544,514) outer code. The switch's FEC counters then no longer see the optical margin, so read the module's media-side pre-FEC BER through CMIS VDM as well, and record link-layer retry counters where the link uses retry, because retry masks errors just as FEC does (Chapter 8.9).

Before any optic lights, take the installer's passive-plant certification for every fiber: continuity, insertion loss and reflectance measured with the contract's reference method and uncertainty, an end-face inspection record for every connector, and polarity checked against the keyed map. Chapter 8.9 sets the loss budget and Chapter 8.10 the polarity scheme and end-face drawings. Inspect and clean every end face before it mates, because a contaminated ferrule can erase more margin than the whole fiber run. Make certification the entry rule: a link that fails BER on an uncertified channel goes back to inspection and cleaning first. Without that rule, the BER screen turns into a cleaning campaign run one link at a time instead of a hunt for marginal optics.

A standard's interface BER objective, or a threshold from someone else's sweep, is not an installed-link threshold. Your acceptance plan has to name the Ethernet or InfiniBand generation, lane rate, FEC, measurement interface (for example MAC/PLS versus device/link counters), instrument and counter semantics, traffic pattern, dwell, temperature, sample size and statistical confidence. Express BER as a maximum (ceiling), then read pre-FEC margin and uncorrectable or frame-loss behavior against the named OEM and standard profile. Flag any link whose pre-FEC BER sits above its optic's spec, even if FEC is still masking it, for an optic swap, reseat or cable replacement before any higher-layer test runs over it. This is the highest-yield step in fabric commissioning. The proposed 512-GPU map in Chapter 8.5 has 1,024 back-end optical channels and 2,048 module ends to screen for marginal optics after factory integration, and finding them here costs hours of a technician's time. Finding them after the proxy training run starts costs days of wall-clock time and a straggler hunt across the whole job.

The second physical-layer screen is link-flap. A link that trains up, drops and re-trains, even once an hour, will eventually drop mid-collective, and on a synchronous fabric one transient flap can stall the entire job. Zero every port's flap counter at baseline, then run a sustained soak under line-rate synthetic load at realistic inlet temperature. Take the dwell from the project and OEM test plan (Keysight presents a 24–72-hour long-term interconnect example). Watch for any port that logs a single flap, a single FEC-uncorrectable event, or a creep in pre-FEC BER. Soak is where thermal-marginal optics reveal themselves: a transceiver that passes a cold-start BER check at 22 °C can fail once the hall warms to its operating point. A soak run cold has not tested the link you will operate.

A link can be physically perfect, with zero flaps and BER far inside spec, and still be wrong because it lands on the wrong port. In a rail-optimized design, corresponding GPU/NIC positions belong to the same rail, and a rail can span several leaves and spines. GPU 0 belongs to rail 0 and GPU 1 to rail 1, for example, and their matching positions on other nodes can land on additional leaves in those rails. A plane is a separate fabric attachment, not another name for a rail. Reconcile the actual endpoint-to-port assignments against the list in Chapter 8.5; same-rail traffic stays on one leaf only when both endpoints attach there. Suppose a rail-3 endpoint is cross-patched into a leaf assigned to rail 5. Nothing at the physical layer need complain: the link trains, BER is clean and the soak passes. But that endpoint can lose its intended rail path and stall every collective that needs its rank. Re-patch against the rail/port map, then prove the endpoint either forwards correctly or fails and restarts within its budget, because the wrong leaf does not guarantee an alternate route through a spine. BER screening cannot catch this defect class, and it is common at scale: with tens of thousands of cables run by hand, even a fraction of a percent miswired is dozens to hundreds of wrong links (0.5% of 40,000 is 200).

Topology validation is therefore a distinct, mandatory gate: an automated reconciliation of the discovered topology against the designed cabling map. The fabric manager (UFM and ibdiagnet for InfiniBand, the equivalent SDN/telemetry controller for Ethernet, NMX for the NVLink domain) enumerates every link's two endpoints by switch GUID and port. A script diffs that discovered graph against the reference architecture's expected adjacency: every leaf-to-spine link, every node-to-leaf link, every rail assignment. NVIDIA's UFM Cable Validation Tool is one implementation. It compares discovered neighbors against a topology file and reports wrong-neighbor, wrong-port and extra-cable links. Mismatches print as a punch list of specific ports to re-patch. Run this before point-to-point bandwidth testing, because a mis-cabled topology produces bandwidth numbers that look like a congestion or tuning problem, and you will burn days chasing the wrong layer. Verify the graph is the graph you designed, then test how fast it is.

With the physical layer clean and the topology proven, you measure whether each path delivers its design bandwidth and latency, using synthetic RDMA traffic rather than real collectives. On InfiniBand the canonical tools are the perftest suite: ib_send_bw and ib_write_bw for per-pair bandwidth, and ib_send_lat/ib_write_lat for latency. Perftest times latency as a round trip and reports half of it, so the figure is an estimated one-way latency, not an independently measured direction. On RoCEv2 the same perftest binaries run over the Ethernet fabric, alongside rping and vendor equivalents. Set per-path payload and completion-tail bounds from the qualified host, memory, message size, queue depth, transport and port map in Chapter 8.4/Chapter 8.5. Record the tool version, command line, CPU/NUMA placement, MTU and warm-up with each result. A switch-hop latency or another deployment's line-utilization percentage does not set this pair's acceptance threshold. If a result is low after BER and topology pass, look at endpoint placement, PCIe, memory, transport and queue behavior, and isolate those causes before changing congestion settings.

Point-to-point comes before collective benchmarking for debugging isolation. A pair-wise ib_write_bw test exercises one host-to-host path, so a low number narrows the search to that path and its two endpoints (the NICs and their PCIe lanes, memory placement and NUMA binding, and the hops in between) rather than the fabric at large. An all-reduce sweep exercises thousands of links at once, so a low number says the aggregate is slow without saying which link dragged it down. Establish per-pair correctness first, then layer collective behavior on top. The collective sweep waits for Chapter 13.9 because a true NCCL all-reduce acceptance run needs real GPUs driving the NICs, which overlaps node burn-in (Chapter 13.8) rather than pure fabric commissioning.

A fabric that benchmarks beautifully on idle point-to-point tests can still collapse when real collective traffic creates contention. The mechanisms that handle contention (priority flow control, ECN/DCQCN, adaptive routing, QoS classes) are exactly what idle tests never exercise. Congestion verification deliberately creates the failure case: an engineered incast (many senders to one receiver), a synchronized many-to-one pattern that mimics an all-reduce's reduction phase, and sustained saturation across shared spine links. Under that load, watch the congestion telemetry (PFC pause frames, ECN marks and CNP responses, switch egress drops, queue occupancy) and confirm the fabric does what its configuration claims.

This gate exists to catch the pathologies of lossless Ethernet (Chapter 8.6): a PFC deadlock or pause storm that propagates backward and freezes a whole region of the fabric; head-of-line blocking, where one congested flow stalls unrelated flows sharing a queue; victim flows dropped because a misconfigured ECN threshold never engaged; and QoS classes that do not actually isolate, such as storage traffic starving compute traffic or the reverse. On a RoCE fabric, trace ECN marking, reverse CNP delivery and sender response using the DCQCN recipe in Chapter 8.6. Test a watchdog event separately. It can drop traffic to break a pause condition, so acceptance requires the documented recovery action and correct bounded completions, not an unconditional zero-drop guarantee. Verify that the chosen path distribution respects the ordering rules in Chapter 8.4. The rule is the same as for the timing plane: an operational mechanism tested only in the happy path has not been tested.

Installed-fabric acceptance record. Before the first fault, the test lead freezes the serials, port/cable map, software and configuration hashes, and the traffic manifest: workload, seed, rank placement, message sizes, concurrency and commands. Each fault also gets its own completion, error and restart deadlines; the ≤1 s forwarding budget is not an endpoint deadline. The network owner records injection and clear timestamps, affected paths, FIB and queues. The endpoint owner records status, retries, timeouts and completion tails. Join the two under the timestamp-error requirements in Chapter 8.7, and retain raw counter deltas, resets and wraps, sampling intervals, missing samples and artifact hashes. A record too coarse to tell one cause from another does not pass.

Pass the physical, topology and matched healthy perftest runs first. Then inject one fault at a time and replay identical traffic after convergence and after restoration. Exercise every send, read, write and atomic operation the workload needs, and check destination data and completion status.

Simulated result. In rail 3, drive 32 ib_write_bw pairs that cross the SU cut, then take down spine Y, one of the two spines serving that rail in Chapter 8.5's port map. Forwarding recovers in 0.6 s against the 1 s bound. Converged cross-cut payload is 705 GB/s per direction, against the 720 GB/s Chapter 8.5 predicts and the 600 GB/s floor. Every destination checksum matches, and pause clears without a watchdog event. Through 8.5's step formula, 705 GB/s predicts a degraded step of about 1,030 ms, inside 1,100 ms, so the simulated profile passes. The installed system is not accepted until real traces exist, and after node qualification in Chapter 13.8 real GPUs must still meet the step limits in Chapter 13.9.

Flip. A converged payload of 585 GB/s, what thirteen healthy links deliver, predicts about 1,110 ms and fails, even with a 0.6 s recovery and clean counters on every live link. Find the missing capacity link by link before touching DCQCN settings, because congestion tuning cannot restore links the fabric has lost.

Release to Chapter 13.9 after hardware and test inputs are frozen, every fabric row has an attributable passing run and restoration reproduces baseline. Missing records, incorrect data, a missed recovery bound or a persistent pause blocks the release. Final workload acceptance still needs the healthy and degraded step limits on real GPUs. Releasing early hands an unresolved fault to the most expensive part of bring-up.

Everything above concerns the scale-out fabric, the InfiniBand/Ethernet network of NICs, leaves and spines that carries inter-node collectives. The scale-up fabric is a different animal (Chapter 8.2): the NVLink/NVSwitch (or UALink, or vendor-equivalent) mesh that binds the GPUs within a node or rack into a single coherent accelerator, at roughly an order of magnitude more per-GPU bandwidth than the scale-out NIC. It gets its own gate because it fails its own way. A degraded NVLink lane does not drop packets you can count on a switch. It silently holds intra-domain bandwidth below spec, capping tensor- and expert-parallel throughput inside the node without tripping any scale-out alarm.

Validation starts by querying the NVLink/NVSwitch topology to confirm the domain is whole: an NVL72 should present all 72 GPUs in one NVLink domain with every expected link present, not 71 with one GPU silently demoted to PCIe. Then measure per-link NVLink bandwidth and confirm each lane runs at full rate against the installed platform's datasheet. GB200 and GB300 NVL72 (NVLink 5) are rated at 1.8 TB/s per GPU and ~130 TB/s per rack. Vera Rubin NVL72 (NVLink 6, shipping spec) is rated at 3.0 TB/s per GPU and 216 TB/s per rack. Checking a Vera Rubin domain against the Blackwell figures would pass a link running well below spec. The gate also confirms the boundary between the two fabrics is clean: scale-up traffic stays on NVLink and does not spill onto the scale-out fabric, and the IMEX/MNNVL partitioning is configured so the domain is schedulable as designed.

NVIDIA's DCGM 4.7 gives the gate its tools: nvloom validates the multi-node NVLink fabric, mnnvbandwidth checks cross-node reachability and bandwidth, and an IMEX health check watches the domain. Log the DCGM and driver versions next to every result. Then break things on Vera Rubin NVL72, where NVIDIA's September 2026 NVLink 6 resiliency article documents the recovery behavior: the NMX controller migrates to another tray within seconds, forwarding continues through a switch-CPU reset, link errors are contained and rebalanced, and a replaced tray's links stay out of service until someone verifies them. Take down the tray that hosts the NMX controller and time the takeover. Reset a switch CPU while traffic runs, inject link errors, and swap a switch tray. NVIDIA describes the behavior; the timings you measure are yours to record. On earlier generations, drill only what the vendor documents for that generation.

NVLink links that run between racks over optics get the same optic screening as the scale-out fabric. NVIDIA's March 2026 design account sketches one such domain, the eight-rack Rubin Ultra NVL576, as a forecast design; its timing is contested (Chapter 8.2). Apply BER, FEC margin, retry and flap acceptance to those rack-to-rack links. Inside today's NVL72 racks, the scale-up domain is still all copper.

Chapter 8.7 made the case for a real PTP timing plane and insisted that time-sync accuracy be a commissioning acceptance gate, not a configure-and-hope step. This is where that gate is executed. A timing plane that is designed but never measured is trusted on faith, and faith fails at the worst moment: the night a 4,000-GPU run stalls and the forensic reconstruction across thousands of telemetry streams turns out to be a smear because every clock was drifting by milliseconds. The gate turns "we configured PTP" into "the clock is provably trustworthy."

The gate has three measured criteria. First, offset-from-master across the fleet: a sweep confirming every node's PHC sits within an agreed offset bound of the grandmaster. Sub-microsecond is the target, and tens of nanoseconds is achievable on modern Spectrum/ConnectX-class silicon. Second, accuracy under load: the bound must hold with the data plane saturated, because PTP accuracy that degrades under traffic is the accuracy you lose during the incident you are trying to diagnose. Third, holdover and failover: inject a GNSS-loss event to confirm the grandmaster coasts on its oscillator within the drift budget for the required holdover window, and force a primary-GM failure to confirm BMCA elects the standby within spec. Run the gate for real. Pull the GNSS feed and measure the drift instead of trusting the oscillator datasheet, the same design-for-the-failure-case discipline as the black-building test in Chapter 13.6. Pass these three and the timeline every later incident is reconstructed against is real.

Chapter 8.7 hands this chapter a second gate alongside timing: out-of-band reachability with the production data plane severed. Sever it deliberately, then prove from the management network that every endpoint in the inventory still answers: BMCs, rack PDUs, switch consoles, CDU and cooling controllers. Prove also that a representative control action lands on each class: power-cycle a node, read and switch a PDU outlet, open a serial console, command a CDU setpoint. Trace the management path's own dependencies while you are there (DHCP, DNS, the jump host, the credential store, the console server's power feed) and confirm none of them ride the fabric that just failed. Record the endpoint inventory, the actions demonstrated and the recovery time. An OOB plane exercised only while the data plane was healthy is untested, and the night it is needed is the night the data plane is gone.

The same fabric-commissioning failures recur, because each one comes from skipping a layer or testing only the happy path:

• Accepting on post-FEC BER alone. Every link that delivers clean frames today passes, and the pre-FEC margin FEC is burning goes unread. The marginal optics pass commissioning and become production stragglers the first warm afternoon. Screen on margin, not on the masked symptom.
• Skipping topology validation because BER passed. A cross-patched rail trains clean and benchmarks low, mimicking a congestion or tuning problem. Days are lost at the wrong layer because nobody diffed the discovered graph against the cabling map first.
• Commissioning the fabric cold. BER and flap checks run at ambient and accepted, then thermal-marginal optics fail once the hall reaches its operating temperature. Soak at realistic inlet temperature, or you have not tested the link you will operate.
• PTP configured but never gated. Time-sync treated as a setup step instead of a measured acceptance criterion: no offset sweep across every node, no under-load test, no holdover pull. The first uncorrelatable incident shows the timeline was never trustworthy. → Chapter 8.7.
• No baseline captured. The fabric is declared ready without saving the as-commissioned fingerprint, which leaves operations with nothing to diff against and no way to catch a degrading link before it fails outright. → Chapter 14.2.

Accept the installed fabric on correlated physical, forwarding and transport test results, then hand the qualified paths and their failure limits to Chapter 13.9. Keep failed or unobserved paths out of service until they are repaired and retested. Accepting on a bandwidth average instead buys an earlier handover, paid for with a longer, harder fault investigation during the workload run.