Deployment Velocity & Cabling at Scale

You have fought for an interconnection slot, energized a hall, plumbed it for liquid and taken delivery of costly racks. None of it earns a dollar until the cluster passes its contracted training or serving release. Between the loading dock and that milestone sits a physical-layer buildout that almost nobody scopes as a first-class engineering problem until it holds up go-live.

That buildout is install velocity: the rate at which racks are set, busway connections are made (isolated, locked out and verified dead first), manifolds are coupled and, above all, links are cabled and lit. Energized megawatts are the scarce input in a power-bound era, and they sit against a running depreciation clock. Install rate converts a date on the interconnection agreement into time-to-goodput. Two clusters with identical bills of materials and identical power can differ by months in time-to-revenue purely on how fast and how cleanly they cabled.

Three decisions run through a domain usually treated as commodity labor: field-terminate vs pre-terminate, screen optics off-line vs accept in-line, and prevent mis-cabling vs debug it at acceptance. Choosing wrong costs days of stranded capital and percentage points of lost goodput. This chapter owns the install act. The acceptance and validation that prove the cluster works (fabric BER soak, NCCL all-reduce, the reference run) live in Part 13. See Chapter 7.14 for where the rack gets built (the L1–L12 model) and Chapter 8.10 for the structured-cabling fiber plant deployed here.

The right unit of program velocity for an AI cluster is racks set per day and links lit per day, sustained, on the floor, with first-time-right quality. Megawatts per month and racks shipped are the wrong measures. Everything upstream (the master schedule, long-lead procurement, factory integration) exists to feed a floor that absorbs work at a finite rate. Everything downstream (acceptance, the reference run, first revenue) cannot begin until that rate has carried the buildout to completion. Install rate is the throttle on the whole pipeline.

The arithmetic is dominated by fan-out. A single GB300 NVL72 rack is one set-in-place event but on the order of 5,000+ physical connections. Three groups of connections matter. The first is ~5,184 in-rack copper twin-ax NVLink cables forming the scale-up spine. On Vera Rubin NVL72 a printed-circuit-board midplane replaces the cables inside each compute tray. That removes terminations at factory integration, not on the floor: the copper NVLink spine and the scale-out fiber still run as cables (Chapter 7.13). The second is at least 54 tray-level liquid connections (a supply and a return quick-connect for each of the 27 liquid-cooled trays, 18 compute and 9 switch, per the Lenovo GB300 NVL72 product guide), before counting any inside the trays. Each is a leak-test point, so get the rack maker's connection schedule for the exact revision. The third group is the one this chapter cares about most: the scale-out back end. Back-end bandwidth per GPU sets its size: 400G on GB200 NVL72 (ConnectX-7), 800G on GB300 NVL72 (ConnectX-8, run as two 400G planes) and 1.6 Tb/s on Vera Rubin NVL72 (ConnectX-9), the rack shipping since June 2026. In 800G equivalents that is 36, 72 or 144 back-end ports per rack, alongside front-end and management ports.

Every back-end port needs a link at each switch tier above it. Counted per rack in 800G equivalents, a GB300 rack needs about 144 links in a two-tier fabric and 216 in a three-tier one; a Vera Rubin rack needs about 288 and 432. Whether it is DR8, 2×DR4 or SR8, a parallel-optic 800G link consumes 16 fibers. Fiber quantity follows the topology: count the links, multiply by fibers per link, then add redundancy and patching overhead. Sterlite Technologies, a fiber maker, puts the multiple high: it told investors in November 2025 that GPU-intensive data centers need up to 36 times more fiber, without naming the baseline.

The tier count matters most, and it depends on how each GPU's bandwidth is split into planes. A 102.4 Tb/s switch run as 512 × 200G ports reaches about 131,000 GPUs in two non-blocking tiers (512²/2). That case fits GPUs whose bandwidth is carried on 200G planes: four per GPU in a quad-plane 800G design, eight for Vera Rubin's 1.6 Tb/s. GB300's two 400G planes run the switch as 256 × 400G, and two tiers then stop near 32,000 GPUs (256²/2). Run the same switch as 128 × 800G and two tiers stop at about 8,000 GPUs (128²/2). Past whichever limit applies, a third tier is needed, adding a link per GPU. Corning's CEO, on the Q2 2026 call (July 2026), called that roughly 50% more optical content for every GPU. Your crew budget scales with it. Multi-plane designs are how large clusters stay at two tiers; topology sizing is in Chapter 8.5.

Set the rack in an hour; it does not produce goodput until every one of its external links is cabled to the correct port, with correct polarity, inside its loss budget, and verified. The link is the unit of work, and the link count is what sets the schedule.

A velocity-engineered program inverts the naive sequence. The naive plan treats cabling as a trailing activity after racks land. The engineered plan treats cabling capacity as the binding constraint and works backward: it stages pre-terminated trunks before racks arrive, screens optics before they reach the floor and verifies topology continuously rather than in one terminal sweep. The reward is a floor that never stalls waiting on a cable crew. The penalty for skipping it is a common and expensive failure mode: a fully energized, fully rack-loaded hall that cannot pass acceptance because thousands of links are mis-mapped, marginal or not yet run.

If install rate is the metric, pre-terminated structured cabling moves it most. It takes termination, the slow, skill-intensive, error-prone step, off the critical path and into a factory, where it is done under controlled conditions, tested and labeled before it reaches the floor. On site, a crew no longer splices. It routes trunks and snaps in cassettes, which is fast, repeatable and far less dependent on rare certified-tech hours. Skilled trades are short across data-center construction, electricians above all, and certified fiber technicians are a scarce slice of that pool, so moving termination off the floor removes one of the few schedule risks the owner can buy out in advance.

The density argument compounds the velocity argument. A high-fiber MPO trunk consolidates many discrete cables into one routed assembly: a single 144-fiber MTP/MPO trunk can carry the equivalent of eighteen MPO-8 runs. A cluster that would otherwise route thousands of individual jumpers then routes tens of trunks, broken out at the rack by cassettes. When it replaces like-for-like discrete cables, high-density trunking reduces pathway congestion, tray weight and airflow obstruction. Fewer physical objects to route means fewer chances to mis-route, so density and quality improve together.

Staging is what makes this real. Velocity-engineered programs build a structured-cabling staging operation, a kitting and pre-lay workflow often run from a tent or warehouse next to the hall. Trunks are received, inventoried against the port map, labeled to the as-designed topology, pre-laid into trays or coiled at the rack positions, and held ready before the racks they serve arrive. When the rack lands, its links are already waiting at the cabinet, labeled and tested; the crew connects, not constructs. Investing in staging decouples cabling progress from rack-delivery timing, and it is the practical core of the fastest documented buildouts.

Forward-proofing rides alongside. Use a certified OS2 plant where its reach and lifecycle economics fit, but carry an explicit application migration map. 800GBASE-DR8 uses eight parallel single-mode lanes over 16 fibers. Whether 1.6T or 3.2T can reuse the same plant still depends on the lane and wavelength map, fiber count, connector and polarity, channel loss and reflectance, reach, module and host requirements. Re-cabling a live hall is a major velocity event, so test reuse against the new application's requirements before scheduling an upgrade, and never assume OS2 makes it a transceiver-only swap. The single-mode-vs-multimode and MPO-polarity engineering is canonical in Chapter 8.10; the copper-inside/optics-outside reach frontier that decides which links are even fiber is in Chapter 8.9.

An optical transceiver can pass at the bench and fail under load. Thermal margin is an acceptance variable: require the supplier's laboratory TDECQ results for the optical interface, and have field acceptance log DOM/DDM temperature, power, pre-FEC BER and link errors under contracted traffic and temperature. A link that comes up clean during a quiet install can start flapping the moment the cluster is loaded. That is when it costs most, because in a synchronous training fabric one flapping link stalls the thousands of GPUs barriered behind it. Optical-link faults are a recurring source of "gray failures" in AI fabrics, and marginal transceivers can gate acceptance.

The decision this forces: screen transceivers off the critical path, or perform the contracted screen in-line. Off-line screening means soaking modules at temperature, under traffic, against a BER/pre-FEC threshold, on a vendor screening line or in your own staging operation. You bin out the marginal population before it reaches a switch cage on the floor. Infant-mortality and thermally marginal units fail in the tent, on your schedule, where a failure costs a swap and a note rather than a stalled acceptance run or a production goodput hole found weeks later. In-line acceptance (plug everything in, run the fabric soak, chase whatever flaps) can cost less to set up. But every marginal module then surfaces as a debugging excursion on the acceptance critical path, and the ones that pass on install day but fail under production thermal load become the operations team's problem indefinitely.

Co-packaged-optics (CPO) switches move part of this screen out of your hands. Their optical engines sit inside the switch package, so there is no switch-side module to soak in the tent. In NVIDIA's design the part a technician swaps on site is the external laser source (ELS) module, which NVIDIA says comes out without disturbing the rest of the switch (NVIDIA technical blog, August 2025). Whether an engine itself is field-replaceable varies by product. The server-side ends of those links are still pluggables, so the tent stays. Budget acceptance time for a loaded soak of each CPO switch, and get the supplier's FRU list and engine-repair procedure before the switches land (Chapter 8.10).

No settled industry standard exists yet for soak duration or BER thresholds, and how much screening is economically optimal remains an open question. The economics are conditional. Catching a bad optic before install is a fixed, schedulable expense; catching it after is a failure-dependent cost paid in crew and GPU-hours. On a velocity timeline, screening pays when the avoided expected rework and delay exceed the station, labor and extra handling cost. The reliability physics, the pre-FEC/DDM telemetry that forecasts a flap, and per-link MTBF math are in Chapter 8.9; fabric BER-soak as an acceptance gate is in Chapter 13.7.

Ask anyone who has commissioned a large AI fabric what gated the green tag, and mis-cabling is one of the answers you will hear most often. In a rail-optimized topology, one link plugged into the wrong port (right cable, wrong destination) does not just lose a connection. It can collapse the collective bandwidth of an entire rail and silently degrade the all-reduce the training job depends on. A 100,000-GPU three-tier fabric at 800G per GPU is about 300,000 links, or millions of fiber terminations. At that scale, an error rate of just one link in a thousand still leaves hundreds of mistakes, and the acceptance team must find each one among correctly cabled links that look identical. Mis-cabling is not a quality nuisance. It is a first-order schedule risk of the acceptance phase, and the velocity-engineered answer is to prevent it rather than debug it.

Prevention is three disciplines, applied together. First, labeling and port-mapping: every trunk, jumper and port carries an unambiguous as-designed label tied to a port map and a digital twin, so the crew connects to a named destination rather than a guessed one. This is most of why pre-terminated, pre-labeled assemblies cut errors as much as they cut time. Second, polarity discipline: MPO systems have a polarity scheme (Method A/B/C) that must be consistent end-to-end across trunks, cassettes and jumpers, or the link is dark or crossed. Choose one method and enforce it across the whole plant. Chosen wrongly or applied loosely, polarity produces failures that are maddening to chase because the cable is physically fine. Third, automated topology verification: rather than a human tracing links, query the fabric's own LLDP/management plane to confirm that every observed neighbor matches the intended topology. That flags mis-maps continuously as the buildout proceeds, instead of in one terminal sweep at acceptance.

The trade-off is stark. Spend on labeling rigor, a single enforced polarity method and automated verification tooling, and mis-cabling becomes a continuously cleared backlog that never reaches the acceptance gate. Skip it, and mis-cabling becomes the acceptance gate: a multi-week manual hunt across a fully built hall, with the depreciation clock running and the reference run blocked. The cheapest place to find a mis-cabled link is the moment it is plugged; the most expensive is the cluster-scale NCCL benchmark. Commissioning Level 5 (the integrated systems test) and the fabric validation that surface these failures if they slip through are in Chapter 13.6 and Chapter 13.7; the rail-optimized topology whose geometry makes a single mis-map so destructive is in Chapter 8.5.

Colossus is the public reference point for install-velocity engineering, and it works best as a case study in every lever this chapter names, not as a stunt. xAI and its partners turned an abandoned Electrolux factory shell in Memphis into a live 100,000-GPU training cluster in 122 days. By xAI's account, they then doubled it to 200,000 in a further 92. The industry norm for systems of that size is measured in many months to years. The number a velocity engineer should focus on is the smaller one: 19 days from the first rack rolling onto the floor to the start of training. That gap between rack arrival and training start is the install act this chapter owns, compressed to nearly nothing.

Colossus sets the schedule target; the decision forks above define how to pursue it. As arithmetic, 100,000 Hopper GPUs at 64 per reference rack is about 1,563 rack-equivalents, or about 12.8 a day over the 122 calendar days. That denominator is the whole build, shell, power and cooling included, so the figure is neither an observed rack count nor a crew's set-and-cable rate. Pursue the target with parallel workstreams against pre-staged, factory-terminated, modular cabling, so rack-setting and link-lighting run concurrently rather than in series. The payoff shows in the network result: across all three tiers of the scale-out fabric, NVIDIA reports the cluster sustained roughly 95% throughput with zero packet loss from flow collisions. Clean, correctly mapped, in-budget cabling is there to protect that result: one lossy, dark or mis-mapped link drags down the collective that every GPU in the job is waiting on.

The honest part of the story is the chaos. Assembling 200,000 interconnected GPUs at that speed surfaced mismatched BIOS firmware, cable snarls and even cosmic-ray bit flips, the predictable tax of compressing a year of work into months. The engineering response is to move the slow, error-prone work (termination, optics screening, topology verification) off the floor and ahead of the racks, so the floor only does fast, parallelizable work. Colossus shows that install rate is an engineerable quantity, not a fixed property of physics. The integrated master schedule and critical-path discipline that makes parallel workstreams possible is in Chapter 2.1; the long-lead procurement that must land the pre-terminated cabling and screened optics in time is in Chapter 2.3.

How much of this chapter's work lands on your floor is itself a decision, fixed upstream at the L11/L12 integration boundary. Pull cabling and optics integration into the factory (racks delivered with in-rack cabling done, manifolds coupled, optics seated and screened, the assembly tested as a unit) and the field buildout shrinks to inter-rack trunking and final fabric lacing, cutting floor time from months toward weeks. Push it into the field (barebones racks integrated on site) and you own the full physical-layer buildout at the slowest, most labor-constrained point in the chain. Factory integration is a velocity choice before it is a logistics choice, which is why the same rack BOM can show wildly different time-to-goodput depending on where it was built.

The countervailing pressures are real: heavier, more fragile shipments, the logistics and risk of shipping wet or dry, and serviceability and RMA terms that change once more is sealed at the factory. The point for this chapter is narrow and firm: the further toward the factory you set the integration boundary, the less of the install-rate constraint lands on your critical path, and the more your time-to-power advantage converts into time-to-goodput instead of evaporating against a cable backlog. The L1–L12 manufacturing-level model, the build-vs-buy fork, and the wet-vs-dry shipping tradeoff are owned in Chapter 7.14; the acceptance and validation that proves the installed cluster works (burn-in, fabric BER soak, the reference training run, go-live and handover) are owned across Chapter 13.6, Chapter 13.7, Chapter 13.8 and Chapter 13.9.

Links = 72 × 16 = 1,152; installed modules = 2,304. Screen ceil(2,304/0.95) = 2,426 modules. Station capacity = 4 × 32 × (8/2) = 512 modules/day; ceil(2,426/512) = 5 screening days before floor work. Let W = 1,152 × 0.25 × 1.10 = 316.8 crew-h. Four crews provide 24 crew-h/day: ceil(W/24) + 1 = 15 floor-work days. Six provide 36 crew-h/day: ceil(W/36) + 1 = 10 days. With six crews, the nine installation days allow 324 crew-h; maximum rework = 324/(1,152 × 0.25) − 1 = 12.5%. Above 12.5%, installation rounds to a tenth day and release moves beyond day 10.

In this case, stage the screened inventory and fund six crews: more screening stations cannot fix a floor that four crews cannot finish in time. Release each link only when its endpoint and polarity record, end-face inspection and cleaning (Fluke Networks, fiber inspection and testing guidance), module identity, optical loss and link-error tests meet the contracted limits. TDECQ stays a supplier laboratory test; the installed fabric soak and reference run belong to Part 13 (Chapter 13.7).

Now scale the pod to a real GB300 back end: 144 links per rack across two tiers, counted in 800G equivalents, with plane splits adding physical cables on top. The same rates then need about 53 crews to finish 10,368 links in nine installation days, and about 35 stations to screen 21,828 modules in five. Read both as floors.