Scale-Across: Multi-Campus & Cross-Region DCI for Portfolio and Qualified Cross-Site Workloads

The first two network tiers in this Part work from the inside out. Scale-up binds a defined accelerator group, such as the 72 GPUs of a GB200 NVL72, through supported peer operations and collectives (Chapter 8.2). Scale-out, a Clos or other chosen fabric, stitches scale-up domains into the campus’s larger GPU cluster (Chapter 8.5). This chapter covers the third tier, scale-across: optional DCI between sites for portfolio capacity, replication, service failover, burst or overflow capacity, and cross-site computation for jobs that pass the tests below. The binding constraint of the 2026 era is megawatts at the substation, not accelerators in the supply chain (Chapter 16.1). Portfolio capacity can split among sites without forcing one synchronous job to span them. A job crosses only when its algorithm, latency and bandwidth budget, placement plan and correlated-fault model have each been proven for that boundary.

Cross-site training is one demanding use case, not the definition of the tier. Google trained Gemini Ultra across multiple data centers on TPUv4 pods, combining SuperPods over its intra- and inter-cluster network with latency and bandwidth sufficient to keep the run synchronous. That is the origin case, and the hardware under the tier has turned over since. Microsoft’s Fairwater is the current cross-region example. Microsoft describes an AI WAN connecting Wisconsin and Atlanta for training in which GPUs across sites update the model together, with 120,000 miles of dedicated fiber across the network, some newly built and some repurposed (November 2025). Microsoft also says the WAN lets it assign diverse workloads across the combined system dynamically. Route lengths, step times and the synchronization scheme are unpublished, so read Microsoft’s account as a description of the architecture, not a latency result (→ Chapter 1.2).

Meta gives the most detailed operator account. It runs Prometheus, about 1 GW in 2026, as one cluster across several buildings in one region, joined by an Ethernet super-spine it calls the backend aggregation (BAG) layer. Meta reports that latency between building fabrics is several times the latency inside one, and that collective libraries that ignore that distance can fall off a performance cliff. Its answer is adaptive routing plus job scheduling that keeps as much collective traffic as possible off the BAG layer (Meta, October 2026). Across the industry, campus-scale clusters are not all NVIDIA: TPU, Trainium, AMD Helios and other ASIC systems deploy at the same scale, and TrendForce's April 2026 forecast puts ASICs at ~27% of 2026 AI-server shipments. OpenAI's Stargate program plans more than 8 GW across seven U.S. sites (October 2025), a portfolio placed where power can be delivered rather than one synchronous machine.

The three-tier hierarchy is the core mental model for this Part, and scale-across is where the numbers drop by orders of magnitude. Inside scale-up, NVLink supplies the platform-specific peer operations and per-GPU bandwidth recorded in Chapter 8.2. Scale-out hands exchanges to the NICs, 400 Gb/s to 1.6 Tb/s per GPU depending on the generation (Chapter 8.5), with topology, software and workload costs budgeted in Chapter 8.1. Neither a switch-hop latency nor a peak interface rate is an application collective time. Scale-across drops again, by orders of magnitude on both axes at once. Per-site DCI capacity is the number of fiber pairs you light times what each line system carries. Coherent Corp.’s 800G module announcement puts one fiber at about 32 Tb/s in the C-band and about 64 Tb/s once the L-band is added (September 2024). Frontier operators buy at petabit scale: Meta’s links between BAG layers carry 16–48 Pb/s for a pair of regions (Meta, February 2026). The latency floor comes from propagation in the glass: standard single-mode fiber adds ~5 µs per kilometer, one way. Hollow-core fiber guides light through air instead, at roughly 3.4 µs/km, about a third less. Microsoft has run hollow-core on some Azure routes since 2023. But it is a route-specific build and in short supply, so do not assume it on leased or existing fiber.

The 5 µs/km figure for standard fiber governs everything downstream in this chapter. Light moves at ~300 m/µs in vacuum but ~30% slower in silica (refractive index ~1.47), so standard SMF-28 fiber adds ~4.9 µs/km one-way: call it 5. A campus-to-campus link 40 km apart has a ~200 µs one-way floor and ~400 µs round trip, before a transponder, amplifier, FEC block or switch hop adds its own delay. Two regions 1,000 fiber-km apart sit at ~5 ms one-way, ~10 ms RTT. An all-reduce that fits a short local exchange can spend milliseconds waiting on a WAN route, and a synchronous step pays that delay for each dependent round on its critical path. MFU falls when those waits displace useful compute; large local compute intervals can absorb delays that a fine-grained collective cannot. Relaxed synchrony, gradient compression and hierarchy, the algorithms later in this chapter, all exist to hide or amortize that floor.

Before any optics or algorithm, decide why the portfolio needs more than one site. Separate independent jobs, replication, service failover and overflow capacity from the decision to split one training job. A power or delivery constraint can make another site valuable without justifying a synchronous WAN dependency. If a job must cross, price its exposed communication and correlated-failure cost against the alternative schedule, using the project’s utility and equipment need-by dates. It is still a race between two clocks: depreciation on accelerators already bought, and the utility-service or upgrade schedule for the megawatts they need. Idle silicon loses value while it waits, but splitting one job adds exposed communication and correlated-failure cost, and independent jobs at the second campus may capture its power without that WAN dependency. → portfolio rationale in Chapter 16.1; siting and delivery in Chapter 3.2.

Once you decide to split, ask what the second site is for. The two answers buy very different amounts of fiber. A training peer runs part of the synchronous (or relaxed-sync) job and exchanges gradients or pseudo-gradients with the primary on a cadence the algorithm sets. It is bandwidth-hungry and latency-sensitive. A checkpoint-and-burst replica holds a copy of model state and absorbs elastic inference or batch overflow. It needs enough bandwidth to ship checkpoints (gigabytes to terabytes on a slow cadence) but does not care about per-step latency. Mixing these up is a classic overspend: peer-grade DWDM for a site that only needed checkpoint-grade bandwidth, or the reverse, starving a true training peer and watching the run stall on a link you under-bought.

A third arrangement sits between them. Consider a site that owns entire stages of the workload, for example RL rollouts, evaluation or synthetic-data generation. It trades datasets and checkpoints with the training site instead of per-step gradients, so replica-grade bandwidth is enough for those transfers. A rollout site also pulls refreshed policy weights each time the policy updates, and for a large model that can outweigh the checkpoint traffic, so size its link to the refresh cadence. It still does not need per-step gradients. This is the guide’s design option; Microsoft has not said it places stages this way.

Inside the hall, the interconnect debate is copper versus optics over single-digit meters (Chapter 8.9). Across sites, traffic travels kilometers of glass, and the debate is which grey or coherent optical application carries it on your route, together with the protection and transport service it needs. The fork has three branches and one in-between option, and they trade reach, cost per bit and how much transport plumbing you own.

Grey optics over dark fiber light a point-to-point application with no wavelength multiplexing in a line system. Pick the exact PMD and reach. A four-lane parallel DR4 application uses four transmit/receive pairs, while a four-wavelength FR4/LR4 application combines wavelengths onto one pair, and each rate has its own PMD definition. Check which you have before pricing pairs or patching. Coherent-lite sits between grey optics and ZR. OIF’s 800LR agreement (April 2025) defines one fixed 800G wavelength over an unamplified point-to-point span of up to 10 km, carrying one 800GE or two 400GE clients, for campus and DCI links. Check it before paying ZR prices for a building-to-building link. 800LR’s reach ends at 10 km, and a route that runs through amplifiers on a DWDM line needs ZR-class optics. Coherent ZR/ZR+ pluggables move line termination into a supported router or switch port, which cuts separate transport equipment when the host supplies the power, cooling and control interface. OIF’s October 2024 800ZR agreement targets single-span amplified 80–120 km DWDM. Its C-band channel plan holds 32 waves of 800G, about 25.6 Tb/s per fiber pair each way; Coherent’s 32 Tb/s figure above implies 40 waves in a wider band. Chapter 3.6 sizes a two-site link with the OIF figure. Fitting coherent line termination into a supported QSFP-DD or OSFP port can replace separate transponders, but the line budget still has to work. A ZR+ label alone does not establish a regional reach or a shared interoperability mode. The next rate is now defined. OIF’s 1600ZR agreement, published in September 2026, carries one 1.6T Ethernet client per wavelength over the same amplified single-span reach as 800ZR. Cignal AI puts 1600ZR-class modules in 2027, so a buyer in 2026 usually gets 800ZR/ZR+. Before buying, check that your line system and host ports can move up to 1.6T. Full OTN / managed transport adds a separate service boundary for grooming, protection and operations. Choose it when those responsibilities belong with the transport provider. → channel physics in Chapter 8.9; fibers and repair access in Chapter 8.10.

The scale-across edge sits between the optics and the collective. A long RTT means a large bandwidth-delay product. At 40 GB/s and a 2 ms RTT, the bandwidth-delay product is about 80 MB per direction, more than a shallow-buffer switch built for microsecond RTTs can hold without dropping or pausing traffic. Vendors now market deep-buffer routing silicon for this boundary. Broadcom sells Jericho4, with its HBM packet buffer (Chapter 8.3), for scale-across and says it keeps RoCE lossless past 100 km. Cisco says systems built on its 51.2T Silicon One P200 are in production at Microsoft, and that Alibaba’s backbone uses a P200-based deep-buffer router (OCP 2026). NVIDIA’s Spectrum-XGS is Spectrum-X with congestion control and load balancing that account for distance. On a two-site link 10 km long, NVIDIA reports NCCL all-reduce bandwidth up to 1.9x that of standard Ethernet. All of these are vendor claims, and the NVIDIA test ran at metro distance.

Meta, an operator rather than a vendor, builds its BAG layer on deep-buffer Jericho3 line cards (February 2026). Its FBOSS switch software measures each installed cable’s length and sizes lossless buffer headroom to that measurement instead of to the worst-case distance. On BAG, eBGP link-bandwidth signaling spreads traffic in proportion to each path’s remaining capacity, damped so that one link failure does not churn routing across the domain. Meta’s regional backbone is a separate network that links its campuses to each other and to public cloud capacity, and it takes another approach: SRv6 routing over open shallow-buffer switches that use Cisco Silicon One (Meta, October 2026). Deep buffers buy lossless distance; traffic engineering buys survivable capacity. Know which problem your route has before you pay for either. Size edge buffers and PFC headroom to the bandwidth-delay product of your longest surviving route (Chapter 8.6).

The rest of the chapter’s decisions follow from this one. Once a job spans a metro or regional radius, you must choose how synchronization happens across the thin WAN pipe. The choice trades convergence quality and simplicity against inter-site bandwidth and tolerance to latency.

Branch A: synchronous scale-across. Treat the remote campuses as one extended fabric. Data- and pipeline-parallel groups that pass the step-budget test may span sites while tensor- and expert-parallel groups stay site-local, and every step still ends in a global all-reduce that now crosses the WAN. This preserves the synchronous update structure, and it is what Google did for Gemini Ultra; numerical reduction order and reproducibility still depend on the implementation. The cost is acute sensitivity to inter-site bandwidth and latency. You must overlap communication with computation aggressively, and place the parallelism dimensions so the chattiest collectives (tensor-parallel) stay inside a site and only the more tolerant ones (data-parallel, pipeline) cross the WAN. Bisection bandwidth across sites is your hard throughput ceiling. Get the placement wrong and the run waits on WAN bandwidth or dependent round trips while MFU falls. This branch is viable wherever the protected-route calculation and measured overlap keep exposed exchange time inside the step budget.

Branch B: relaxed synchrony (local-SGD family). Stop synchronizing every step. Each site runs many local optimizer steps independently, then the sites exchange and average their accumulated updates (“pseudo-gradients”) only periodically. This is the DiLoCo paradigm (Distributed Low-Communication): an inner AdamW optimizer runs H local steps, and an outer Nesterov-momentum optimizer averages across workers. DiLoCo on 8 workers matched fully synchronous optimization on C4 while communicating ~500x less, and OpenDiLoCo reproduced this training a 1.1B-parameter model across two continents and three countries at 90–95% compute utilization. Streaming DiLoCo (2025) syncs the model in fragments, one subset of parameters at a time, overlaps each exchange with compute and quantizes what it sends, which cuts peak bandwidth. Follow-on work (SparseLoCo, which combines top-k sparsification with 2-bit quantization) works out to roughly 30–94x less total traffic than DiLoCo and ~890–2,810x less than standard DDP, depending on sparsity. The cost is no longer bandwidth but convergence risk. Relaxed synchrony introduces staleness and can perturb final model quality, the inner/outer optimizer and sync interval are extra hyperparameters that must be tuned, and the failure modes are subtler than synchronous SGD’s clean restart from checkpoint.

Branch C: hierarchical or async hybrids. Between per-step WAN synchronization and fully independent local work sits a pragmatic middle that you must test before relying on it: synchronous within a site, relaxed across sites. Each campus is one tightly coupled synchronous fabric, and the campuses are coupled with local-SGD-style periodic averaging or bounded-staleness asynchrony. This matches the algorithm to the physics tier by tier: you pay the full synchronous cost only where bandwidth is cheap (inside the DC) and the relaxed-sync convergence tax only where it is expensive (across the WAN). Gradient and pseudo-gradient compression (quantization, top-k sparsification, low-rank or momentum decoupling) layers on top of any branch to shrink what crosses the WAN further.

If your sites are close enough that a global all-reduce fits the per-step compute budget, run synchronous scale-across. It protects model quality, and you pay for it in fat, expensive DWDM and aggressive overlap engineering. If they are not, buy more surviving payload, change placement, or adopt a hierarchical / DiLoCo-class algorithm with a less frequent or compressed exchange schedule. That trades fiber demand for convergence tuning and recovery work, so measure the traffic saving and check the same model-quality target before accepting it. The naive failure is to light a thin WAN pipe and run an unmodified synchronous job across it, leaving both campuses’ GPUs blocked on an all-reduce that physics won’t let finish in time. → checkpoint and fault-domain consequences below.

Worked decision: admit DCI after one route loss

Healthy communication is 8.0 × 2³⁰ B / (40 × 10⁹ B/s) + 2 × 0.0020 s = about 219 ms. The step is 1,000 ms + max(0, communication − 150 ms) = about 1,069 ms, which passes the 1,100 ms limit. After one route loss, communication is 8.0 × 2³⁰ / (20 × 10⁹) + 0.0040 s, about 433 ms, and the step is about 1,283 ms: it fails.

Communication must fit 150 + (1,100 − 1,000) = 250 ms. Removing the 4.0 ms dependent-round cost leaves 246 ms for bytes, which requires 8.0 × 2³⁰ B / 0.246 s, about 35 GB/s of surviving training payload. Buy that survivor capacity or refuse synchronous admission. At these rates the shortfall is 15 GB/s of payload, which one more 800G wavelength per route covers. At frontier step volumes the same arithmetic calls for whole line systems, and that is where the fiber bill bites. Switching to the relaxed mode streams 4.0 GiB per step at 20 GB/s: communication returns to about 219 ms and the step to about 1,069 ms. That passes the timing model, but the relaxed mode still has to show the same model quality and a clean recovery before you admit it. Sending a burst every second step would not prove either step’s deadline; the assumed streaming schedule has to be demonstrated.

Preserve the reservation after failure. If an assumed complete 1.0 TiB checkpoint is produced every 900 s and receives 10 GB/s of the surviving reserved service, the transfer takes 2⁴⁰ B / (10 × 10⁹ B/s), about 110 s, within the interval. That is a transfer check only; committed model, optimizer, RNG and metadata state plus a successful restore belong to Chapter 9.4. Verify route diversity by duct, entrance, line equipment and power, then measure failover and reverse feedback under load. The OIF client/line mapping supplies the transport boundary; Chapter 13.7 executes the installed acceptance.

Intra-DC reliability engineering already carries correlated failure: the shared bus, the shared CDU and the fleet-wide firmware push, priced as a beta factor in Chapter 12.5. But its day-to-day arithmetic runs on independent failures. A GPU dies, a NIC flaps, a cable goes bad, and you ride it out with hot spares and checkpoint-and-resume (Chapter 12.2). Scale-across adds shared-risk groups wider than any inside one building. A fiber cut between campuses, a regional grid event, a utility curtailment, a metro-wide cooling-water issue or a site evacuation takes out an entire site’s worth of accelerators at once, one correlated failure spanning thousands of GPUs. The DCI link itself becomes a first-class fault domain, and a flat intra-DC reliability posture no longer covers you.

Four things change. Path diversity stops being optional: inter-site fiber must be physically diverse (separate conduits, separate entrances, ideally separate carriers) or the WAN link is a guaranteed correlated-failure single point. Checkpoint placement must become site-aware, because a checkpoint that lives only on the campus that just lost power is no checkpoint at all. Relaxed synchrony leaves recent model copies at each site between exchanges. That can become a recovery advantage, but only with durable optimizer and RNG state, agreed membership and a committed global exchange boundary. Test restart from that boundary and rejoin after site loss; a local model copy alone is not a restorable global checkpoint.

Recovery becomes a fork. You can restart every site from the last global checkpoint, or exclude the failed site’s replica and keep training on the survivors. Meta reports the second pattern on its BAG-connected clusters: when a replica fails, it is excluded automatically and the surviving replicas keep synchronizing across the BAG layer (October 2026). Exclusion keeps the run going at reduced data-parallel width, but it needs elastic membership, a rebalanced global batch and a tested rejoin procedure. Restart is simpler to get right, and it idles every site for the length of the restore. Blast-radius accounting must treat “site” as the failure unit. The question is no longer what happens when a node fails but what fraction of the run survives when a whole campus drops, and how fast the remaining sites can continue or resync. → the checkpointing math (interval, bandwidth, and the storage/network trade behind multi-region durability) is worked in Chapter 9.4; the goodput-vs-availability reframing is in Chapter 12.2.

The chapter resolves to an ordered sequence of forks, each constraining the next, now stretched across the WAN.

• Does one job have to span sites? Portfolio capacity, replication, failover and independent jobs split across sites without any synchronous-training penalty. Size their DCI on data movement, durability and user latency; the algorithm fork below does not apply to them. A single job crosses when one campus cannot hold it, or when the operator designs one cluster across sites, as Microsoft does with Fairwater and Meta with Prometheus. Either way the job pays the exposed communication computed above, which stays small when placement and survivor capacity are right. → Chapter 16.1.
• What radius can you energize? Intra-metro, metro or regional, set by which substations you can interconnect, not by preference. The radius picks your latency floor at ~5 µs/km. → Chapter 3.2.
• Which DCI optics and transport? Choose a specific grey client PMD, coherent line mode and protection service whose host, optical and recovery limits work on the route. → Chapter 8.9, Chapter 8.10.
• Synchronous or relaxed synchrony? Go synchronous only if surviving payload and dependent rounds leave the all-reduce inside the step budget. Otherwise buy capacity, or change the exchange schedule and check model quality. This is the model-quality-versus-bandwidth fork. → algorithms above.
• How do fault domains and checkpoints span sites? Route-diverse fiber, site-aware checkpoint placement, blast radius measured per site. → Chapter 9.4, Chapter 12.2.

Walk the sequence and a multi-campus run becomes a chain of well-posed decisions, each with a known downstream cost. Skip the first fork (spanning one job across sites when only your capacity needed to move) and you pay exposed communication and a site-wide failure radius for nothing. Skip the protected-route calculation and you can light expensive fiber and watch both campuses idle on an all-reduce that the surviving bandwidth and round-trip delay cannot finish before its deadline.