Fiber, Latency & Network Connectivity (Secondary Screen)

Network connectivity sits oddly in the 2026 siting hierarchy. For a frontier training campus, power and land come first (→ Chapter 3.1). A training campus can sit far from end users, yet its fiber still faces hard pass/fail budgets for communication, data movement and route failure. Online inference has to be near its users only when a regional path misses the client SLO you measured. Batch inference is judged on throughput and completion time. Check each site against what its workload needs from the network before you compare power costs (→ Chapter 1.1). Set the weight of the fiber screen from the workload, then diligence to a standard that catches failure modes before they strand a site.

Everything downstream follows from the delay of light in glass, which splits into three forks: can two campuses train as one machine (the distributed-training latency wall), can a metro meet the workload’s percentile and network-RTT budget (inference proximity), and how do you procure the path (lit vs dark vs new build, plus the middle-mile, make-ready and permitting tail that decides whether a power-perfect site is reachable at all). One lesson kept recurring in 2026: fiber alone can knock out a site when its physically diverse routes, or the dates they arrive, miss the service budget.

Light in a vacuum covers a kilometer in 3.34 microseconds. In single-mode fiber the glass core has a refractive index near 1.47, so the signal travels at roughly two-thirds of c. Each kilometer therefore costs ~4.9 microseconds of one-way delay, which this guide rounds to 5 us/km, or ~10 us there and back (guide derivation; M2 Optics, 9 January 2012). Better optics, a faster switch ASIC or a fatter pipe cannot buy it down. The speed of light and the index of the glass set it, and it is the floor under every latency budget in this chapter. The floor holds for standard solid-core single-mode fiber. Hollow-core fiber guides light through air at roughly 3.4 us/km, and Microsoft has run it on some Azure routes since 2023, but it is a route-specific build, not something to assume on leased fiber.

Three consequences follow. First, geography is latency: 1,000 km of fiber adds ~10 ms round-trip before a single packet is switched, queued or serialized. Second, propagation follows the fiber's actual route, so the great-circle distance does not set it. Work out latency from the documented lengths of the working and protection fibers, with rights-of-way detours and cable slack counted in. Third, each workload pays the floor differently. A synchronous training collective crossing 100 km of fiber incurs ~1 ms propagation-only RTT on that route, and you charge the critical-path part of it to step time. An inference request crossing 1,500 km of haul incurs ~15 ms propagation-only RTT before switching, queuing and model execution, and you charge that against the serving budget. Every fork below is a way of spending against the 5 us/km floor.

In 2025–2026 the largest training runs outgrew a single campus's power limits. When you cannot energize a gigawatt under one roof, you split the run across campuses and stitch them together with fiber, which turns the 5 us/km floor into a hard design constraint. Google trained Gemini Ultra synchronously across several data centers, running model parallelism inside SuperPods and data parallelism between them (Gemini technical report, December 2023). In November 2025 Microsoft described an AI WAN linking its Fairwater sites in Atlanta and Wisconsin. The Atlanta site had begun operating that October, and Microsoft said its first Wisconsin facility was fully operational on 23 June 2026. Every microsecond of inter-site delay on the critical path of a synchronous gradient exchange is goodput you do not get back.

The fork is how tightly coupled the cross-site training is, and it sorts into three regimes by tolerance for inter-site delay and bandwidth. Synchronous data-parallel across sites exchanges gradients every step and is the most demanding. Its viable distance depends on step time, collective bytes, topology, available bandwidth, added RTT and measured goodput, not on a universal kilometer limit. Hierarchical or async SGD (DiLoCo and its successors) synchronizes every tens or hundreds of steps, which cuts exchange frequency. Test the resulting bytes, convergence and goodput on your workload before choosing a multi-hundred-kilometer or cross-region split (Google DeepMind DiLoCo; Epoch AI, 2025). Fully decentralized over the public internet is the loosest regime, trading convergence efficiency for the ability to pool stranded capacity anywhere. The engineering of these methods lives in Chapter 8.8; the siting consequence is what matters here: the training method you choose sets the maximum distance between your halls, and the distance you can secure between candidate sites bounds the methods you can run.

The distances in the table are a budget you spend against. Suppose your power strategy forces a metro power-pool: four campuses inside an 80 km radius because that is where the megawatts are. You then have to test capacity, latency and route-failure behavior between the sites, over managed wavelengths or dark fiber lit with your own line equipment. Now suppose the only stranded gigawatts are 400 km apart. That is roughly 4 ms of propagation-only RTT at the 5 us/km above, before route detours and equipment. The cross-region case bites hardest: beyond an illustrative 500 km route the floor exceeds 5 ms RTT.

Frontier synchronous runs have been reported spanning more than one data center, so distance by itself does not rule out synchronous training. Distance sets the exposure; the budget settles the method. Collective payload, available bandwidth, step compute time, overlap and the parallelism boundary decide whether a synchronous run survives the link or you take an asynchronous method and its convergence penalty. If the measured budget fails, price gradient compression or intermittent synchronization, and their convergence cost, in GPU-hours. Fiber and power are not separate screens you score independently: they constrain each other, and the campus topology that results is the joint solution.

Bandwidth sets how many fiber pairs you buy. Between two sites, plain data-parallel training with bf16 gradients moves about 2 bytes per parameter each way per synchronous step, so a 1-trillion-parameter model exchanges about 2 TB (16 Tb) per step. If the exchange may take 2 seconds of the step, the link needs about 8 Tb/s of payload each way. A full 800ZR line system, whose reach is one amplified span of 80–120 km, carries 32 waves of 800 Gb/s on one fiber pair, about 25.6 Tb/s each way (OIF 800ZR implementation agreement, October 2024), so this example fits on one lit pair. To survive a cut, each diverse route must carry the full 8 Tb/s alone, so you buy the capacity twice. Make the model ten times larger, or the exchange window ten times shorter, and each route needs about 80 Tb/s: four pairs at that ceiling. Hierarchical reduction, sharding and DiLoCo-class methods change how many bytes actually cross the boundary, so take the byte count for your parallelism split from Chapter 8.8. A route longer than one span needs ZR+ or performance-class modems, and their payload per pair depends on the modem and the reach. Adding the L-band to the line system raises the capacity of each pair. Ask your line-system vendor for the per-pair figure at your route length, and see Chapter 8.10.

For latency-sensitive online or edge inference the user is waiting, but the SLO is workload-specific. Use client-observed TTFT, which already includes network, queuing and prefill, plus token-generation time; when you split those components, count network RTT once. Record the use case, percentile, TTFT, TPOT/inter-token latency, output length, concurrency, replica-routing model and network-RTT budget before translating the requirements into a site radius. Apply direct-manipulation and control targets only to the edge or control cases that set them.

Put latency-sensitive inference wherever the measured service path meets its SLO (→ Chapter 1.3, Chapter 1.5). If the regional path passes, a regional site stays in the running. Metro or edge placement is worth its extra cost when that path fails. Metro power typically costs ~1.5–2x the cheapest US industrial rates (EIA, July 2026; 2.5x or more only in California- or New Jersey-type markets), which is part of that price. The fork inside the fork is centralized regional vs. distributed metro/edge. A few large regional sites minimize cost and operational sprawl but put a floor under tail latency for distant users. Adding metro/edge nodes in a denser mesh can cut network delay, but every service path still has to pass its SLO. Each added site also multiplies the operating overhead, and the mesh strands capacity at low utilization. Most operators run a hybrid, with large regional cores carrying most traffic and a thin edge tier serving the latency-critical slice. The SLO draws the line between them.

Once the workload has set the screen's weight and the topology has set the distances, the procurement question is concrete: how do you acquire the path? The fork has three positions, trading time-to-light against control and unit cost. Lit services (managed wavelengths or Ethernet circuits from a carrier) are the least capital-intensive, but you inherit the carrier's routing, latency and capacity ceiling, and you pay a recurring premium for capacity you do not control. Dark fiber, leased or held under an indefeasible right of use (IRU), lets you pick the terminal equipment and decide how much capacity to light on the strands you contract for. The route contract still sets the physical routing, the access rights and who restores a cut. Lit wavelengths can do the job too, if the contract specifies the route, diversity and latency budget. Hyperscalers and infrastructure investors have moved toward owning or co-investing in dark assets for AI-grade links (Landgate; DCConnect, 2025). New build, your own conduit in the ground, gives total control and a future-proof strand count. It costs the longest lead time and the full weight of permitting and make-ready.

Time-to-light depends on how ready the contracted route and both endpoints are. Zayo's wavelength service description (v1.4, 15 February 2024 revision) is a useful benchmark. Counted from order acceptance, and varying with the protection option, its standard intervals are 30–45 calendar days to an on-net PoP or data center that is already equipped, 90 days for off-net service through a third party, and 120–150 days where Zayo must build off-net and the build is approved. Pre-provisioned capacity is a separate, route-dependent offering. Dark fiber lights up once strands are free, splicing is finished, and equipment and access rights are in place. A new alignment brings its own civil and permitting schedule. Put the carrier's order-specific delivery date and any unresolved construction dependencies on the master schedule, because the capacity ramp the fiber must serve arrives on a fixed date. For an AI campus the right answer is frequently a blend: lit capacity for day-one connectivity and management/out-of-band traffic, dark fiber on the diverse routes that carry the heavy east-west and DCI load, and new build reserved for the one critical gap no existing route fills. The structured-cabling and DWDM/OTN engineering that lights these paths is in Chapter 8.10; this chapter's job is to get the right strands to the right place on the right schedule.

Whatever the procurement choice, the diligence standard is the same, and it is stricter in 2026 than most underwriting models assume. A site without at least two physically diverse paths, each with enough strands, is increasingly treated as functionally stranded, however good the power and land. Size each path’s usable wavelengths, strands and spare capacity against the surviving service you need, with documented physical route separation. The reason is that a single path is a single point of failure: one backhoe, one rail-corridor fire or one severed conduit, and a campus that cost billions and waited years for power goes dark.

The trap that catches careful teams is the diversity illusion: two circuits sold as diverse that share a conduit, a bridge crossing, a building entrance or a piece of make-ready for part of their run. On a map they look redundant; in the ground they fail together. Real diligence requires the carrier to produce the physical route maps and prove a minimum separation distance along the entire path, including the last hundred meters into the building (diverse entrance facilities on opposite sides of the structure). Conduit ownership is the second buried fact. Ownership, route rights and restoration terms tell you who controls the civil plant. The service contract and the terminal equipment decide which capacity and latency choices are yours. Hyperscalers own or co-invest in conduit to control that plant directly. The diligence checklist is short but unforgiving: how many physically diverse paths, who owns each conduit, what is the documented separation, how many strands are dark and available for growth, and where are the shared-risk segments that couple the 'diverse' routes.

Buy diversity in the delivered service. Require each carrier to identify the contracted route and rights, shared ducts and bridge crossings, building entrances, powered line equipment, restoration responsibilities and delivery date. Trace the shared-risk segments across both offers: two logos and two map lines do not prove separation. Then get measured end-to-end latency and usable capacity for the specified surviving route, including protection and failover behavior and competing reservations. If a shared segment defeats the required failure case, reject that pair or redesign it. If the carrier cannot supply the proof, hold site approval. 8.8 tests the DCI service, and 3.13 carries its pass, fail or unknown state into the site memo.

Beyond raw diverse fiber, three network anchors sharpen the screen, and the workload sets their weight. Internet exchange points (IXPs) and carrier-neutral facilities are where networks peer. Connecting at an IXP can shorten the path to enterprise and access networks, provided the peers you need are present and the interconnection agreements and routing policies are in place. Being near the exchange does not give you that path. That matters for an inference fleet serving the public and very little for a training campus that talks mostly to its own peer campuses over private fiber. Cable landing stations, where subsea cables make landfall, are the gateways for intercontinental traffic. A connection at a landing station can shorten an international service path, if the contracted cable and its terrestrial backhaul reach the destinations you need. Test that route's latency and capacity against what the workload needs; a coastal address tells you nothing about its performance. Nordic and Iberian campuses lean on subsea landings this way (→ Chapter 3.13). Long-haul backbone access can shorten the access route you have to build, if an interconnection point, the rights you need, enough capacity and an independent second route all exist. A backbone running past the parcel proves none of them.

The decision is which anchor you optimize for. Optimize for IXP proximity and you bias toward established metros, where power is expensive and constrained. Optimize for backbone access and stranded power and you bias toward exurban sites on a fiber corridor, the sweet spot for a training campus. Optimize for landing-station connectivity and you bias toward coastal and subsea-rich geographies, accepting their power and land realities. Weight the three anchors by how much each adds to the service you need, and test each candidate on documented paths, access terms and delivery dates.

Pulling the forks together, the connectivity screen is a short ordered sequence, and the order matters because each step gates the next. (1) Set the weight from the workload before you score any site. Training is judged on communication and data movement, batch inference on completion time, and online or edge inference on client latency. (2) Derive the latency and bandwidth budget from the topology: the inter-campus distance the training method can absorb and the surviving Tb/s each route must carry, or the metro radius the inference SLO allows. (3) Diligence the diverse paths early, in parallel with power and land, demanding physical route maps, conduit ownership, separation distance and strand counts, and hunting the diversity illusion. (4) Pick the procurement fork (lit, dark or new build) against the time-to-light the schedule can tolerate, treating any middle-mile access build as a long pole. (5) Score the network anchors (backbone, IXP, landing station) by their contribution to the required service.

Done in that order, fiber rarely surprises a project. Done out of order, by scoring anchors before setting the weight or running fiber diligence only after a site has cleared everything else, it strands a power-perfect campus or ships an inference fleet that misses its SLO. The screen is cheap to run early and very expensive to discover late.