Physical-Layer & Interconnect Taxonomy

The fabric chapters before this one assume bits arrive intact at the far end of every link: scale-up (8.2), scale-out protocols and topology (8.4, 8.5), and the switch and NIC silicon (8.3). This chapter covers the wire and glass that make that true, and one physical decision that runs through all of them: for a given reach at a given lane rate, what is the cheapest, lowest-power medium that still meets the link budget? There are three candidates: a passive copper cable, an electrically boosted copper cable, or an optical link. Signal-integrity physics sets the boundary between them, not taste, and every step up the SerDes ladder pulls it inward, toward optics and a bigger bill.

In practice the answer is a portfolio. An AI hall can start from "copper inside, optics outside": keep short in-rack links electrical where the channel tests clean, and spend optics where reach, routing or density justify them. Where you place that seam is the most consequential physical-layer fork in the building. Push it too far toward copper and you stretch DACs past their margin and eat link flaps. Push it too far toward optics and you burn megawatts on transceiver DSPs and pay for pluggables on spans copper could have carried for free. The rest of the chapter walks the link budget, the SerDes ladder, FEC and TDECQ margin discipline, the medium taxonomy, form factors and DSP placement. It then treats optics as what they are at scale: a recurring failure and operating-cost exposure whose component events and repairs can interrupt useful work.

A link is an accounting problem. The transmitter launches a signal with some power and quality. The channel (copper trace, cable, connectors, or fiber and its splices) subtracts loss and adds distortion. The receiver needs the signal to arrive above a threshold with enough eye opening that the forward-error-correction engine can clean up what remains and hit the target bit-error rate. The link budget is that tally: launch power and equalization on one side, insertion loss and impairments in the middle, receiver sensitivity and FEC coding gain on the other. Every entry in the interconnect taxonomy balances the same tally over a different distance.

Two physical facts dominate that tally in 2026. First, modern datacom links use PAM4, not the older NRZ. PAM4 encodes two bits per symbol with four amplitude levels, doubling throughput at a given symbol rate, but it cuts the vertical eye opening to roughly a third of NRZ's. The same channel loss therefore costs far more margin, and the link leans hard on equalization and FEC. Second, copper loss rises steeply with frequency. For fixed modulation and coding, doubling the per-lane service rate (50G to 100G to 200G to 400G) doubles the symbol rate and the Nyquist frequency. Return loss, crosstalk and package or connector impairments also consume receiver equalization budget. That pressure drives the migration from copper to optics, but it is not a universal reach-halving law: trace geometry, cable gauge, host capability and interface limits decide which span works.

The SerDes (serializer/deserializer) is one interface in the complete link: switch or NIC package → host board → module electrical attachment → optical PMD → remote endpoint. An 800G service can use eight 100G or four 200G electrical lanes, and 1.6T can use eight 200G lanes. A gearbox can change the lane count before the optical PMD, so optical lanes and fibers need not match the host. Record the exact port mode, coded electrical rate, symbol rate and lane/FEC mapping, and use Chapter 8.10 for the resulting fiber and polarity schedule. The P802.3dj status tile below separates draft interface work from available products. A module expecting a 112G-class electrical attachment cannot simply accept a port forced into a 224G-class mode. Backward operation needs an explicitly supported mode: a newer ASIC does not make arbitrary modules interoperable.

What keeps a PAM4 link alive across the budget is forward error correction. Datacom uses named Reed-Solomon FEC profiles, including RS(544,514) KP4, but no pre-FEC input maps universally to one post-FEC result. State the Ethernet generation, lane rate, code/FEC mode, measurement interface and counter semantics. The P802.3dj MAC/PLS service-interface objective is not a portable installed-link threshold, and older Ethernet or InfiniBand profiles must be evaluated against their own normative and OEM limits. FEC is not free: it adds latency and a small bandwidth tax, and it has a cliff. Below its correction threshold the link is pristine; cross the threshold and errors leak through catastrophically. That makes margin the real engineering quantity.

For optical transmitters, TDECQ (Transmitter Dispersion Eye Closure Quaternary) quantifies a transmitter penalty under a specified reference receiver and dispersion test. It is not a field receiver margin reading. A transmitter near its permitted TDECQ penalty leaves less tolerance for connector contamination, temperature drift and aging elsewhere in the system, and those impairments can push the assembled link into errors or flaps. Installed margin also depends on the passive channel and the remote receiver, so keep transmitter conformance results separate from the field error record. TDECQ discipline at acceptance test is therefore a goodput decision, not only a metrology one.

Accepting a channel takes three separate records. The supplier provides transmitter and receiver conformance, including the prescribed TDECQ test and operating range. The installer provides every fiber's continuity, insertion loss, reflectance and end-face record, taken with the specified reference method and its uncertainty. The system test provides loaded per-lane pre-FEC statistics, corrected and uncorrectable codewords, frame loss, link resets and recovery at temperature. A clean post-FEC counter does not show reserve margin, and optical receive-power telemetry is neither calibrated channel certification nor TDECQ.

Worked decision A: 800G DR8 budget and loaded acceptance

Use the passive channel boundary from module interface to remote module interface. Fiber loss is 0.100 km × 0.40 dB/km = 0.040 dB. Connector loss is 4 × 0.50 dB = 2.0 dB. Total passive loss is 2.040 dB. Adding the 0.40 dB reserve and 0.20 dB uncertainty gives a required budget of 2.640 dB, about 2.6 dB. Against the 3.0 dB application limit, the unrounded remaining allowance is 0.360 dB, about 0.4 dB: the planned channel passes.

One extra mated pair adds 0.50 dB and makes the reserved total 3.140 dB, about 3.1 dB: fails. Keep four pairs, or buy and test a lower-loss arrangement; otherwise a fifth pair's serviceability benefit spends the reserve. The calculation does not credit FEC gain a second time against a PMD channel limit.

The arithmetic is a plan, not an acceptance. Certify every strand in both directions at the specified wavelength, identify the worst path, and confirm continuity, polarity and reflectance before connecting equipment. Then run the loaded test at the thermal condition of a populated host, recording the exact pre-FEC limit and lane counter semantics alongside corrected and uncorrectable errors and resets. Accept the channel only when both the field loss check and the loaded test pass. The DR8 application overview supplies the loss basis, Chapter 8.10 supplies the plant map, and Chapter 13.7 runs the loaded test. This chapter is the optical-budget home reached from Chapter 4.11.

A faster application on the same fiber starts from the installed loss record, not from the old application's acceptance. Keep the current DR8 plant record unchanged and file the new endpoints' PMD test results beside it.

With the budget and the ladder in hand, the taxonomy is a reach ladder. As distance grows you climb from passive copper to electrically assisted copper to optics, paying more power and money at each rung for more reach. The principle behind copper inside, optics outside is to keep short, dense, latency-critical scale-up links electrical when the channel and service access allow it, then spend optics where reach, routing or density improves the whole system. The rack boundary helps lay out routes; it is not a physical cutoff.

Passive DAC (direct-attach copper) is a plain shielded twinax cable with no active electronics: zero added module power, lowest cost, lowest latency and no active parts to fail. It also has the shortest reach, about 1–2 m at 800G-class rates and shorter again on 224G/lane channels (SemiAnalysis, January 2026). ACC/AEC (active copper / active electrical cable) adds a small redriver or retimer IC in the connector to boost and re-clean the signal, buying a few more metres. Credo's 800G ZeroFlap AECs run 1–7 m, but only its OSFP breakout assemblies reach 7 m; straight OSFP-to-OSFP cables are listed at 1–3 m (Credo product page, 2026). The cost is retimer power, firmware and interoperation constraints, and another population of components that can fail. These are 2025–26 reference points, so test your host and cable together at your lane rate.

AOC (active optical cable) is a captive, factory-terminated optical assembly: optics on both ends, fiber in between, sold as one fixed-length cable. It reaches tens to hundreds of metres, with end count and length fixed by the assembly, and breakout requires a purpose-built, supported cable. Pluggable transceivers are the fully modular option: a transceiver in a cage on each end, with field-installed structured fiber between them. Pluggables are the most flexible rung and remain the basis of almost every large scale-out fabric. Even where switches moved to co-packaged optics in 2026, the server end still uses pluggables (Chapter 8.10). At ~15–17 W per 800G DSP module, they are also the most power-hungry and failure-prone rung on the ladder.

Once you choose pluggable optics, the form factor is a long-lived commitment because the cage and connector belong to the host. QSFP-DD and OSFP provide eight electrical lanes; OSFP-XD provides sixteen, as defined by the QSFP-DD hardware specification and OSFP/OSFP-XD MSA. Those lanes, cage dimensions and thermal surfaces constrain the switch you live with, but they do not define the optical PMD. Before ordering, check module width, electrical attachment, power class, flat or finned heat-removal arrangement and every supported backward mode. A 1.6T label alone cannot establish lane count or compatibility, and a module that fits mechanically can still exceed the host's cooling budget. Package and FRU consequences belong to Chapter 8.10.

Watch the next cage. In October 2026 NVIDIA and Cisco proposed USFP, a form factor for 400 Gb/s electrical lanes. The first planned modules carry 16 full-duplex 400G lanes, 6.4 Tb/s, in an OSFP-sized footprint; the connector is designed to grow to 32 lanes and 12.8 Tb/s. A module can dissipate up to 100 W, and TE Connectivity and Amphenol are building cages, connectors and modules. USFP is still a planned multi-source effort: nobody has published a specification or a ship date. The 4x density and 1.3x efficiency gains are vendor figures, measured against 200G/lane pluggables. Arista’s XPO multi-source agreement, announced 12 March 2026 with no specification or ship date, takes the liquid-cooled route instead. Its 64-channel module carries 12.8 Tb/s (Arista does not say whether that counts one direction or both) and has an integrated cold plate rated to 400 W. Arista says that fits 204.8 Tb/s per OCP rack unit. If your hall's switches turn over after 2028, find out now which 400G/lane cage your switch vendor plans to adopt. Then size faceplate airflow for 100 W-class modules, or plan a liquid-cooled alternative such as XPO.

The thermal lid is a second fork. Finned-top OSFP modules carry their own heatsink and rely on the switch's airflow, which is standard in air-cooled halls. Flat-top OSFP modules omit the integral fin because the host supplies the heat-removal surface: a riding heat sink pressed onto the module, which can itself be air-cooled or liquid-cooled. NVIDIA's air-cooled DGX H100/H200 links run flat-top modules. Finned and flat-top are not freely interchangeable in a given cage, and what decides the fork is the host SKU's cage geometry and the thermal interface its vendor supports, not the hall's cooling architecture. Get the host-SKU and module-SKU compatibility matrix (cage, heat-sink interface, power class, and the airflow or coolant conditions each is rated for) before you order optics. Discovering a mismatch later means a faceplate redesign, not a swap.

One last mechanical point follows from the packaging: high-lane-count modules multiplex links, and the links in one module share its fate. A switch-side twin-port module serves two distinct endpoints from one field-replaceable unit. In a GB200 or HGX-class fabric at 400G per GPU that is an 800G module driving 2×400G. NVIDIA's twin-port OSFP modules explicitly feed up to two adapters, and four of them reach all eight GPUs of a SuperPOD node (NVIDIA, 2025). At GB300's 800G per GPU on Quantum-X800 InfiniBand the switch-side part doubles: the switch carries its 144 800G ports in 72 OSFP cages, so each 1.6T module serves two 800G endpoints. Spectrum-X Ethernet runs the same ConnectX-8 as two 400G planes, which keeps 2×400G modules in service on that fabric. Removing, swapping or wholly losing one module therefore interrupts both endpoints' links at once (a single-engine fault may take only one). Draw the breakout map so fate-shared pairs never back each other up, never putting both planes of one accelerator through one module, and size spares by module count, not link count.

A conventional pluggable transceiver contains a power-hungry DSP that re-times and re-equalizes the signal on both the host (electrical) and line (optical) sides. That retiming costs power and money, which is why linear and partially retimed architectures remove some or all of it. The saving moves channel responsibility toward the host, so count the host's equalization power and test burden before claiming a lower total bill. The result is a three-way fork that trades optics power against host-board co-design burden and interoperability risk.

Fully retimed pluggable is the incumbent: a full DSP retimes both sides and isolates the optic from the host channel. It is the most robust, the most plug-and-play and the most multi-vendor interoperable, and also the most power-hungry, at roughly 15–17 W for an 800G module (SemiAnalysis, January 2026). LRO (linear receive optics, also sold as RTLR or half-retimed; the names describe the same architecture, so treat them as aliases, not separate purchases) retimes the transmit side and runs the receive path linear. It recovers the receive-side retiming power but leaves the channel equalization and testing burden with the host. LPO (linear pluggable optics) removes the DSP entirely and relies on the host ASIC's own SerDes equalization to drive the optic directly. It cuts 800G module power to about 8.5 W on a shipping DR8 part (FS, September 2025), close to halving it, but demands host-board co-design and end-to-end qualification because the optic no longer cleans up the host channel.

Price and power belong to a specific module at a specific rate and reach, not to the category label. The closer you get to LPO, the more module retiming you remove and the more electrical margin the host must own. That can save power and cost, but compare equal delivered bandwidth, reach and temperature, including host power. An alternate vendor's module still has to pass the complete channel before it becomes a substitution option.

At 1.6T every option in the fork moves up a power class. A GB300-class fabric feels that first on the switch side: with 800G per GPU, a Quantum-X800 switch fills each of its 1.6T cages with one twin-port module serving two ports. FS builds its 1.6T DSP modules around a Broadcom 3 nm DSP; the DR8 is rated at 25 W maximum and the dual-duplex 2×FR4 at 26 W (December 2025). NVIDIA's own illustration puts a 1.6T pluggable at about 30 W, 20 W of DSP and 10 W of laser (August 2025). Dell'Oro expects 1.6T switches, sampling in 2Q 2026, to ramp in volume from 2H 2026 and faster than 800G did. Linear modules get harder at the same step: at 200G per lane the host channel has less margin to give, so 800G LPO results do not carry over. Ask the switch vendor which 1.6T LPO or LRO modules it supports on your host before counting the saving.

At cluster scale, optics are a population, and a component event rate is only the first line of the reliability model. Define a hard failure as loss that requires replacement, a flap as transient service loss, and a repair disturbance as an interruption caused while working on another component. Measure mean time to flap separately from hard-failure MTBF: a link that drops and re-establishes can stall a collective without ever needing replacement. Collect count, exposure in module-hours, temperature, firmware and lot identity separately. For an independent constant-rate model, expected events over time t are N × t / M, for population N and per-component mean interval M. The illustrative cadence below uses assumed rates; it is not observed fleet telemetry or a transferable job-failure rate.

Suppose one unstable link forces an illustrative 32,000-GPU job to roll back a full hour. That wastes 32,000 GPU-hours of progress, and the loss depends on that restart and the checkpoint age. Trace each event into the installed topology before pricing it. A failed optical end can leave a job running over a surviving plane at reduced bandwidth, while two nominally redundant links that share a module, laser, tray or power source can disappear together. Join the affected port and route identities to the admitted jobs, then separate rerouted service, timeout/retry, communicator failure and restart. Correlated temperature or lot faults and maintenance disturbances need their own event groups, not independent-rate multiplication. The MRC production account shows why path and transport recovery matter, but its rates do not describe an unmeasured fleet. Hand restart probability and affected jobs to Chapter 9.4 for checkpoint consequences, and useful-output loss to Chapter 14.1.

The physical layer is three boundaries, each derived from the layer above it and each costly if drawn wrong. First, the copper-to-optics boundary, set by your SerDes generation's reach physics. Keep scale-up and the shortest scale-out hops on passive copper, with no module power and no active electronics to fail. Cross to active copper only for the reach it still covers, and reserve optics for spans copper physically cannot carry. Draw it too far toward optics and you burn on the order of 20 kW/rack of needless DSP power on an NVL72-class design (NVIDIA's counterfactual estimate); draw it too far toward copper and you eat link flaps.

Second, the DSP-placement boundary: how much optics power you cut against how much host co-design and interoperability risk you absorb. Fully retimed is safe and power-hungry. LPO halves module power but makes you the owner of the whole electrical path. Third, the form-factor boundary, set by the host SKU's cage geometry and supported thermal interface: flat-top where the host supplies the riding heat sink, finned where the module carries its own. Only a faceplate redesign unwinds it.

Around all three sits the reliability discipline, because the physical layer is where goodput is silently won or lost. Two fabrics that look identical on a topology diagram can lose very different amounts of training work if their optics differ in field thermal margin, shared FRUs and repair access, so commission and operate the physical population, not just the logical links. The taxonomy tells you which medium to use; commissioning discipline tells you whether it will carry your bits for the life of the cluster.