Management, Out-of-Band Fabric & PTP/IEEE-1588 Timing

Every earlier chapter in Part 8 covered the fabrics that do the work: the scale-up domain that binds GPUs into one accelerator and the scale-out fabric that carries the all-reduce. This chapter covers the two fabrics that let you operate, observe and trust them. They carry no training traffic and earn no FLOPs, so they get scoped last, under-funded, and discovered missing during an outage.

The first is the out-of-band (OOB) management fabric, the independent network that reaches a server's baseboard management controller (BMC). Through it you power-cycle the node, re-image it, read its sensors and update its firmware, especially when the in-band data plane is dead. The second is the timing plane: PTP/IEEE-1588 or another chosen clock distribution that keeps each recorded event within its measured uncertainty. With it, a telemetry record from one switch, a RoCE drop counter from another and an NCCL stall on a third can sit on a single timeline and be correlated.

Four forks organize the chapter: physical separation vs logical segmentation, Redfish vs IPMI, PTP vs NTP, and boundary vs transparent clocks. Each has a cost if you choose wrong. Both fabrics are insurance you buy before the incident. Their value stays invisible until the night the controller is unreachable and you need to find which of 4,000 GPUs stalled the run. Then you discover you cannot reach the BMC and your logs are 200 ms out of alignment. By then it is too late to add either one.

The management fabric exists to answer one question reliably: can I reach this machine when everything else has failed? A GPU server has, in effect, two computers in it. One is the host: CPUs, GPUs and the in-band NICs that carry training traffic. The other is the BMC, a small always-on management processor with its own CPU, its own dedicated NIC and a standby power domain that stays alive when the host is off. That standby domain is lost if chassis power is removed. Through the BMC you read inlet temperatures and fan speeds, switch the chassis on and off, mount a virtual ISO to re-image, capture a serial console and push firmware.

If the path to the BMC shares fate with the production data plane, the failures you most need management for also cut you off from the fix. A wedged data-plane NIC, a black-holing leaf switch, a runaway in-band control agent and a misapplied ACL that severs the host all fall in that group. You are left with a remote-hands dispatch and a row walk, which at AI-cluster scale means hours of stranded GPUs.

That is the whole argument for keeping the management fabric out of band: it must not share fate with the thing it manages. In a well-built cluster the OOB fabric is its own physical network. It has its own management switches (typically modest 1/10/25 GbE top-of-rack switches, one per rack or per row), its own cabling on its own pathways, its own IP addressing and DNS, and its own uplink to a management aggregation tier that lands in the operations/orchestration zone rather than the tenant data plane. The BMC port, PDU management port, CDU and cooling-controller ports, switch management/console ports and GNSS/timing appliances all home onto this fabric. It is small, cheap and low-bandwidth next to the data plane, and when a row goes dark it is the difference between a console session and a dispatch.

The management fabric is only as useful as the protocol you speak over it, and there is a real generational fork. The legacy answer is IPMI, the Intelligent Platform Management Interface, a 1998-era protocol for power control, sensor reads and serial-over-LAN. It works and it is everywhere, but it is also a security and operability liability: a clunky binary protocol, a long history of CVEs, weak authentication and no clean model for the firmware-update and inventory operations a modern fleet needs.

The modern answer is Redfish, a DMTF standard that exposes the BMC as a RESTful, JSON, HTTPS-secured API with a structured resource model. Redfish makes a fleet programmable. You enumerate inventory, drive power and boot order, stream telemetry and orchestrate firmware updates through one API across vendors, instead of scripting a different IPMI dialect per OEM. Underneath, component firmware updates increasingly flow over PLDM-over-MCTP, the DMTF transport for talking to GPUs, NICs and other devices behind the BMC. OCP's GPU firmware-update specification is built on top of it, and a hardware root-of-trust decides which firmware is allowed to land.

The protocol is half the question; the firmware that serves it is the other half. Ask which BMC stack ships: the OEM's proprietary stack, or open-source OpenBMC, a project whose founders were Google, IBM, Intel, Microsoft and Facebook (now Meta). The answer decides who ships security fixes, how fast a published BMC vulnerability gets patched across your fleet, and how much of Redfish the BMC actually implements.

Standardizing on Redfish rather than IPMI pays off in three places that all bear on goodput and ramp velocity.

Bring-up velocity. Zero-touch provisioning (PXE/HTTP boot, image push and config) is far cleaner against a Redfish API, and bare-metal bring-up is an under-appreciated economic lever when you are racing a depreciation clock to first job.

Fleet firmware management. GPU, NIC and BMC firmware drift is a real source of silent performance regressions and RoCE pathologies. A Redfish + PLDM update path with secure OOB is how you keep a 50,000-accelerator fleet on a known-good firmware baseline. Switches are getting the same treatment. OSPA (Open Switch Platform Architecture) is a Meta, Microsoft and Alibaba contribution in OCP Foundation review as a revision 1.0.0 candidate (October 2026; Chapter 0.4). It runs sensor, fan, inventory and firmware management on the switch's host CPU and keeps the BMC for power, host bring-up, leak protection and troubleshooting. Each OSPA switch has two front-panel management ports, so decide whether your OOB network cables one or both.

Attack surface. The management plane is a high-value target because it can power, re-image and re-flash every machine. Moving to authenticated, TLS-protected, RoT-gated management is a security decision as much as an operational one. → firmware-update, root-of-trust and BMC security in Chapter 11.4; management-plane isolation in Chapter 11.7 and the threat model in Chapter 11.1.

Reachability through the outage depends on every service between the operator and the managed device. Trace the remote-access carrier, firewall, bastion, DNS, identity/MFA, credentials, management uplinks, console server and BMC standby supply, including their rack PDU and upstream power sources. Two management cables fed by one failed PDU are not two recovery paths. Two uplinks also leave a single-homed BMC dependent on its management leaf. Keep local recovery credentials and addressing available through the identity/DNS outage you plan for, with the audit trail and access restrictions defined in Chapter 11.7.

Commission this dependency graph by removing, one at a time, the production path, the external identity/DNS service and the management power source. For each fault, record which endpoints stay powered, which console and BMC operations succeed, the authentication path and the time to restore against the owner's recovery-time limit. Mark devices that lost power as unreachable; do not report the surviving switch as an OOB pass. Redfish gives you a management API and resource model, but it does not give you network or standby-power independence. Test results go to Chapter 13.7. Loss of bulk telemetry is a separate data-quality problem.

The second operational fabric distributes time. When an illustrative 4,000-GPU run stalls, diagnosis means reconstructing thousands of telemetry streams: a switch's egress-drop counter, a NIC's RoCE NACK, a CDU's flow dip, a GPU's thermal throttle, an NCCL collective timeout. To reconstruct what happened first, you place those events on one timeline. If clock-and-capture errors exceed the gap between events, cause and effect smear together. Software scheduling, delayed logging and packet-delay variation can use up the error budget before clock distribution is even considered. Billing logs can tolerate a different bound from a hardware capture of a congestion microburst, so choose the protocol and timestamp path from the event requirement.

What separates useful implementations is where timestamps are taken and how the clock reaches the consumer. PTP can timestamp at the NIC or switch wire interface, which avoids OS scheduling and interrupt jitter at that capture point. The NIC's PHC (PTP Hardware Clock) is disciplined to the grandmaster, but the application often stamps CLOCK_REALTIME after additional delay. LinuxPTP phc2sys makes that PHC-to-system transfer explicit. chrony also supports hardware timestamping, so the real choice is between measured deployments, not a permanent millisecond-versus-nanosecond split by protocol name.

The published hardware figures each describe their own measurement boundary. NVIDIA measured about 10 ns of PTP accuracy across a Spectrum-3 switch acting as a boundary clock, and under 4 ns between hardware units inside the Spectrum ASIC; Meta reports sub-microsecond hardware-timestamped PTP and SPTP across its fleet. None of them bounds an application log written later. The OCP Time Appliances Project's draft white paper (v0.9.1, 1 October 2026) marks the other limit. A shared time base improves cross-host observability, one-way latency attribution and time-triggered functions, but the correctness of collectives, checkpoints and causal traces normally comes from protocol ordering, logical progress, versioned state and trace relationships. Choose the least complex clock service that meets the application's full event-error budget under load and failure.

Distributing time across a multi-tier Clos fabric takes more than a grandmaster broadcasting the time. Every switch hop adds and varies delay, and PTP has two architectural answers for compensating it.

A boundary clock (BC) terminates PTP at each switch. The switch is a slave to its upstream parent and a master to everything downstream, recovering and regenerating time at every tier. That isolates each segment from the packet-delay variation above it, and it can carry timing through a deep Clos when each tier's measured error fits the accumulated end-to-end budget. The catch is that every switch in the timing path must be a capable, configured BC.

A transparent clock (TC) does the opposite. The switch does not recover time. It measures how long each PTP packet dwelt inside it and writes that residence time into the message's 64-bit correction field, so the end clock can subtract the accumulated switch delay. The correction field resolves to sub-nanosecond, and per-hop accuracy depends on the switch's timestamping hardware. But the end node must trust and sum the corrections across the whole path, and forward/reverse path asymmetry remains the dominant residual error.

So use boundary clocks for the scaled, multi-tier production fabric, because they bound error per tier and do not ask the endpoint to reason about the whole path. Use transparent clocks where the topology is shallow or you want maximum per-hop fidelity with minimal switch state. NVIDIA Spectrum and Broadcom Tomahawk/Jericho switches are concrete timestamping candidates (Intel stopped Tofino development in 2023). Whichever you pick, confirm boundary/transparent-clock mode, NIC, firmware, PTP profile and one-step/two-step support on the system you actually buy; ASIC capability alone does not prove the path works. The expensive mistake is mixing modes inconsistently across tiers, or running a tier of non-PTP-aware switches in the timing path. Every such switch injects uncompensated, variable delay that no grandmaster precision can recover.

Pin the PTP profile first, because it fixes the transport and how much every switch must do. The IEEE 1588 default profile runs over Ethernet or UDP/IP and suits a single-operator fabric with boundary clocks at each tier. ITU-T G.8275.1 assumes full on-path support: Layer 2 multicast, a boundary or transparent clock at every hop, and SyncE carrying frequency over the physical layer. Juniper, for one, requires SyncE and hybrid mode whenever this profile is configured. The profile gives the tightest bound and asks the most of each switch. G.8275.2 runs unicast over IPv4 or IPv6 and tolerates hops that do not process PTP, at the price of the uncompensated asymmetry those hops add. The IETF Enterprise Profile (RFC 9760, May 2025) mixes multicast and unicast delay messages on enterprise IP networks, and Meta's unicast SPTP (2024) cut the load on its time appliances. SyncE helps under the other profiles too: a boundary clock that stays frequency-locked through a burst of packet-delay variation recovers time more steadily. Ask the switch vendor which profiles, transports and SyncE options the exact model and software release support.

The timing tree has a root, and the root has to get time from somewhere. One widely used source is GNSS (GPS and its peers). A rooftop antenna and a coax or fiber down-lead feed a grandmaster appliance with a one-pulse-per-second (PPS) signal plus time-of-day, which disciplines the grandmaster's internal oscillator. On Linux that chain is concrete, and it is what you actually operate. ts2phc steers a NIC's PHC from the external PPS/GNSS timestamps. ptp4l implements IEEE-1588 to distribute and recover that time across the fabric. phc2sys slaves the host's system clock (CLOCK_REALTIME) to the disciplined PHC. The Best Master Clock Algorithm (BMCA) elects the active grandmaster and fails over to a standby if the primary degrades, which is why you deploy at least two grandmasters, ideally fed by independent antennas on independent pathways.

What separates a robust timing plane from a fragile one is holdover: what happens when the GNSS reference is lost to a jammed or spoofed signal, a failed antenna, a cut down-lead or a bad-weather fade. The grandmaster then coasts on its internal oscillator. How long it can coast before time-of-day drifts past your correlation tolerance depends on the holdover stability you paid for, including initialization, temperature and aging. TCXO, OCXO, rubidium and CSAC trade cost against stability, and the spread is three orders of magnitude or more. Meinberg's published oscillator table (read October 2026) gives the phase drift after 24 hours of holdover at constant temperature. Its figures assume at least 30 days of continuous operation, the final day locked to GNSS, which a newly installed grandmaster cannot have. On that basis rubidium drifts about ±800 ns, Meinberg's best OCXO (the DHQ) ±4.5 µs, its OCXO-SQ ±65 µs and a TCXO ±4.3 ms. Temperature widens it: across 5–70 °C the table allows the OCXO-HQ up to ±1 × 10⁻⁸ of extra frequency error. So knowing the oscillator class is not enough to predict the coast time: the worked example below shows a 1 × 10⁻⁹ oscillator holding a 2 µs budget for ten minutes and a 1 × 10⁻⁸ one failing it, and the same OCXO can be either, depending on the room. Buy against the vendor's holdover specification at your temperature profile, then prove it with an outage test.

This is a money decision as well as an engineering one. You size the holdover oscillator against the worst GNSS outage you are willing to ride through without degrading your telemetry timeline, and without dependent systems (security event correlation, multi-DC ordering) losing trust in the clock. GNSS is also a security surface: it is jammable and spoofable from outside the fence line, so the holdover budget doubles as your defense against a denial-of-time attack.

A false grandmaster is a different threat, and the defense starts with port roles. Put switch ports that face hosts or lower tiers in server-only (master-only) mode, and run every host client-only. Then no endpoint can win BMCA, even if a NIC is misconfigured or a host is hostile. Where the grandmaster and switches support it, also restrict which grandmasters are accepted, using an acceptable-master table, a unicast master table or a clock-class threshold. Finally, raise an alarm whenever the grandmaster's identity changes or the offset steps. Management-plane isolation, which keeps timing traffic off reachable networks, is in Chapter 11.7.

Authenticate the time the management plane consumes as well. NTS (RFC 8915) signs NTP packets so BMCs, bastions and log collectors can verify their source, and Meta enabled it on its public time service in October 2026 and released the implementation. Authentication does not stop delay attacks: an on-path attacker can still shift a clock by up to half the added delay. An authenticated server can also still be wrong, so keep several independent sources with quorum and sanity checks. Plan the bootstrap too. NTS key establishment runs over TLS, which needs a roughly correct clock, so a device with a dead RTC needs a trusted local first source on the OOB network.

A timing plane that is designed but not measured is one you trust on faith, and faith fails at the worst moment. Make time-sync accuracy an explicit commissioning acceptance gate. Before a hall is declared ready for training load, it must demonstrate by measurement that every node's clock sits within a defined offset bound of the grandmaster. It must show the bound holds under fabric load, since PTP accuracy must not degrade when the data plane is saturated. And it must show that grandmaster failover and a defined holdover window behave as specified. The measured artifacts are offset-from-master distributions across the fleet, holdover drift over a simulated GNSS outage, and BMCA failover time. They turn "we configured PTP" into "the clock is trustworthy."

The same logic applies to the management fabric. Commissioning should prove that every BMC, PDU, switch console and cooling controller is reachable over the OOB path with the production data plane deliberately severed. An OOB fabric only ever tested with the data plane up has not exercised its one job. Put both acceptance criteria in the network-fabric commissioning plan, next to the RoCE and bisection-bandwidth validation, because they are the operability and observability preconditions for trusting every other test result. → fabric commissioning and validation in Chapter 13.7.

The holdover result bounds event timestamps only while the assumed capture path is intact. Preserve that path's timestamp origin and missing-sample indicators alongside the clock test, so a collector outage cannot pass as a period with no congestion.

These two quiet fabrics reach further into revenue-earning workloads than their cost suggests.

Telemetry correlation is the observability stack of metrics, traces and logs you use to find a regression or attribute a stall. It is only as trustworthy as the clock its records are stamped with. PTP can make cross-device traces orderable when the full clock-and-capture uncertainty stays below the event separation.

RoCE diagnostics (chasing PFC storms, ECN cascades and microburst-induced drops on a lossless Ethernet fabric) are a microsecond-scale forensic exercise. A millisecond-error clock-and-capture path cannot resolve them.

Multi-DC training, meaning asynchronous and hierarchical schemes that span campuses (Chapter 8.8), bounds staleness with logical optimizer steps and synchronization intervals, not PTP. A shared time base makes its cross-site telemetry orderable. If each site takes its time from its own GNSS reference, not over the link you are measuring, it also lets you split one-way latency between the two directions of the inter-site link.

The management fabric is the substrate of goodput recovery itself. When a node stalls a synchronous run, your ability to reach its BMC, read why, power-cycle or drain it, and return the cluster to useful work is measured in minutes of stranded GPUs, which is real money against the depreciation clock.

Fund them deliberately as goodput and ramp enablers, not as cost centers to minimize. A physical OOB fabric and a measured PTP plane are a small fraction of a cluster's capex and an outsized fraction of its operability. They separate an incident that resolves in minutes from a console from one that takes hours of row walking, and a telemetry timeline you can trust from one you cannot. Scope them with the same rigor as the data plane and gate them in commissioning, and they become invisible in the best way: the failure you planned for still leaves an authenticated recovery path and a time-quality bound you can trust. Choosing a shared dependency instead saves equipment up front and can turn the next data-plane fault into a remote-hands outage.