DCIM, Telemetry & Observability for GPU-Dense, Liquid-Cooled Facilities

A GPU-dense, liquid-cooled AI factory is the most heavily instrumented building most operators will ever run. It is also where instrumentation most often fails to answer the question that matters most: why did goodput just drop? Two telemetry universes share the hall, and they almost never share a clock, a namespace or an on-call rotation. The IT plane covers per-GPU temperature, power, SM clocks, ECC and XID/SXID error events, and NVLink and fabric counters. Agents such as NVIDIA's DCGM exporter sample these. The exporter's default collection interval is 30 s; operators who want second-scale detection tighten it to 1–10 s and pay in ingest and storage volume. A time-series database then scrapes the results. XID/SXID errors are driver events, so fast detection should take them from the event path, not from the scrape. The OT plane covers CDU flow and pressure, coolant supply/return temperature, dew point, valve positions, pump speeds, leak-detection zones, and switchgear and UPS state. Polling ranges from seconds for CDU flow and pressure up to a minute for plant trends, over Modbus, BACnet, SNMP and, increasingly, Redfish. A BMS/SCADA front end and a DCIM platform surface the data.

A handful of choices decide whether those two universes become one observable system, and they organize the chapter. It starts with the stack itself, what each layer measures and at what rate. Then comes the central choice, siloed telemetry versus IT/facility correlation, with the protocol and data-architecture decisions that make correlation possible or impossible. Liquid-cooling observability gets its own section, because liquid is where a missed signal becomes a flooded electrical room. Alarm management (an OT discipline on the ISA-18.2 / EEMUA-191 lineage) stays separate from alerting (an SRE discipline of SLO burn), because conflating them builds a system nobody trusts. The chapter closes on the operational twin, the live, as-built sibling of the design-validation twin in Chapter 2.7, and the autonomy ladder a DCIM climbs from dashboard to closed-loop control.

Newcomers treat telemetry as a volume problem: collect everything, store it forever, and the answers will be in there somewhere. On a 100,000-GPU campus that produces a monitoring system that is too expensive and still cannot resolve an incident. Think of it as a layered stack instead. Each layer has a native protocol, a native cadence and a native owner, and the engineering question is how the layers connect, not how much each one emits.

The IT/AI telemetry layer is the fast one. Per-GPU health flows from the device through NVML into DCGM and out through an exporter in Prometheus line format. Applicable uncorrectable errors and fatal XID/SXID events are immediate failure signals. Alarm on PCIe replay activity against the configured DCGM threshold, and on row remaps against NVIDIA's repair and failure flags rather than value > 0. Above the device sit the fabric counters (InfiniBand/UFM or Ethernet/NetQ), the scheduler's view of job and node state, and the application's own loss curve and step time. This layer's cadence is seconds. Its cardinality is very high, since a single GB200 NVL72 rack is 72 GPUs times dozens of metrics each. Its consumer is the SRE who needs to cordon a lemon node before it poisons a collective.

The facility/OT layer is the slow, durable one. The power chain (switchgear, UPS, PDU, branch-circuit metering), cooling plant (chillers, dry coolers, CDUs, pumps, valves), environmental sensors and physical-security systems report through PLCs and a BMS into a SCADA/DCIM front end. Cadence runs from seconds for CDU flow and pressure, through 5–30 s for the power chain, to minutes for slow plant trends; cardinality is modest, and the consumers are the facility engineer and the alarm system. The culture differs too: it values determinism, alarm rationalization and long-horizon trending over the high-cardinality, ephemeral-by-design ethos of the IT side. A thermal event starts in the OT layer and shows up as a goodput collapse in the IT layer, and left alone, the two layers will not tell you it is the same event.

The grid interface needs its own capture, because the polled power chain cannot see a grid disturbance. In the 2024 Virginia event, a run of 42–66 ms voltage sags pushed data-center sites onto backup power, taking about 1,500 MW of load off the grid. The 22 July 2026 event in Ashburn was roughly 2.5 times larger: a single fault, cleared normally, sent about 3.8 GW to backup (Chapter 4.10). For new large loads, the ride-through rules in force are ERCOT NOGRR282 and SPP's large-load requirements, and they act on a timescale no poll can resolve. Under ERCOT's rule, load that transferred must start coming back within 0.25 s of voltage recovery; between the two rules, the deadline for getting 90% of the load back ranges from 0.5 s to 2 s.

A 5–30 s poll can show only that something happened. To show what the site did, put GPS- or PTP-synchronized power-quality meters or disturbance recorders at the point of interconnection and on the MV feeders, capturing sub-cycle waveforms on a trigger. Log UPS, static-transfer-switch and rack battery-backup events to the millisecond on the same clock. NERC's CLO-003-1 adds disturbance-monitoring duties for computational-load sites of 50 MW or more supplied at 100 kV or above, with 36 months for sites already in service. It passed final ballot in October 2026 and awaits Board adoption and FERC approval (Chapter 4.3; ride-through rules in Chapter 4.10). Without those records you cannot show the utility that the site rode through, or tell a grid-caused goodput dip from a plant fault.

Correlation is won or lost at the seam between those layers, in three sub-decisions.

The protocol convergence decision. Historically the IT plane spoke IPMI and the OT plane spoke Modbus and BACnet, and the two never met. The 2026 lever is Redfish, the DMTF's RESTful successor to IPMI, which has grown from a server-management API into a facility-telemetry API. The DMTF added liquid-cooling equipment to Redfish in the 2023.1 release (CoolingUnit, CoolingLoop, LeakDetection, Pump, Reservoir and Filter schemas, described in DMTF's DSP2064 white paper), so a CDU and a GPU can now expose their state through the same schema and HTTP transport. That does not eliminate the BMS, because chillers, switchgear and legacy plant still speak BACnet and Modbus, but it gives you a convergence target. The fork is whether to adopt Redfish as the common model and bridge the stragglers, or keep a polyglot architecture and pay for translation at every integration. Above the wire, OpenTelemetry is becoming the lingua franca for the IT side's metrics, traces and logs, which lets application step time and node health share a schema with SRE tooling.

The time-sync and namespace decision. Correlation is impossible without a shared clock and a shared identity. If a GPU's monotonic clock timestamps a throttle event and a PLC's local clock timestamps a flow anomaly, you cannot align them to causality once the clocks drift by seconds. At one-second IT cadence, seconds of skew destroy the signal. PTP/NTP discipline across both planes is a precondition. Every datum also needs an asset key that resolves the physical topology: GPU → node → rack → CDU → branch circuit → PDU → room. Without that graph you cannot answer which GPUs sit downstream of the pump that just slowed, and that answer is the whole value of correlation.

The store-and-retain decision. A GPU-dense fleet is a cardinality bomb, but not mainly a disk bill. At 1 s sampling, 100,000 GPUs × 50 metrics is 5 million samples a second. At the 1–2 bytes per sample Prometheus documents, that is roughly 0.4–0.9 TB of disk a day, small beside the power bill. The cost sits in ingest and query compute and in series count: per-job, per-rank and per-link labels multiply the series the database must index and hold in memory, so size the cluster to active series and samples per second, not bytes. Store too little, though, and the one transient you needed is gone before the postmortem. The mature pattern is tiered: high-resolution hot storage for days, downsampled rollups for months, and event-triggered full-fidelity capture (a “flight recorder” that keeps raw samples around any XID, throttle or leak event). Tiering decides where the blind spot goes, so keep full fidelity exactly where incidents originate and accept lossy aggregation everywhere else.

In an air-cooled hall, a cooling failure is a slow thermal ramp that buys you minutes and at worst trips a thermal limit. In a direct-to-chip liquid hall the failure modes are faster and more consequential, so the telemetry has to be designed around them. Three matter above all others.

Throttle-by-deviation. For GB200 NVL72 (QCT's QoolRack reference), monitor supply and return independently against the vendor's limits: 45 °C maximum supply and 65 °C maximum return are separate bounds, not a prescribed pair. Establish flow from the approved coolant, the liquid heat load and your design ΔT. The guide's HPE GB200 NVL72 ~140–220 L/min water band is a heat-balance derivation for its 115 kW liquid load across OCP's 7.5–12 K rise, not an OEM flow requirement. A deviation may not fail the rack; it can instead throttle the GPUs while every node stays reachable. A naive availability monitor cannot see that: every node is “up,” every GPU is “healthy,” and goodput quietly falls. How far it falls is a product- and workload-specific curve that you measure from throttle reason code, clock, board power and step time together, not a fixed fraction. Only by correlating the CDU's flow and temperature telemetry with the GPUs' clock-throttle reason codes do you see that a cooling deviation, not a software bug, is eating the run.

Leak detection. Liquid above an electrical room is a step-change in risk, so leak telemetry is zoned, redundant and wired to act: rope or spot sensors per rack and per CDU, pressure-decay detection on the loops, and reservoir-level trending that catches slow loss before it becomes a puddle. Mature designs pair detection with isolation valves that close automatically and limit how much coolant a confirmed leak can release. Some vendors also run the loop below atmospheric pressure. A breach then draws air in rather than pushing coolant out, and it appears as a pressure or level anomaly instead of tripping a wet sensor (Chapter 5.6). The telemetry decision is the confidence threshold: a false trip drops a rack of GPUs, and a missed trip wets a busbar. What happens on a trip comes from the site's approved leak-detection cause-and-effect matrix. Require sensors to vote only where the hazard analysis supports it; a credible single sensor in a high-consequence spot may trip without waiting for corroboration. The leak-detection design, not the matrix, sets the safe state, sensor independence, bypass controls, proof-test interval, who has authority to act, and the response when sensors disagree.

Fluid quality and slow drift. The signals that predict a failure weeks out are undramatic: conductivity and pH creep, particulates and filter differential pressure, biological fouling, and dew-point margin narrowing toward condensation. These feed the predictive-maintenance program (canonical in Chapter 14.5) rather than the alarm system. A siloed IT stack never sees these trends, while a correlated plane surfaces them as a degrading-CDU early warning.

The most common cultural failure in an AI-factory operations center is treating the OT alarm system and the IT alert system as the same thing. They are different disciplines with different standards, and merging them naively produces the worst of both: an alarm flood nobody can action and an SLO-burn signal nobody trusts.

Alarm management is an OT engineering discipline on the ISA-18.2 / IEC 62682 / EEMUA-191 lineage. Its core insight is that an alarm is a request for operator action, not a notification. Every alarm must survive rationalization: it is justified against a written alarm philosophy and documented with a consequence, a required operator action and a time-to-respond. Anything that fails that test is not an alarm. The discipline exists because of a quantifiable failure mode, the alarm flood. By convention that means over ten new alarms within ten minutes for a single operator, a rate beyond what one operator can effectively manage. The standard's countermeasures are specific: rationalization to cut nuisance alarms, prioritization so critical alarms stand out, shelving (operator-controlled, time-bounded, audited suppression of a known nuisance alarm), and state-based suppression so a planned maintenance evolution does not generate a thousand expected alarms. In a liquid-cooled hall this matters acutely. A single CDU trip can cascade flow, temperature and throttle alarms across an entire row, and without designed flood suppression the operator sees a wall of red and cannot find the root cause inside it.

Alerting is the SRE discipline of paging on SLO burn: you define goodput, latency or error budgets and alert when the rate of consumption threatens the objective, deliberately avoiding alerts on every transient symptom. The two systems must coexist. Dumping every OT alarm into the IT pager, or every IT symptom onto the OT board, fails both disciplines at once. What holds up is a federated model: each system rationalized to its own discipline, with a deliberate, narrow set of cross-links. A confirmed cooling alarm that is eating goodput should reach the training SRE, and a fleet-wide goodput collapse should prompt the facility operator to look at the plant. Correlation feeds both; it does not merge them.

A DCIM sits at one position on a ladder, and the operator's strategic decision is how far up to climb. Each rung trades a capability against a risk. Read-only visualization gives dashboards and trends, with no analytics. Advisory analytics adds anomaly detection and predictive-maintenance lead times, and a human acts. Closed-loop control lets the DCIM move setpoints (pump speed, valve position, power caps) within bounded authority. The operational digital twin is a live, as-built model that simulates the consequence of an action before committing it, and increasingly runs an agentic control loop. The 2026 reference designs, NVIDIA's Omniverse DSX digital-twin blueprint and the hyperscalers' in-house plant analytics, point at the top of the ladder, but most operators are right to climb deliberately rather than leap.

Two twins need distinguishing: the operational twin and the design-validation twin of Chapter 2.7. The design twin is built before steel is cut, to validate a thermal and power design against a model; it is a planning artifact. The operational twin is its live descendant, continuously reconciled against as-built reality: real sensor streams, real failed pumps, real firmware revisions. When it predicts the effect of raising a setpoint, it predicts against the building that exists, not the one that was drawn. The decision that makes or breaks it is the as-built fidelity loop. A twin that drifts from reality (a swapped CDU, a re-cabled rack, a derated chiller the model does not know about) gives confidently wrong predictions, which is worse than no twin at all. Closing that loop, with continuous and re-commissioning telemetry feeding the model, is the subject of Chapter 14.14. The autonomy decisions the twin enables are in Chapter 14.13.

The final discipline is editorial. A GPU-dense facility can emit more telemetry than anyone will ever query, so the question is not what you can measure but what is load-bearing for the four numbers that govern the business: goodput, PUE, time-to-detect and time-to-cordon. Work backward from each.

• Goodput (the metric canonicalized in Chapter 14.1) is protected by correlating throttle-reason codes, XID/SXID events, fabric link flaps and CDU deviation, the signals that explain why effective training time dropped. If a metric does not help you attribute a goodput loss, it is not load-bearing for goodput. Utilization is not progress: a hung collective can show full utilization while doing no work (Chapter 14.4 has a three-hour NCCL-hang case). Watch board power instead, calibrated per platform so you can tell compute, communication, data starvation and a stall apart, and combine it with step-progress counters and per-rank heartbeats. Alarm when progress stalls, not when utilization falls.
• PUE / WUE is protected by power-chain and heat-rejection metering (branch-circuit power, CDU and chiller energy, water draw) trended over seasons. This is the OT plane's classic job, and where its long-horizon retention earns its keep.
• Time-to-detect is set by the fastest plane's cadence and the quality of the failure signals on it. Catching a lemon node before it poisons a collective therefore takes event-driven XID/SXID capture plus GPU health sampled in seconds; minute-scale facility polling is too slow.
• Time-to-cordon is set by how directly detection wires to action: the cordon-and-drain path that ejects a failing node, fed by the same health telemetry. It is treated in Chapter 10.7 (autonomous recovery) and Chapter 14.4 (detection, isolation and lemon-node ejection).

Everything else is at best useful context and at worst vanity telemetry you are paying to store. The cardinality-and-retention bargain from earlier in the chapter is the same decision seen from the cost side: keep full fidelity where it serves one of these four numbers, downsample everywhere else, and resist labeling and retaining everything because storage felt cheap on day one. At fleet scale the series count, not the disk, is what stops being cheap.

Close the telemetry incident. Preserve device time, collector time, clock-error bound, quality flag and topology version with the event. On stale or disagreeing data, suspend the affected optimization, keep independent protection active, and page the owner of the failed measurement path. After repair, replay a known event through collection, correlation and alerting, and check the downstream asset list against the as-built circuit and loop. Release automation only after that test passes. Chapter 14.1 keeps the useful-output tally; this chapter owns whether the data behind it arrived intact.

Make that replay a procurement requirement. Require a BMC/Redfish stack that can submit simulated events. Redfish EventService test events are the minimum, and Microsoft's October 2026 OCP reliability work points further, toward simulated fault events before deployment. At commissioning (Chapter 13.5), inject one simulated event per critical fault class and confirm it reaches an alert and a cordon. A test event only proves delivery; it says nothing about how the device really behaves when it faults. Until your own fleet measures them, treat vendors' repair-time gains as claims.