Observability, Telemetry & GPU Health

A frontier training cluster fails constantly. Meta's published Llama 3 405B snapshot recorded 419 unplanned interruptions over 54 days on 16,384 H100s — roughly one every three hours — with 78% hardware-caused and 58.7% GPU-related (Meta, Llama 3 paper, 2024). A best-in-class operator running mature H100 fleets sees a mean time between failures of about seven days per 512 GPUs (SemiAnalysis, 2025); scale that to a 100k-GPU site and you are absorbing a hardware fault every few minutes around the clock. The question is never whether components fail. It is whether your observability stack notices the failure in seconds, attributes it to the right node, and triggers the right recovery — or whether it lets a synchronous job limp along behind a straggler, or worse, keep training on corrupted gradients no counter flagged.

This chapter treats observability as the operational nervous system that closes that loop. We build it from the bottom: the compute-plane telemetry (DCGM/NVML and the XID/SXID error taxonomy), the fabric-plane telemetry (congestion and health counters that tell you whether the network or the GPU is the bottleneck), and the facility-plane telemetry (liquid-cooling and inlet correlation). We then confront the hardest detection problem of all — silent data corruption, the failure that produces no error at all — and elevate goodput from a marketing number to the headline metric that every other signal exists to protect. We close on the architecture and cost decision that quietly dominates the others: self-hosted versus managed, and the cardinality budget that governs what you can afford to watch. Each detection fork is scored by the goodput or dollar cost of getting it wrong.

A GPU node emits signal from three independent stacks that historically belonged to three different teams. The compute plane — accelerator utilization, memory, power, temperature, clocks, ECC counts, XID/SXID events — flows from NVIDIA's NVML library and is aggregated by DCGM (Data Center GPU Manager), whose dcgm-exporter publishes Prometheus-format metrics that any open observability platform can ingest (NVIDIA DCGM docs, 2025). The fabric plane — link flaps, CRC/symbol errors, PFC pause frames, ECN marks, queue depth, NCCL collective timings — comes from switch counters (InfiniBand UFM/NetQ, or RoCE switch telemetry) and from the collective library itself. The facility plane — coolant inlet temperature and flow, CDU pressure, leak detection, rack PDU draw — comes from DCIM and BMS systems on entirely different protocols (Redfish, Modbus, SNMP).

Collecting each plane is solved. Correlating them to a single node, at the moment of an incident, is where most stacks fall down — and the failure is expensive. A GB200 NVL72 GPU throttles up to ~50% when its coolant inlet drifts above the DLC envelope; if the compute plane reports a throttled, underperforming GPU while the facility plane separately reports a CDU pressure excursion, and nothing joins them, you dispatch a hardware tech to RMA a perfectly good GPU while the real fault — a cooling loop — keeps degrading the rest of the rack. The correlation key is mundane but decisive: every metric, in every plane, must carry a consistent node/rack/job label set so a thermal event, a throttle, and a slow collective resolve to one root cause instead of three unrelated pages. This is the same correlation discipline the facility side builds in Chapter 14.2; the difference here is that the join must happen at job granularity, fast enough to cordon a node before it poisons a synchronous step.

NVIDIA's XID errors are the primary hardware-fault vocabulary for GPUs; SXID is the NVSwitch equivalent for the scale-up fabric. The taxonomy is not flat — XIDs span the full severity range from purely informational to instantly fatal — and the central operational decision is where you draw the line between auto-remediate, page a human, and log only. Draw it too loosely and corrupt work propagates; draw it too aggressively and you cordon healthy nodes on transient noise, shrinking the cluster and tanking goodput from the recovery side.

The canonical fatal signatures are unambiguous and should trigger automated cordon-and-drain with no human in the path. XID 79 ("GPU has fallen off the bus") means the GPU is gone from the host's view — dead or dying, immediate node eviction (NVIDIA XID reference, 2025). XID 48 (double-bit ECC) is an uncorrectable memory error; a single event can be a cosmic-ray fluke, but a repeat within a week is progressive DRAM failure and an RMA. XID 94 (contained ECC) versus XID 95 (uncontained ECC) is the sharpest fork in the catalog: 94 means the hardware contained the error to a single application and the rest of the node is trustworthy; 95 means it could not be contained and workload outputs may be corrupted — an XID 95, especially alongside a rising rate of XID 94, is a memory-degradation trajectory, not a point event. The table below is the decision the chapter exists to force: a default routing policy you tune to your false-positive tolerance.

Two routing subtleties separate a mature stack from a noisy one. First, attribution direction: an XID 13 or 31 (MMU/graphics fault) is usually a tenant's bad kernel, not a sick node — route it to the job's owner, and only escalate to hardware suspicion when the same signature appears across many nodes (a real hardware or driver-version pattern). Auto-evicting on every app-level XID hands tenants the power to shrink your cluster with buggy code. Second, SXID maps to a domain, not a device: an NVSwitch fault degrades an entire NVLink scale-up domain (8, 72, heading to 576 GPUs), so the remediation unit is the rack-scale block, not the single GPU — a distinction that gets sharper as scale-up domains widen on the density ramp toward Kyber-class racks. The fabric-side counters that disambiguate "is it the GPU or the network?" — PFC pause frames, ECN marks, queue depth — are the bridge to congestion engineering in Chapter 8.6: a collective that suddenly runs slow is a network-health question first and a GPU-health question second, and only correlated telemetry tells you which.

Every error discussed so far announces itself. Silent data corruption (SDC) does not. A marginal core or memory cell computes the wrong answer — a flipped bit in a multiply, a miscomputed gradient — and emits no XID, no ECC count, no log line. The hardware believes it succeeded. In a training run, a single SDC can quietly poison a gradient, corrupt an optimizer state, or push a loss curve subtly off-trajectory; you may not discover it for days, and when you do, the only safe recovery is to roll back to a checkpoint taken before the corruption — discarding all the goodput in between. This is the failure mode that makes a stack with 99% XID coverage still untrustworthy.

The scale of the problem is now well-characterized and uncomfortable. Hyperscalers — Meta, Google, Alibaba — independently converged on roughly 1 in 1,000 machines harboring an SDC-prone defect (Meta Engineering, 2025; corroborated by OCP's SDC-in-AI whitepaper). Meta reports that for large-scale AI training, an SDC event is expected every one to two weeks, and recorded six SDCs during its 54-day, 16K-H100 Llama-3 run. Because the defect is silent, you cannot wait for it to surface — you have to go hunting. Meta's published architecture is the reference: two complementary detectors, Fleetscanner (opportunistic — a machine is fully drained, exhaustively tested, then quarantined or returned to rotation) and Ripple (in-production — test loads are sliced into the gaps between real workloads), together running about 2.5 billion test seeds per month across the fleet (Meta Engineering, 2025).

Goodput is the metric every other signal in this chapter exists to protect, and it has a precise definition worth adopting verbatim. Google Cloud frames it as productive ML throughput net of badput — and the discipline is in the badput taxonomy, because it forces you to attribute every lost GPU-second to a category you can attack. Badput buckets include: scheduling badput (waiting for resources), provisioning/initialization badput (the slow ramp before steady state — see Chapter 10.5), disruption badput (the failure-detect-and-restart loop), wasted-progress badput (work done since the last checkpoint, thrown away on restart), and SDC badput (work that must be rolled back because it was corrupt). Each bucket points at a different fix: disruption badput is a detection-latency problem; wasted-progress badput is a checkpoint-cadence problem (the Young/Daly optimal-interval math is canonical in Chapter 9.4); SDC badput is a hunting problem.

The job-level observability that produces these numbers must be assembled deliberately. Raw DCGM utilization is necessary but not sufficient — it cannot distinguish a GPU doing useful work from one busy-waiting on a straggler or recomputing a corrupted step. Real goodput accounting requires joining hardware telemetry to training-loop telemetry: step time, samples/sec, optimizer-step commit events, and checkpoint-write completions, emitted by the framework itself. Only at that join can you compute the fraction of wall-clock GPU-time that produced committed, correct progress — and only then does the headline number on the wall mean what operators think it means. SemiAnalysis's ClusterMAX rating makes goodput and health-check rigor a first-class scoring dimension precisely because it is the cleanest single proxy for whether an operator's whole stack works.

The architecture fork is the same one every observability buyer faces, sharpened by GPU economics. Self-hosted — Prometheus (or VictoriaMetrics/Mimir/Thanos for scale) plus Grafana, fed by dcgm-exporter and node/fabric exporters — gives full control, no per-host SaaS bill, and data that never leaves your security boundary (which matters for sovereign and air-gapped builds; see Chapter 11.7). The cost is that you now operate a high-cardinality time-series database at fleet scale as a production service of its own. Managed — Datadog, Chronosphere, Grafana Cloud, and the GPU-cloud-native stacks — offloads that operational burden but bills on ingest and active time-series, and at GPU-fleet cardinality those bills get large fast. Chronosphere's own framing is blunt: high observability costs steal budget from GPUs, training, and staff.

The variable that dominates both paths is cardinality, and it is the one most teams underestimate. Cardinality is the number of unique time series, and it multiplies: every metric, times every GPU, times every job, times every tenant, times every region, times every label you attach. DCGM alone exposes dozens of metrics per GPU; multiply by 100,000 GPUs and then by per-job and per-tenant labels and you are into hundreds of millions of active series — high cardinality is consistently cited as the single biggest driver of observability cost overruns across Prometheus, Datadog, and every time-series backend (Grafana Labs; Last9, 2025). The trap specific to AI fleets is cardinality drift: every new tenant, job ID, and model SKU mints fresh label values, so the series count grows without anyone deciding to spend more.

The mature answer is neither pure column — it is a tiered telemetry pipeline with deliberate cardinality control, and it is now the dominant 2026 pattern regardless of which backend you buy. The mechanics: keep high-resolution, high-cardinality raw telemetry (per-GPU, per-second DCGM) in cheap object storage for forensic and ad-hoc analysis, while sending aggregated, lower-cardinality summaries to the primary backend that powers live dashboards and alerts (Last9; NVIDIA DCGM collector docs, 2025). Relabel to drop labels nobody queries, downsample old data, and put a hard cardinality budget on each team's namespace so a new tenant cannot silently 10x the bill. The principle is the same one that governs the whole chapter: you cannot afford to watch everything at full fidelity forever, so you decide — explicitly — what you keep hot, what you keep cold, and what you drop, and you make that a budgeted decision rather than an emergent one. Get it wrong on a power-bound, capital-intense fleet and the observability stack becomes a line item that competes with the GPUs it was built to protect.