Observability, Telemetry & GPU Health

A frontier training cluster fails constantly. Meta's published Llama 3 405B snapshot recorded 419 unplanned interruptions over 54 days on 16,384 H100s — roughly one every three hours — with approximately 78% hardware-caused; the paper reports 58.7% as GPU issues, although Table 5's counts and printed percentages do not reconcile (Meta, 2024). SemiAnalysis reported about seven days of MTBF for one 512-H100 cluster at a top-tier operator in October 2024. That is not a per-GPU hazard rate and cannot be scaled linearly to a 100k-GPU cadence; use the named fleet's measured job-level interruption distribution, including correlated hardware, shared-service, software, membership, and detection effects. Components will still fail; what varies between operators is whether the observability stack notices in seconds, attributes the failure to the right node, and triggers the right recovery — or lets a synchronous job limp along behind a straggler, or worse, keep training on corrupted gradients no counter flagged.

This chapter treats observability as the operational nervous system that closes that loop. We build it from the bottom: the compute-plane telemetry (DCGM/NVML and the XID/SXID error taxonomy), the fabric-plane telemetry (congestion and health counters that tell you whether the network or the GPU is the bottleneck), and the facility-plane telemetry (liquid-cooling and inlet correlation). Then comes the hardest detection problem — silent data corruption, which produces no error at all — and the case for elevating goodput from a marketing number to the headline metric that every other signal exists to protect. We close on the architecture and cost decision that quietly dominates the others: self-hosted versus managed, and the cardinality budget that governs what you can afford to watch. Each detection fork is scored by the goodput or dollar cost of getting it wrong.

A GPU node emits signal from three independent stacks that historically belonged to three different teams. The compute plane — accelerator utilization, memory, power, temperature, clocks, ECC counts, XID/SXID events — flows from NVIDIA's NVML library and is aggregated by DCGM (Data Center GPU Manager), whose dcgm-exporter publishes Prometheus-format metrics that any open observability platform can ingest (NVIDIA DCGM docs, 2025). The fabric plane — link flaps, CRC/symbol errors, PFC pause frames, ECN marks, queue depth, NCCL collective timings — comes from switch counters (InfiniBand UFM/NetQ, or RoCE switch telemetry) and from the collective library itself. The facility plane — coolant inlet temperature and flow, CDU pressure, leak detection, rack PDU draw — comes from DCIM and BMS systems on entirely different protocols (Redfish, Modbus, SNMP).

Collecting each plane is solved. Correlating them to a single node, at the moment of an incident, is where most stacks fall down — and the failure is expensive. A GB200 NVL72 GPU throttles up to ~50% when its coolant inlet drifts above the DLC envelope; if the compute plane reports a throttled, underperforming GPU while the facility plane separately reports a CDU pressure excursion, and nothing joins them, you dispatch a hardware tech to RMA a perfectly good GPU while the real fault — a cooling loop — keeps degrading the rest of the rack. The correlation key is mundane but decisive: every metric, in every plane, must carry a consistent node/rack/job label set so a thermal event, a throttle, and a slow collective resolve to one root cause instead of three unrelated pages. This is the same correlation discipline the facility side builds in Chapter 14.2; the difference here is that the join must happen at job granularity, fast enough to cordon a node before it poisons a synchronous step.

NVIDIA's XID errors are the primary hardware-fault vocabulary for GPUs; SXID is the NVSwitch equivalent for the scale-up fabric. The taxonomy is not flat — XIDs span the full severity range from purely informational to instantly fatal — and the central operational decision is how each event enters a versioned, platform-specific diagnostic flow before automation changes node state. Draw it too loosely and corrupt work propagates; draw it too aggressively and you cordon healthy nodes on transient noise, shrinking the cluster and tanking goodput from the recovery side.

Xid/SXid codes are evidence inputs, not universal dispositions. Resolve the code against NVIDIA's current Xid catalog for the named GPU, driver/CUDA release, MIG/fabric topology, and linked Xids; capture nvidia-bug-report, DCGM/field-diagnostic output, recurrence and scope before repair decisions. For memory events, include the recovery-action flag, pending/failed row remap or page retirement, and reset result. XID 94 is contained to the affected application on supported GPUs: restart that application and validate or discard its uncommitted output, while unaffected applications remain outside that error's containment boundary. XID 95 is uncontained: reset/reboot as the current platform guidance requires and treat work on the affected GPU since the last validated boundary as suspect. XID 48 and XID 79 likewise enter the current debug/field-diagnostic flow; neither a fixed recurrence window nor one code alone creates a universal RMA decision. DCGM can recommend monitor, reset, triage, or isolate actions, but does not determine RMA eligibility; the applicable OEM/support contract does.

Two routing subtleties separate a mature stack from a noisy one. First, attribution direction: an XID 13 or 31 can arise from application, driver, or hardware causes. Use the current catalog trigger conditions, application replay/debug evidence, recurrence, and cross-node/platform scope before assigning ownership. Auto-evicting on every app-level XID hands tenants the power to shrink your cluster with buggy code. Second, SXID scope follows the reported switch/port, severity, topology, and Fabric Manager guidance; do not infer a single-GPU or entire-rack drain boundary from the SXID label alone — a distinction that gets sharper as scale-up domains widen on the density ramp toward Kyber-class racks. The fabric-side counters that disambiguate "is it the GPU or the network?" — PFC pause frames, ECN marks, queue depth — are the bridge to congestion engineering in Chapter 8.6: a collective that suddenly runs slow is a network-health question first and a GPU-health question second, and only correlated telemetry tells you which.

Every error discussed so far announces itself. Silent data corruption (SDC) does not. A marginal core or memory cell computes the wrong answer — a flipped bit in a multiply, a miscomputed gradient — and emits no XID, no ECC count, no log line. The hardware believes it succeeded. In a training run, a single SDC can quietly poison a gradient, corrupt an optimizer state, or push a loss curve subtly off-trajectory; you may not discover it for days, and when you do, the only safe recovery is to roll back to a checkpoint taken before the corruption — discarding all the goodput in between. This is the failure mode that makes a stack with 99% XID coverage still untrustworthy.

The scale of the problem is now well-characterized and uncomfortable. Meta's published AI-hardware reliability work reports roughly 1 in 1,000 machines harboring an SDC-prone defect (Meta Engineering, 2025). Meta reports that for large-scale AI training, an SDC event is expected every one to two weeks, and recorded six SDCs during its 54-day, 16K-H100 Llama-3 run. Because the defect is silent, you cannot wait for it to surface — you have to go hunting. Meta's published architecture is the reference: two complementary detectors, Fleetscanner (opportunistic — a machine is fully drained, exhaustively tested, then quarantined or returned to rotation) and Ripple (in-production — test loads are sliced into the gaps between real workloads), together running about 2.5 billion test seeds per month across the fleet (Meta Engineering, 2022).

Goodput has a precise definition worth adopting verbatim. Google Cloud frames it as productive ML throughput net of badput — and the value is in the badput taxonomy, because it forces you to attribute every lost GPU-second to a category you can attack. Badput buckets include: scheduling badput (waiting for resources), provisioning/initialization badput (the slow ramp before steady state — see Chapter 10.5), disruption badput (the failure-detect-and-restart loop), wasted-progress badput (work done since the last checkpoint, thrown away on restart), and SDC badput (work that must be rolled back because it was corrupt). Each bucket points at a different fix: disruption badput is a detection-latency problem; wasted-progress badput is a checkpoint-cadence problem (the Young/Daly optimal-interval math is canonical in Chapter 9.4); SDC badput is a hunting problem.

The job-level observability that produces these numbers must be assembled deliberately. Raw DCGM utilization is necessary but not sufficient — it cannot distinguish a GPU doing useful work from one busy-waiting on a straggler or recomputing a corrupted step. Real goodput accounting requires joining hardware telemetry to training-loop telemetry: step time, samples/sec, optimizer-step commit events, and checkpoint-write completions, emitted by the framework itself. Only at that join can you compute the fraction of wall-clock GPU-time that produced committed, correct progress — and only then does the headline number on the wall mean what operators think it means. SemiAnalysis's ClusterMAX rating makes goodput and health-check rigor a first-class scoring dimension precisely because it is the cleanest single proxy for whether an operator's whole stack works.

The architecture fork is the same one every observability buyer faces, sharpened by GPU economics. Self-hosted — Prometheus (or VictoriaMetrics/Mimir/Thanos for scale) plus Grafana, fed by dcgm-exporter and node/fabric exporters — gives full control, no per-host SaaS bill, and data that never leaves your security boundary (which matters for sovereign and air-gapped builds; see Chapter 11.7). The cost is that you now operate a high-cardinality time-series database at fleet scale as a production service of its own. Managed — Datadog, Chronosphere, Grafana Cloud, and the GPU-cloud-native stacks — offloads that operational burden but bills on ingest and active time-series, and at GPU-fleet cardinality those bills get large fast. Chronosphere's own framing is blunt: high observability costs steal budget from GPUs, training, and staff.

The variable that dominates both paths is cardinality, and it must be counted from the telemetry actually emitted. For a declared scrape and relabeling model, total active series is the sum of the series emitted by each metric family and label set: C_total = Σ_m|S_m|. A Cartesian product is valid only if the exporter or recording rules really emit every combination. Inventory each metric family, scrape target and post-relabel label set, then measure backend active-series count, new-series creation and churn rather than relying on a theoretical multiplication. AI fleets make unbounded job, request, pod, session and tenant identifiers especially dangerous: isolate those identifiers in logs, traces, object storage or a dedicated bounded store, and aggregate or drop them before the primary time-series database. High cardinality remains a major observability-cost driver (Grafana Labs; Last9, 2025), but the budget must follow the declared emission model and measured series behavior.

The mature answer is neither pure column — it is a tiered telemetry pipeline with deliberate cardinality control, and it is now the dominant 2026 pattern regardless of which backend you buy. The mechanics: keep high-resolution, high-cardinality raw telemetry (per-GPU, per-second DCGM) in cheap object storage for forensic and ad-hoc analysis, while sending aggregated, lower-cardinality summaries to the primary backend that powers live dashboards and alerts (Last9; NVIDIA DCGM collector docs, 2025). Relabel to drop labels nobody queries, downsample old data, and put a hard cardinality budget on each team's namespace. Verify that budget against measured active-series count and series-creation churn, and isolate unbounded job, request, pod and session identifiers outside the primary time-series store so a new tenant cannot silently multiply the bill. You cannot afford to watch everything at full fidelity forever, so decide explicitly what stays hot, what goes cold, and what gets dropped — and make that a budgeted decision rather than an emergent one. Get it wrong on a power-bound, capital-intense fleet and the observability stack becomes a line item that competes with the GPUs it was built to protect.