Observability, Telemetry & GPU Health

A frontier training cluster fails constantly. Meta's published Llama 3 405B snapshot recorded 419 unplanned interruptions over 54 days on up to 16,384 H100s, roughly one every three hours, with Table 5 attributing 148 events to faulty GPUs and 72 to HBM3 errors (Meta, July 2024). SemiAnalysis reported about seven days as the typical MTBF for a 512-H100 cluster at a top-tier operator in October 2024. That figure is not a per-GPU hazard rate and cannot be scaled linearly to a 100k-GPU cadence. Use your own fleet's measured job-level interruption distribution, which includes correlated hardware, shared-service, software, membership and detection effects. Components will still fail. What differs between operators is whether the observability stack notices in seconds, pins the failure on the right node and triggers the right recovery, or lets a synchronous job limp along behind a straggler or, worse, keep training on corrupted gradients no counter flagged.

Observability is the operational nervous system that closes that loop. The chapter builds it from the bottom: compute-plane telemetry (DCGM/NVML and the XID/SXID error taxonomy), fabric-plane telemetry (congestion and health counters that show whether the network or the GPU is the bottleneck) and facility-plane telemetry (liquid-cooling and inlet correlation). It then takes the hardest detection problem, silent data corruption, which produces no error at all, and makes the case for promoting goodput from a marketing number to the headline metric that every other signal exists to protect. The last is the architecture and cost decision that dominates the others: self-hosted versus managed, and the cardinality budget that sets what you can afford to watch. Each detection fork is scored by the goodput or dollar cost of getting it wrong.

A GPU node emits signal from three independent stacks that historically belonged to three different teams. The compute plane covers accelerator utilization, memory, power, temperature, clocks, ECC counts and XID/SXID events. On NVIDIA it comes from the NVML library and is aggregated by DCGM (Data Center GPU Manager), whose dcgm-exporter publishes Prometheus-format metrics that any open observability platform can ingest (NVIDIA DCGM docs, 2025). On AMD the data comes from AMD SMI and AMD's device metrics exporter. ROCm 10.1 (October 2026) stopped building the older, deprecated ROCm SMI, so anything scripted against rocm-smi has to move to AMD SMI. The fabric plane covers link flaps, CRC/symbol errors, PFC pause frames, ECN marks, queue depth and NCCL collective timings. It comes from switch counters (UFM for InfiniBand; NetQ or switch streaming telemetry for Spectrum-X and other RoCE Ethernet fabrics) and from the collective library itself. The facility plane covers coolant inlet temperature and flow, CDU pressure, leak detection and rack PDU draw. It comes from DCIM and BMS systems on entirely different protocols (Redfish, Modbus, SNMP).

Collecting each plane is solved. Correlating them to a single node at the moment of an incident is where most stacks fall down, and the failure is expensive. A coolant inlet or flow excursion can drive a GB200 NVL72 GPU into thermal throttling, and the throttle reason, clocks, power and step time for your rack profile are what let you measure the lost output. Suppose the compute plane reports a throttled, underperforming GPU while the facility plane separately reports a CDU pressure excursion, and nothing joins them. You dispatch a technician to RMA a perfectly good GPU while the real fault, a cooling loop, keeps degrading the rest of the rack.

The correlation key is mundane but decisive. Every metric in every plane must carry stable node and rack dimensions. Join job through a time-bounded allocation map instead of stamping it into every hardware series. Then a thermal event, a throttle and a slow collective resolve to one root cause instead of three unrelated pages. Correlatable does not mean identically labelled: job, request and pod identifiers churn, and writing them into facility metrics causes exactly the cardinality explosion the budget later in this chapter exists to prevent. The facility side builds the same correlation discipline in Chapter 14.2; the difference here is that the join must happen at job granularity, fast enough to cordon a node before it poisons a synchronous step.

NVIDIA's XID errors are the primary hardware-fault vocabulary for GPUs. SXID is the NVSwitch equivalent for the scale-up fabric. The taxonomy is not flat: XIDs run from purely informational to instantly fatal, and the central operational decision is what each event triggers, and on what data, before automation changes a node's state. Draw that flow too loosely and corrupt work propagates. Draw it too aggressively and you cordon healthy nodes on transient noise, which shrinks the cluster and cuts goodput from the recovery side. The OCP RAS API specification (v0.96.0, 9 October 2026, still pre-1.0) defines a vendor-neutral fault path for CPUs, GPUs and accelerators. Components report errors as UEFI CPER records and accept actions, up to part replacement, as CPADs over existing transports such as Redfish and PLDM. In a mixed-vendor fleet, ask each supplier for CPER output and a RAS API roadmap so the fault pipeline survives a supplier or generation change.

An Xid or SXid code is one input to a decision, not the decision. Look each code up in NVIDIA's current Xid catalog for your GPU, driver/CUDA release and MIG/fabric topology, and check the Xids that arrived with it. Capture nvidia-bug-report, DCGM/field-diagnostic output, recurrence and scope before any repair decision. For memory events, also record the recovery-action flag, any pending or failed row remap or page retirement, and the reset result. XID 94 is contained to the affected application on supported GPUs: restart that application and validate or discard its uncommitted output, while other applications keep running. XID 95 is uncontained: reset or reboot as the platform guidance requires, and treat everything the affected GPU computed since the last validated checkpoint or validation point as suspect. XID 48 and XID 79 likewise go through NVIDIA's debug/field-diagnostic flow; neither a fixed recurrence window nor one code alone makes an RMA decision. DCGM can recommend monitor, reset, triage, or isolate actions, but does not determine RMA eligibility; your OEM or support contract does.

Two routing subtleties separate a mature stack from a noisy one. First, attribution direction: an XID 13 or 31 can come from the application, the driver or the hardware. Before assigning ownership, check the catalog's trigger conditions, replay the application and read the debug logs. Then look at recurrence, and at whether the code also shows up on other nodes or on machines running the same driver release. Auto-evicting on every app-level XID hands tenants the power to shrink your cluster with buggy code. Second, SXID scope follows the reported switch and port, severity, topology and Fabric Manager guidance. Do not infer a single-GPU or whole-rack drain boundary from the SXID label alone, a distinction that gets sharper as scale-up domains widen on the density ramp toward Kyber-class racks.

The fabric counters that answer "is it the GPU or the network?" (PFC pause frames, ECN marks, queue depth) are the bridge to congestion engineering in Chapter 8.6. A collective that suddenly runs slow is a network-health question first and a GPU-health question second, and only correlated telemetry tells you which.

Every error discussed so far announces itself. Silent data corruption (SDC) does not. A marginal core or memory cell computes the wrong answer, such as a flipped bit in a multiply or a miscomputed gradient, and emits no XID, no ECC count and no log line. The hardware believes it succeeded. In a training run, one SDC can poison a gradient, corrupt an optimizer state or push the loss curve subtly off-trajectory. You may not find out for days, and recovery may then require rolling back to a checkpoint taken before the corruption, discarding all the goodput in between. No XID dashboard, however complete, covers a computation that never raises one.

The published examples establish a screening need, measured on different boundaries. Meta's published AI-hardware reliability work reports roughly one fault per 1,000 devices, with no exposure period to turn it into an annualized rate or a defective-machine prevalence (Meta Engineering, 2025). Separately, Meta’s July 2024 Llama 3 report records six SDC-attributed interruptions during 54 days of 405B pre-training on up to 16,384 H100s (Table 5). That campaign count is not a complete inventory of silent errors or a transferable event interval. Because the defect is silent, you cannot wait for it to surface; you have to go hunting.

Meta's published architecture is the reference, with two complementary detectors. Fleetscanner is opportunistic: a machine is fully drained, tested against its diagnostic suite, then quarantined or returned to rotation. Ripple runs in production, slicing test loads into the gaps between real workloads. Ripple alone runs about 2.5 billion test seeds per month, and Fleetscanner's coverage is reported as a separate cumulative total (Meta Engineering, 2022). In July 2025 Meta reported that both detectors, along with its kernel-level Hardware Sentinel, now run in production across its whole fleet, AI and non-AI hardware alike. Fleetscanner revisits the fleet every 45–60 days, and Ripple does so in days. The 2.5 billion seeds are still the 2022 CPU-fleet count, so do not read them as GPU coverage.

Use the canonical Chapter 14.1 useful-output ledger for the job and fleet boundaries. Google Cloud frames goodput as productive ML throughput net of badput, and the value is in the badput taxonomy, because it forces you to assign every lost GPU-second to a category you can attack. The buckets are scheduling badput (waiting for resources), provisioning/initialization badput (the slow ramp before steady state; see Chapter 10.5), disruption badput (the failure-detect-and-restart loop), wasted-progress badput (work done since the last checkpoint, thrown away on restart) and SDC badput (work rolled back because it was corrupt). Each bucket points at a different fix. Disruption badput spans detection, fencing, allocation, reload and validation. Wasted-progress badput is a checkpoint-cadence problem, and the Young/Daly optimal-interval math is canonical in Chapter 9.4. SDC badput is a hunting problem.

Producing these numbers takes deliberate job-level observability, starting with signals that show real work. GPU utilization (nvidia-smi's GPU-Util, DCGM's GPU_UTIL) is the share of the sample period in which one or more kernels was executing. One kernel resident on a single SM is enough to read 100%, which proves only that something is running, not that the job is moving forward. DCGM's profiling fields (DCGM_FI_PROF_*) look inside the chip: SM_ACTIVE (time with at least one warp active, averaged across SMs), SM_OCCUPANCY (resident warps against the maximum), PIPE_TENSOR_ACTIVE (cycles the tensor pipe was busy, the nearest hardware signal to model FLOPS utilization), DRAM_ACTIVE, and NVLink and PCIe bytes in each direction. NVIDIA's DCGM guide treats SM activity of 0.8 or more as necessary but not sufficient for effective use, and below 0.5 as a likely sign of waste.

The profiling fields have costs. They sample at 1 Hz by default, multiplex some counters, and conflict with Nsight Systems and Nsight Compute, so pause collection (dcgmi profile --pause) while a developer profiles the node. For the collectives, NVIDIA's NCCL Inspector profiler plugin (December 2025) logs algorithmic bandwidth, bus bandwidth and duration for each collective on each communicator, and it is light enough to run in production. A straggler stretches the collective on every rank that waits for it, so the plugin alone cannot tell a slow network from a slow GPU. Read against step time and the DCGM fields, its records help show which of the two cost the throughput. Chapter 10.11 applies the same utilization signals to inference autoscaling.

None of these signals is goodput. MFU measures achieved model FLOPS against the hardware's peak; goodput is the share of wall-clock time that advances the job, so a busy tensor pipe on a step that later rolls back is still badput. Real goodput accounting joins hardware telemetry to training-loop telemetry that the framework itself emits: step time, samples/sec, validated optimizer-step events, and checkpoint capture and durable-commit events. Only at that join can you compute the fraction of wall-clock GPU time that produced committed, correct progress, and only then does the number on the wall mean what operators think it means.

The spread between providers shows what detection and recovery are worth. SemiAnalysis's April 2026 ClusterMAX model counts the overhead of failures, recovery and fault tolerance as a goodput expense, a cost no provider invoices, which works out as a percentage on top of the rental bill. For one pretraining scenario on 5,184 GB300 GPUs at $4/GPU-hr it comes to 6.1% for a Gold-tier provider, 10.5% for a hyperscaler and 20.9% for a Silver-tier provider, on SemiAnalysis's own fault-tolerance and repair-time assumptions. Those are modelled outputs, not measured loss rates. SemiAnalysis's ClusterMAX 3.0 (September 2026) rates GPU clouds on itemized criteria across ten categories, with security graded separately. It is a useful outside check on a provider, not a substitute for measuring your own output and losses.

The architecture fork is the one every observability buyer faces, sharpened by GPU economics. Self-hosted means Prometheus (or VictoriaMetrics, Mimir or Thanos for scale) plus Grafana, fed by dcgm-exporter and node and fabric exporters. It gives full control, no per-host SaaS bill, and data that never leaves your security boundary, which matters for sovereign and air-gapped builds (see Chapter 11.7). The cost is that you now operate a high-cardinality time-series database at fleet scale as a production service of its own. Managed means Datadog, Chronosphere (part of Palo Alto Networks since January 2026), Grafana Cloud or a GPU-cloud-native stack. It offloads that operational burden but bills on ingest and active time series, and at GPU-fleet cardinality those bills get large fast. Chronosphere's own framing is blunt: high observability costs steal budget from GPUs, training and staff.

The variable that dominates both paths is cardinality, and you must count it from the telemetry actually emitted. For a given scrape and relabeling model, total active series is the sum of the series emitted by each metric family and label set: C_total = Σ_m|S_m|. A Cartesian product holds only if the exporter or recording rules really emit every combination. So inventory each metric family, scrape target and post-relabel label set, then measure backend active-series count, new-series creation and churn instead of relying on a theoretical multiplication. AI fleets make unbounded job, request, pod, session and tenant identifiers especially dangerous. Isolate those identifiers in logs, traces, object storage or a dedicated bounded store, and aggregate or drop them before the primary time-series database. High cardinality remains a major observability-cost driver (Grafana Labs; Last9, 2025), but the budget has to follow your emission model and measured series behavior.

The mature answer is neither pure column. It is a tiered telemetry pipeline with deliberate cardinality control, where you place each stream by cost, query latency and incident evidence. One implementation keeps high-resolution, high-cardinality raw telemetry (per-GPU, per-second DCGM) in cheap object storage for forensic and ad-hoc analysis, and sends aggregated, lower-cardinality summaries to the primary backend that powers live dashboards and alerts (Last9; NVIDIA DCGM collector docs, 2025). Relabel to drop labels nobody queries, downsample old data, and put a hard cardinality budget on each team's namespace. Verify that budget against measured active-series count and series-creation churn, and keep unbounded job, request, pod and session identifiers out of the primary time-series store so a new tenant cannot silently multiply the bill.

You cannot afford to watch everything at full fidelity forever, so decide explicitly what stays hot, what goes cold and what gets dropped, and budget that decision instead of letting it emerge. Apply the same residency, support-access, backup and outage checks to managed and self-hosted backends, because self-hosting alone does not prevent an export. Get this wrong on a power-bound, capital-intense fleet and the observability stack becomes a line item that competes with the GPUs it was built to protect.