Failure-Mode / FMEA Catalog

This appendix collects the failure modes that AI load adds or changes, scattered through the engineering chapters, into one FMEA register with uniform fields. Four chapters refer to it: resilience standards (Chapter 12.1), the reliability rethink (Chapter 12.2), component failure rates (Chapter 14.3) and integrated systems testing (Chapter 13.6). It also supplies candidate top events and common-cause couplings to the availability model in Chapter 12.5, once the project has confirmed its equipment, operating state, fault domain and test results. The rows are not executable model inputs. The engineering for each mode lives in the chapter named in the right-most column; this catalog is the index and cross-walk, not the derivation. Faults in the conventional power chain (UPS, generator, switchgear), plus switching mistakes and fleet-wide pushes that can also cut power, appear as F-R13 to F-R17 under the availability model below.

An attack and a random failure may end in the same top event. Each still gets its own record, and the two are linked only through a top event the as-built topology proves they share (callout after the coupling table). Not every random failure has an attacker twin. The cyber-physical analysis is in Chapter 11.10.

Every failure mode is recorded in the same six fields across the three master tables, so the catalog sorts and compares cleanly:

• Trigger: the initiating event. Random causes come first; adversarial initiators stay in separate Part 11 records, linked to a proven shared top event.
• Propagation path: how the fault spreads and, above all, which shared mitigation it defeats on the way from local fault to cascade.
• Detection: the sensing modality and its latency relative to propagation speed, which is the decisive ratio.
• Blast radius: the fault domain affected at full propagation: node, rack, row, hall, or campus/POI.
• Mitigation: the preventive or containing control, classed as preventive (stops the trigger), inertial (buys ride-through time) or reactive (acts after detection).
• Recovery: the path back to service, kept as two separate numbers. Service-restoration time covers detect, isolate, transfer, repair or replace, restart and validate. Where the mode costs work, replayed work plus any integrity look-back to a known-clean state is the second number. Add them only where they cover distinct, non-overlapping parts of the job timeline; they are not interchangeable.

Where propagation outruns detection-plus-actuation, the reactive loop is too slow. Prevent the trigger, add inertia, or prove a faster independent protection path. The ratio matters because neither an operator nor an untested standby earns timing credit.

A direct-to-chip loop runs on two clocks. With flow maintained, the coolant and metal can buffer a heat step for tens of seconds, where the measured mass and load support it. With flow lost, the local mass remains but circulation no longer carries heat away. The rack's throttle and trip states can then arrive in seconds, on a window that the OEM's transient data and a commissioning pump-drop test set for the specific product. The leak and CDU/pump rows are engineered against the second clock. Detection and switchover must finish inside it, which is why standby pumps switch over automatically (on UPS-backed feeds where the ride-through study shows the source transfer outlasts the window) and why GPU power capping, not operator action, is the last layer. Chapter 5.11 engineers and tests the cause and effect; the measured response feeds the reliability model in Chapter 12.2.

Adversarial initiators: prove the shared physical event. Keep each security scenario's access, command authority and persistence separate from random-fault rates. F-R01 and F-R03: spoofing a dew-point setpoint can force condensation where the command drives exposed surfaces below dew point; the coolant-leak trace below follows that mechanism. F-R02: malicious VFD firmware or controller denial of service can stop circulation only where the compromised path can defeat the required flow and its independent protection. F-R04: CDU disablement that holds flow at zero links to HBM overtemperature only through the package's measured thermal response. F-R05: malicious power-cap firmware can force a synchronized load step only over the endpoints it controls, so use the measured MW step and timing. F-R07: BMS spoofing that disables cell balancing or thermal protection reaches the battery hazard only where independent cell/pack protection does not veto it. Chapter 11.10’s OT destructive-primitive and control-dependency assessment owns these five initiators.

F-R09: a deliberate cut of an undiverse fiber route belongs to Chapter 11.2’s fiber-vault and sole-path sabotage assessment. The shared physical event is the lost route, not a new random cut rate. F-R10: a thermal attack on the optics environment is a candidate 11.10 OT scenario only if cooling authority or physical access can drive the transceiver outside its rated environment and link/error tests reproduce the flap mechanism. F-R11: fault injection on a known-marginal device is a candidate scenario only if the attacker controls a relevant voltage, clock or thermal path and isolated known-answer tests establish undetected wrong results; a crash or corrected ECC event is not SDC. 11.5 bounds the confidential-computing threat model, 11.10 assesses reachable controls and 14.3 owns SDC evidence. No attack frequency is inferred from component failure data. Credit protection only after it survives the assessed attacker authority.

Detection differs by mode. An active fiber-path cut alarms immediately at the physical layer, while an untested route-diversity defect and SDC stay latent. Optics degradation is trend-detectable, and water curtailment shows up in make-up-water telemetry or on receipt of the curtailment order. An SDC event can poison a checkpoint and sit undetected for days. An optics flap can erode MFU before anyone traces tail latency to a marginal transceiver. An untested protection path shows its fault when the active route is cut. Match the control to the mode: a dedicated SDC-hunting program, per-port flap/FEC telemetry, verified-clean checkpointing, and periodic protection-path and physical-diversity verification bound the respective blast radii. The empirical fault taxonomy and the AFRs that feed these rows into the availability model are the canonical content of Chapter 14.3. That chapter also treats the SDC detection program, and Chapter 13.6 demonstrates it at commissioning.

For the availability model, the catalog's most useful output is the couplings between modes: the shared resources whose failure makes two independent-looking modes fail together. A fault tree or RBD that omits these common-cause terms will badly overstate availability. The table below lists the cross-mode interlocks worth modeling explicitly.

The rows above feed the reliability work, and each field maps to a model term. Top events come from the project's loss criteria and confirmed fault domains; a row becomes a top event when its blast radius reaches a domain the contract prices. Basic-event rates may take the fleet evidence in Chapter 14.3 where component, population, maturity and duty match; otherwise use project evidence and carry the uncertainty. Shared events come from the coupling table, confirmed against the as-built topology: a confirmed coupling is an explicit shared event, never an assumed beta. Recovery distributions use measured detection, protection, repair, restart and validation times, and checkpoint cadence couples the training rows' recovery. IST cases are the rows the design basis names for demonstration, by the test method Chapter 13.6 sets; the catalog does not itself authorize a hazardous live event. The model mechanics are in Chapter 12.5. For scale, the Llama 3 tiles above come from one fleet: 419 unexpected interruptions in 54 days on up to 16,384 H100s, about one every three hours, with faulty GPUs (148) and HBM3 memory (72) far ahead of switches and cables (35). Use them to check which rows should dominate a training model, not as your basic-event rates.

Add the conventional power-chain modes before you model. The tables above index the modes AI load adds or changes. They do not repeat the faults behind the largest share of outages: in Uptime Institute’s Annual Outage Analysis 2026 (May 2026), power was the primary cause of 45% of respondents’ most recent impactful outages in 2025, down from 54% in 2024 and still the largest category. An RBD or fault tree built only from F-R01 to F-R12 leaves out its biggest top events. Carry five more rows into the register:

• F-R13 · UPS or static-switch failure. A failed transfer or an inverter fault drops one UPS block and every PDU below it; 2N or distributed-redundant blocks and maintenance-bypass discipline contain it (Chapter 4.5).
• F-R14 · Generator fail-to-start or fail-to-run. A utility outage that the UPS was bridging becomes a site outage when the battery runs out; the generator’s load-acceptance class and start reliability set the risk. A spare set (N+1), regular test runs under real load and fuel-quality checks keep it contained (Chapter 4.8; acceptance in Chapter 13.3).
• F-R15 · MV switchgear, transformer or busway fault. The blast radius is one lineup or busway run, or the whole substation where buses are tied. Selective protection, which opens only the breaker nearest the fault, keeps it to the faulted bus, and condition monitoring gives the early warning (Chapter 4.2, Chapter 4.4, Chapter 4.6, Chapter 14.5).
• F-R16 · Switching or maintenance error. A wrong breaker or a skipped step in a method of procedure (MOP); the blast radius is whatever the operator is working on, and because maintenance usually takes one path out of service first, an error on the remaining path drops the load. The controls are a peer-reviewed MOP and switching order, a check that the redundant path is healthy before the first path comes out, and a second person confirming each breaker operation (Chapter 14.12).
• F-R17 · Fleet-wide firmware or configuration push. One bad image or setting reaches every UPS, PDU, BMS controller or rack running it at once. Push it one fault domain at a time after a canary, pin versions, and keep a rollback you have tested (Chapter 14.8).

Model F-R16 and F-R17 as common-cause events, not independent component failures: their reach is set by the procedure or the rollout, not by the hardware.

F-R02.1, normal duty: a pump bearing fails while the standby’s control and power survive independently. Losing flow exposes four racks to throttle or trip and the job to interruption. The budget is sensing 0.50 s + logic 0.50 s + standby to full flow 3.0 s + 1.0 s allowance = 5.0 s, leaving 1.0 s before the assumed 6.0 s limit. If the standby takes 4.0 s, the total reaches 6.0 s, equal to the limit, and the case fails because flow must be back before the limit. Shorten recovery or change the protection or thermal design, then retest.

Neither result is hardware acceptance. Get the rack’s allowable interruption from the OEM and test the whole response; the assumed six-second limit authorizes no live pump-drop test. The commissioning lead records the approved test and equipment IDs, heat load, inlet state, minimum worst-branch flow, timestamps, uncertainty and restart results. Accept only when the required flow is back before the limit and temperatures stay inside the OEM’s limits, and report lost work separately from service restoration. F-R02.2, shared controller lost: when one controller runs both pumps, its failure disables both trains, so the standby earns no credit and the four-rack consequence stands. The design lead fixes the independence; the commissioning lead keeps acceptance open until signed test results show the protection works. Chapter 5.11 owns hydraulic protection, Chapter 13.6 test authorization, and Chapter 12.5 the reliability model these results feed.