Level 5 Integrated Systems Testing (IST) & Failure-Mode Demonstration

Every level of commissioning before this one tested a subsystem in isolation against its own design intent: the switchgear in Chapter 13.3, the generators and microgrid in Chapter 13.4, the cooling plant and CDUs in Chapter 13.5. Level 5 Integrated Systems Testing (IST) verifies the integrated facility against the defined design-basis scenarios. Execute each scenario by the approved safe method—controlled switching, secondary injection, HIL, surrogate load, or an isolated test scope—with named preconditions, witnesses, hold points, abort limits, expected results, and restoration. IST is not authority to improvise arbitrary concurrent faults or place revenue-bearing GPUs at risk.

And it is built on a lie of convenience. IST is run against load banks — heaters that consume the design power so the building has something to cool and power while you trip its redundancy. Load banks are how you can pull the plug on 100 MW without owning 100 MW of irreplaceable accelerators. But a load bank is a fundamentally different electrical and thermal object than a GPU cluster. It draws a smooth, steady, controllable load; a GPU fleet draws a violent, synchronized, millisecond-scale sawtooth. That gap gets its canonical treatment here. After the IST master sequence and the failure-mode demonstration, the sections below set out what each load-bank technology can and cannot reproduce, across both the electrical and the thermal dimensions, and how a staged, production-representative run in Chapter 13.9 closes workload-specific normal-operation evidence that facility emulators cannot fully reproduce.

IST does not begin until every feeding subsystem has its own L4 acceptance signed and its baseline fingerprint captured (Chapter 13.2). The reason is diagnostic, not bureaucratic: if a cooling pump fails during defined-boundary loss-of-source or transfer evidence, you must already know it passed its standalone test, or you cannot tell whether IST found an integration fault or merely an un-commissioned component. IST tests seams, not parts. Sequencing it after the parts pass is what makes a failure during IST interpretable.

The master sequence is a planned escalation. It typically runs steady-state proving first (the building holds design load at design conditions for a sustained soak), then single-fault scenarios (lose one utility feed, one generator, one UPS module, one CDU), then approved utility-loss and system-transfer evidence at a defined boundary, using simulation/HIL, secondary injection, staged artificial load, and only where specifically justified a controlled live-source interruption — then the approved single-fault and specifically justified combined-fault matrix that proves the redundancy topology actually delivers the tier it was sold as. Planning a full-facility IST takes weeks to months; the execution window for a hyperscale hall is commonly a continuous multi-day campaign so that thermal soak and battery-recharge behavior are observed across real time, not inferred. The deliverable is a witnessed, time-synchronized data record of how the integrated building behaved at every transition, not a pass/fail stamp.

A redundancy claim is a claim about concurrent failures, and IST is where it is cashed. Losing one feed is the easy case; a 2N electrical topology is proven by demonstrating, against a protected load bank, the specific concurrent-failure combinations the design actually guarantees — each drawn from the facility FMEA with preconditions, instrumentation, expected result, and abort criteria. 2N promises survival of the design-basis events, not of an arbitrary second fault improvised on the live survivor; the hidden single points of failure — two nominally independent systems sharing a breaker, a controller, a cable tray, or a cooling loop — are found by tracing the as-built topology and by relay secondary injection, not by faulting the path that is carrying the load. The cascading-fault matrix is built directly from the facility FMEA (the consolidated catalog lives in Appendix F): each high-severity failure mode becomes an IST scenario, executed in the order most likely to surface a shared dependency.

For AI factories, the dominant integration risk is no longer electrical — it is thermal ride-through, the seam that did not meaningfully exist in legacy IT. In an air-cooled hall, a brief cooling lapse during power transfer is absorbed by the thermal mass of the room and the air; you have minutes. In a direct-to-chip liquid hall, the thermal mass at the die is almost nothing. A GB200-class rack accepts coolant across the W45-class envelope (up to ~45 °C supply / ~65 °C return; ~25 °C was NVIDIA's launch operating example, not a floor); excursions beyond that envelope throttle the GPUs by up to ~50%, and a sustained loss of flow requires the selected product's transient response to be established from OEM data, controls tests or an engineering calculation. So the IST question is specific: during the worst-case power transition, does coolant flow and temperature stay inside the envelope, or does the cluster throttle or trip? Critical cooling protection and re-energization must be sized and validated against the selected transient response and failure case; no universal protected-bus topology or timing is implied. IST is where you measure that gap against a stopwatch, with a load bank standing in for the heat the GPUs would have produced. This is the explicit interlock between facility Cx and cluster burn-in flagged in Chapter 13.5.

The control and monitoring stack is one of the things under test, not a witness to IST. Three layers must be proven to agree: the BMS (mechanical/electrical building automation), the SCADA / power-management system (switchgear, generators, the microgrid controller of Chapter 13.4), and the DCIM that operations will actually watch on day 2 (Chapter 14.2). The classic IST finding is not a hardware failure at all — it is that the building did the right thing while the DCIM displayed the wrong thing, or raised forty alarms for one event, or missed the event entirely because a Modbus/BACnet mapping was transposed during integration. An alarm flood is itself a failure: an operator who cannot find the root-cause alarm under a cascade of consequential ones will mis-diagnose the next real incident. IST validates the alarm hierarchy, the automatic control sequences (not just manual operation), and the point-to-point mapping from physical sensor to operator screen — the data path that the entire day-2 reliability program (Chapter 14.1) is built on top of.

The central limitation of every IST ever run: the load you test against is not the load you will run. A load bank exists to consume power and reject heat on command. A GPU cluster running a synchronous training step does something a load bank was never built to do — it swings its entire draw, in lockstep, across thousands of accelerators, on the cadence of the collective-communication pattern. When the all-reduce stalls compute, tens of megawatts can fall in milliseconds; when compute resumes, it returns just as fast. This is the transient physics made canonical in Chapter 4.5, and it is the reason the GB300 NVL72 ships with ~65 J/GPU of in-shelf energy storage to smooth a ~30% peak-grid reduction, and why facility BESS designs add their own power-smoothing role. IST poses it directly: which load-bank technology do you commission against, and therefore which of these dynamics do you leave un-demonstrated?

The table is a ladder of fidelity bought at rising cost and risk. Resistive load banks are cheap, ubiquitous, and prove the steady-state power and cooling capacity — they are the right tool for the bulk of L4 and the soak portion of IST. Reactive banks add power-factor and ripple stress that resistive banks miss, exercising the UPS, the generator AVR, and protection devices closer to real conditions. Dynamic / AI-emulating banks — transistor-based or, for 800 VDC architectures, programmable DC loads — are the newest tier and the only load-bank class that even attempts the millisecond step-load that defines AI draw; they let you fire a scripted swing at the BBU and BESS mitigation stack and watch it absorb. But every load bank shares one fatal limitation for liquid-cooled facilities: almost all of them reject heat to air, not into cold plates. The liquid loop, the CDU control response, and the worst-case-branch thermal-hydraulics — the things Chapter 13.5 flagged as un-testable without real silicon — stay un-exercised at realistic transient heat flux no matter how good your electrical emulation is. You can perfect the electrical side of the lie and the thermal side remains unproven.

Modern AI facilities defend against GPU transients with a layered stack, and IST is where you decide how much of it you exercise on the real building versus accept on vendor witness test. The layers, from grid inward: the facility-level BESS (and any synchronous condenser) absorbing multi-megawatt swings and supporting ride-through (Chapter 4.5, Chapter 13.4); the BBU / UPS layer bridging power transfers; the in-rack / in-shelf capacitance (GB300's ~65 J/GPU, Vera Rubin's larger reservoir) catching the fastest edges; and the workload-side power smoothing — ramp-rate limiting and power capping via SMI/Redfish — that shaves the peak before it reaches the wire. The problem is recursive: a static load bank cannot create the transient the stack exists to absorb, so an IST run on resistive or reactive banks proves the stack is installed and healthy but never proves it does its job. Only a dynamic load bank (partially) or a real workload (fully) closes that loop. That leaves an explicit risk allocation: which mitigation layers you require to be demonstrated absorbing a real swing during IST, and which you sign off by analysis and recharge-test, knowing the first true exercise comes during the proxy run.

Not every failure mode in Appendix F can or should be physically tripped on the real building. Some are too destructive (you do not deliberately rupture a coolant line on a live hall to watch the leak-detection-and-isolation sequence), some are impractical (you cannot induce a real utility-side 230 kV fault on demand), and some are covered acceptably by a witnessed factory or vendor test plus an installed-and-healthy check. IST forces an explicit triage of the FMEA catalog into three buckets, and the allocation is a risk decision the owner signs, not the Cx agent.

The table's second column is the uncomfortable one. Thermal ride-through, cooling failover, and the alarm hierarchy can be tripped live — but only against load banks, so they land in the 'medium residual risk' bucket no matter how rigorously you run them. That is the structural consequence of the realism gap: the failure modes most specific to AI factories are precisely the ones a load bank can demonstrate against a fault but not against the workload. That is why IST acceptance criteria cannot be a clean binary; they have to read as a bridge.

A defensible IST acceptance package does three things the legacy IT version never had to. First, it states what was proven and against what load — every passed scenario annotated with the load-bank class used, so the residual-risk register is honest about which results carry the realism caveat. Second, it defines the instrumented thresholds that gate GPU energization: coolant temperature and flow stability under the worst observed transition, critical-bus continuity with zero dropped cycles, CDU re-energization time inside the throttle window, and a clean, de-duplicated alarm record. Third, it names the deficiencies knowingly carried into the proxy run — the thermal-hydraulic dynamics and workload-synchronized electrical behavior that no load bank reached — so the proxy run is explicitly chartered to close them rather than being treated as a victory lap.

• Gate to energize GPUs: approved utility-loss, transfer, and fault-set evidence accepted at the stated test load and boundary; thermal ride-through measured inside the coolant envelope; critical cooling confirmed on protected bus with re-energization time bounded; BMS/DCIM/SCADA agreement and a clean alarm hierarchy demonstrated.
• Carried to the proxy run (Chapter 13.9): CDU and secondary-loop control response to real transient heat flux; the cross-rack-synchronized power swing against the full mitigation stack; worst-case-branch thermal-hydraulics under realistic, not air-rejected, load.
• Carried to staged ramp (Chapter 13.10): validation of power-smoothing and ramp-rate-limit configuration against the actual model, preserving live-block redundancy as load builds.

The acceptance gate is therefore not 'IST passed' but 'IST passed, residual risk named, and the conditions for safely meeting the real workload are written down.' That handoff — from a building proven against faults to a building about to meet its first real dynamics — is the seam IST exists to make safe.