Level 5 Integrated Systems Testing (IST) & Failure-Mode Demonstration

Every level of commissioning before this one tested a subsystem in isolation against its own design intent: the switchgear in Chapter 13.3, the generators and microgrid in Chapter 13.4, the cooling plant and CDUs in Chapter 13.5. Level 5 Integrated Systems Testing (IST) is the first and last time the whole building is run as one machine and then deliberately broken. Its job is not to confirm that things work — that was L1 through L4 — but to confirm that when something fails, the response works: that the automatic transfer fires, the UPS rides through, the cooling does not lapse while power transfers, the BMS and DCIM see the event correctly, and the load survives. IST is the rehearsal of every bad day the facility will ever have, performed once, on purpose, before any revenue-bearing GPU is at risk.

And it is built on a lie of convenience. IST is run against load banks — heaters that consume the design power so the building has something to cool and power while you trip its redundancy. Load banks are how you can pull the plug on 100 MW without owning 100 MW of irreplaceable accelerators. But a load bank is a fundamentally different electrical and thermal object than a GPU cluster. It draws a smooth, steady, controllable load; a GPU fleet draws a violent, synchronized, millisecond-scale sawtooth. This chapter is the canonical treatment of that gap. After the IST master sequence and the failure-mode demonstration, it sets out what each load-bank technology can and cannot reproduce, across both the electrical and the thermal dimensions, and why the proxy training run of Chapter 13.9 is the only true emulator the building will ever see.

IST does not begin until every feeding subsystem has its own L4 acceptance signed and its baseline fingerprint captured (Chapter 13.2). The reason is diagnostic, not bureaucratic: if a cooling pump fails during a black-building test, you must already know it passed its standalone test, or you cannot tell whether IST found an integration fault or merely an un-commissioned component. IST tests seams, not parts. Sequencing it after the parts pass is what makes a failure during IST interpretable.

The master sequence is a planned escalation, not a single event. It typically runs steady-state proving first (the building holds design load at design conditions for a sustained soak), then single-fault scenarios (lose one utility feed, one generator, one UPS module, one CDU), then the marquee test — the black-building / pull-the-plug test, a hard, unannounced loss of all utility power with the facility at full load — then the concurrent and cascading-fault matrix that proves the redundancy topology actually delivers the tier it was sold as. Planning a full-facility IST takes weeks to months; the execution window for a hyperscale hall is commonly a continuous multi-day campaign so that thermal soak and battery-recharge behavior are observed across real time, not inferred. The deliverable is a witnessed, time-synchronized data record of how the integrated building behaved at every transition, not a pass/fail stamp.

A redundancy claim is a claim about concurrent failures, and IST is where it is cashed. A 2N electrical topology is not proven by losing one feed — that is the easy case. It is proven by losing one feed and then, while the building is running on the survivor, inducing a fault on the path that should still be carrying load, to confirm there is no hidden single point of failure where two nominally independent systems share a breaker, a controller, a cable tray, or a cooling loop. The cascading-fault matrix is built directly from the facility FMEA (the consolidated catalog lives in Appendix F): each high-severity failure mode becomes an IST scenario, executed in the order most likely to surface a shared dependency.

For AI factories, the dominant integration risk is no longer electrical — it is thermal ride-through, the seam that did not meaningfully exist in legacy IT. In an air-cooled hall, a brief cooling lapse during power transfer is absorbed by the thermal mass of the room and the air; you have minutes. In a direct-to-chip liquid hall, the thermal mass at the die is almost nothing. A GB200-class rack must hold coolant inlet near 20-25 °C; deviation throttles the GPUs by up to ~50%, and a sustained loss of flow risks the cold plates within seconds, not minutes. So the IST question becomes brutally specific: during the worst-case power transition, does coolant flow and temperature stay inside the envelope, or does the cluster throttle or trip? Critical cooling — CDU pumps, primary loop pumps, the heat-rejection path — must be on the protected (UPS/generator-backed) bus and must re-energize fast enough to bridge the gap. IST is where you measure that gap against a stopwatch, with a load bank standing in for the heat the GPUs would have produced. This is the explicit interlock between facility Cx and cluster burn-in flagged in Chapter 13.5.

The control and monitoring stack is one of the things under test, not a witness to IST. Three layers must be proven to agree: the BMS (mechanical/electrical building automation), the SCADA / power-management system (switchgear, generators, the microgrid controller of Chapter 13.4), and the DCIM that operations will actually watch on day 2 (Chapter 14.2). The classic IST finding is not a hardware failure at all — it is that the building did the right thing while the DCIM displayed the wrong thing, or raised forty alarms for one event, or missed the event entirely because a Modbus/BACnet mapping was transposed during integration. An alarm flood is itself a failure: an operator who cannot find the root-cause alarm under a cascade of consequential ones will mis-diagnose the next real incident. IST validates the alarm hierarchy, the automatic control sequences (not just manual operation), and the point-to-point mapping from physical sensor to operator screen — the data path that the entire day-2 reliability program (Chapter 14.1) is built on top of.

Here is the central limitation of every IST ever run, stated plainly: the load you test against is not the load you will run. A load bank exists to consume power and reject heat on command. A GPU cluster running a synchronous training step does something a load bank was never built to do — it swings its entire draw, in lockstep, across thousands of accelerators, on the cadence of the collective-communication pattern. When the all-reduce stalls compute, tens of megawatts can fall in milliseconds; when compute resumes, it returns just as fast. This is the transient physics made canonical in Chapter 4.5, and it is the reason the GB300 NVL72 ships with ~65 J/GPU of in-shelf energy storage to smooth a ~30% peak-grid reduction, and why facility BESS designs add their own power-smoothing role. The fork at IST is: which load-bank technology do you commission against, and therefore which of these dynamics do you leave un-demonstrated?

The table is a ladder of fidelity bought at rising cost and risk. Resistive load banks are cheap, ubiquitous, and prove the steady-state power and cooling capacity — they are the right tool for the bulk of L4 and the soak portion of IST. Reactive banks add power-factor and ripple stress that resistive banks miss, exercising the UPS, the generator AVR, and protection devices closer to real conditions. Dynamic / AI-emulating banks — transistor-based or, for 800 VDC architectures, programmable DC loads — are the newest tier and the only load-bank class that even attempts the millisecond step-load that defines AI draw; they let you fire a scripted swing at the BBU and BESS mitigation stack and watch it absorb. But every load bank shares one fatal limitation for liquid-cooled facilities: almost all of them reject heat to air, not into cold plates. The liquid loop, the CDU control response, and the worst-case-branch thermal-hydraulics — the things Chapter 13.5 flagged as un-testable without real silicon — stay un-exercised at realistic transient heat flux no matter how good your electrical emulation is. You can perfect the electrical side of the lie and the thermal side remains unproven.

Modern AI facilities defend against GPU transients with a layered stack, and IST is where you decide how much of it you exercise on the real building versus accept on vendor witness test. The layers, from grid inward: the facility-level BESS (and any synchronous condenser) absorbing multi-megawatt swings and supporting ride-through (Chapter 4.5, Chapter 13.4); the BBU / UPS layer bridging power transfers; the in-rack / in-shelf capacitance (GB300's ~65 J/GPU, Vera Rubin's larger reservoir) catching the fastest edges; and the workload-side power smoothing — ramp-rate limiting and power capping via SMI/Redfish — that shaves the peak before it reaches the wire. The problem is recursive: a static load bank cannot create the transient the stack exists to absorb, so an IST run on resistive or reactive banks proves the stack is installed and healthy but never proves it does its job. Only a dynamic load bank (partially) or a real workload (fully) closes that loop. The decision is explicit risk allocation: which mitigation layers you require to be demonstrated absorbing a real swing during IST, and which you sign off by analysis and recharge-test, knowing the first true exercise comes during the proxy run.

Not every failure mode in Appendix F can or should be physically tripped on the real building. Some are too destructive (you do not deliberately rupture a coolant line on a live hall to watch the leak-detection-and-isolation sequence), some are impractical (you cannot induce a real utility-side 230 kV fault on demand), and some are covered acceptably by a witnessed factory or vendor test plus an installed-and-healthy check. IST forces an explicit triage of the FMEA catalog into three buckets, and the allocation is a risk decision the owner signs, not the Cx agent.

The table's second column is the uncomfortable one. Thermal ride-through, cooling failover, and the alarm hierarchy can be tripped live — but only against load banks, so they land in the 'medium residual risk' bucket no matter how rigorously you run them. That is the structural consequence of the realism gap: the failure modes most specific to AI factories are precisely the ones a load bank can demonstrate against a fault but not against the workload. This is why the acceptance criteria for IST cannot be a clean binary. They must read as a bridge.

A defensible IST acceptance package does three things the legacy IT version never had to. First, it states what was proven and against what load — every passed scenario annotated with the load-bank class used, so the residual-risk register is honest about which results carry the realism caveat. Second, it defines the instrumented thresholds that gate GPU energization: coolant temperature and flow stability under the worst observed transition, critical-bus continuity with zero dropped cycles, CDU re-energization time inside the throttle window, and a clean, de-duplicated alarm record. Third, it names the deficiencies knowingly carried into the proxy run — the thermal-hydraulic dynamics and workload-synchronized electrical behavior that no load bank reached — so the proxy run is explicitly chartered to close them rather than being treated as a victory lap.

• Gate to energize GPUs: black-building and concurrent-fault matrix passed at full load; thermal ride-through measured inside the coolant envelope; critical cooling confirmed on protected bus with re-energization time bounded; BMS/DCIM/SCADA agreement and a clean alarm hierarchy demonstrated.
• Carried to the proxy run (Chapter 13.9): CDU and secondary-loop control response to real transient heat flux; the cross-rack-synchronized power swing against the full mitigation stack; worst-case-branch thermal-hydraulics under realistic, not air-rejected, load.
• Carried to staged ramp (Chapter 13.10): validation of power-smoothing and ramp-rate-limit configuration against the actual model, preserving live-block redundancy as load builds.

The acceptance gate is therefore not 'IST passed' but 'IST passed, residual risk named, and the conditions for safely meeting the real workload are written down.' That handoff — from a building proven against faults to a building about to meet its first real dynamics — is the seam IST exists to make safe.