Level 5 Integrated Systems Testing (IST) & Failure-Mode Demonstration

Every earlier commissioning level tested a subsystem alone against its own design intent: the switchgear in Chapter 13.3, the generators and microgrid in Chapter 13.4, the cooling plant and CDUs in Chapter 13.5. Level 5 Integrated Systems Testing (IST) tests the whole facility against the design-basis scenarios. Each scenario runs as a script: preconditions, witnesses, hold points, abort limits, expected result and restoration. Each design-basis fault needs a test method that can actually prove it. Installed controlled tests on load banks, injection/HIL and verified analysis cover different links, and none can stand in for an untested installed interface. IST is not authority to improvise concurrent faults beyond the design basis, or to put revenue-bearing GPUs behind the test.

And it is built on a lie of convenience. IST runs against load banks: heaters that consume the design power, so the building has something to power and cool while you trip its redundancy. That is how you can pull the plug on 100 MW without owning 100 MW of irreplaceable accelerators. But a load bank is a different electrical and thermal object from a GPU cluster. A bank held flat draws a smooth, steady, controllable load, and switched or programmable banks add bounded transients. A GPU fleet supplies a software-synchronized waveform on a millisecond scale. This chapter is the canonical home for that gap: what each load-bank technology can and cannot reproduce, electrically and thermally, and how a staged, production-representative run (Chapter 13.9) covers the workload-specific normal-operation behavior that facility emulators cannot fully reproduce.

IST does not begin until every feeding subsystem has its own L4 acceptance signed and had its baseline fingerprint captured (Chapter 13.2). The reason is diagnostic, not bureaucratic. If a cooling pump fails during a loss-of-source or transfer test at the defined boundary, you must already know it passed its standalone test. Otherwise you cannot tell an integration fault from an uncommissioned component. IST tests seams, not parts, and running it after the parts pass is what makes a failure interpretable.

The master sequence is a planned escalation. It typically starts with steady-state proving: the building holds design load at design conditions for a sustained soak. Single-fault scenarios follow: lose one utility feed, one generator, one UPS module, one CDU. Next comes the utility-loss transfer chain. The feed is opened at the approved boundary with the building on load banks, and the UPS/BESS pickup, generator start and mechanical-plant transfer are all timed. Last is the combined-fault matrix the design basis specifies, which proves the redundancy topology delivers the tier it was sold as. Build the IST window from fixture mobilization, required states, witness availability, thermal stabilization, battery recharge and retest dependencies. Observe the durations on the accepted equipment rather than taking them from a generic weeks-to-months allowance. The deliverable is a witnessed, time-synchronized data record of how the integrated building behaved at every transition, not a pass/fail stamp.

Attach a capability receipt to each rung: fixture serial and configuration; kW and kVA/PF; step size, slew, repetition and harmonic spectrum; liquid/air heat split; branch placement; control bandwidth; instrument uncertainty; and the exact failure interface injected. A load-step claim without a measured waveform is as weak as a power rating without a terminal. Avtron's generator load-testing method includes switched-load testing, and Chapter 13.5 applies the independent liquid-path requirements. A missing capability means another fixture or an explicit hold, never an unseen test accepted on its label.

A resilience claim holds only for its design-basis states, and IST is where it is cashed in. A concurrently maintainable design is proved by taking each path out of service in turn with the load held. A fault-tolerant design is proved by tripping each single fault with the load held. The combined faults the design basis names are tripped in the order most likely to expose a shared dependency; a utility loss during a planned maintenance isolation is the classic. 2N promises survival of the design-basis events, not of an arbitrary second fault improvised on the surviving path. Hidden single points of failure, such as two nominally independent systems sharing a breaker, a controller, a cable tray or a cooling loop, are hunted by tracing the as-built topology and by relay secondary injection. They are then confirmed by the approved scripted trip where authorized, or by explicitly scoped injection/HIL and installed-interface checks, with any untested response left open. The cascading-fault matrix comes straight from the facility FMEA (the consolidated catalog is in Appendix F): each high-severity failure mode becomes an IST scenario.

For AI factories, the new integration seam is thermal ride-through, which barely existed in legacy IT. In an air-cooled hall, a brief cooling lapse during power transfer is absorbed by the thermal mass of the room and the air, so you have minutes. In a direct-to-chip liquid hall, the thermal mass at the die is almost nothing. A GB200-class rack accepts coolant inside the limits its vendor documentation gives. QCT's QoolRack reference sets two separate maxima, ~45 °C at the liquid inlet and ~65 °C at the liquid return; take the operating point from the product documentation (Chapter 13.5). ASHRAE W45 describes facility-water supply capability, not a product setpoint. Past those limits the GPUs throttle, and a sustained loss of flow reaches the throttle in seconds. How deep the throttle goes and how long an excursion may last depend on the specific server, coolant profile and workload. Read them from the product documentation and measure them in the approved test instead of carrying a generic percentage into the script. The IST question is therefore specific: during the worst-case power transition, do coolant flow and temperature stay inside the limits, or does the cluster throttle or trip? Critical cooling rides on the protected bus, and its re-energization time is measured against that window, not assumed. IST is where you measure that gap with a stopwatch, with a load bank standing in for the heat the GPUs would have produced. This is the interlock between facility Cx and cluster burn-in flagged in Chapter 13.5.

The control and monitoring stack is itself under test, not a witness to IST. Three layers must be proven to agree: the BMS (mechanical and electrical building automation), the SCADA / power-management system (switchgear, generators, and the microgrid controller of Chapter 13.4), and the DCIM that operations will actually watch on day 2 (Chapter 14.2). The classic IST finding is not a hardware failure at all. The building did the right thing while the DCIM displayed the wrong thing, raised forty alarms for one event, or missed the event because a Modbus/BACnet mapping was transposed during integration. An alarm flood is itself a failure, because an operator who cannot find the root-cause alarm under a cascade of consequential ones will mis-diagnose the next real incident. IST validates the alarm hierarchy, the automatic control sequences (not just manual operation), and the point-to-point mapping from physical sensor to operator screen. The entire day-2 reliability program (Chapter 14.1) is built on that data path.

The central limitation of fixture-based IST: the load you test against is not the load you will run. A load bank consumes power and rejects heat on command, and that is all it was built to do. A GPU cluster running a synchronous training step swings its entire draw in lockstep across thousands of accelerators, on the cadence of the collective-communication pattern. When the all-reduce stalls compute, tens of megawatts can fall in milliseconds, and when compute resumes the load returns just as fast. Microsoft's POLCA study (ASPLOS 2024) measured the pattern in production: its training rows peaked at 97% of provisioned power, leaving about 3% headroom, and swung by up to 37.5% of provisioned power within 2 seconds, against 9% for inference. A load bank held flat reproduces that peak and none of that swing. Chapter 4.5 is the canonical treatment of this transient physics. It is the reason the GB300 NVL72 adds ~65 J/GPU of in-shelf energy storage and why facility BESS designs take on a power-smoothing role. In July 2025, NVIDIA reported up to ~30% lower peak power on its instrumented GB200/Megatron demonstration of these features. IST therefore has to answer: which load-bank technology do you commission against, and so which of these dynamics do you leave undemonstrated?

The table compares independent capabilities, not a ladder from resistors to perfect GPUs. Price the missing waveform or heat path instead of buying a technology label. Resistive load banks are cheap and ubiquitous, and they prove steady-state power and cooling capacity, so they are the right tool for most of L4 and the soak portion of IST. Reactive banks add the power-factor and ripple stress that resistive banks miss, exercising the UPS, the generator AVR and protection devices closer to real conditions. Dynamic / AI-emulating banks (transistor-based or, for 800 VDC architectures, programmable DC loads) are a programmable way to impose the required step-load profile. Like switched resistive and reactive banks, their measured bandwidth decides which edges they can reproduce, and they let you fire a scripted swing at the BBU and BESS mitigation stack and watch it absorb. Air-cooled electrical load banks leave the technology-cooling system unexercised. Liquid-cooled load banks and product-representative emulator racks validate the thermal-hydraulic loop before production GPUs arrive. A staged workload then supplies the hardware- and workload-specific normal-operation results.

Modern AI facilities defend against GPU transients with a layered stack, and IST is where you decide how much of it to exercise on the real building and how much to accept on vendor witness test. From the grid inward, the layers are the facility-level BESS (and any synchronous condenser), which absorbs multi-megawatt swings and supports ride-through (Chapter 4.5, Chapter 13.4); the BBU / UPS layer, which bridges power transfers; the in-rack / in-shelf capacitance (GB300's ~65 J/GPU, Vera Rubin's ~400 J/GPU), which catches the fastest edges; and workload-side power smoothing, meaning ramp-rate limiting and power capping via SMI/Redfish, which shaves the peak before it reaches the wire. The problem is recursive: a static load bank cannot create the transient the stack exists to absorb, so an IST run with resistive or reactive banks held flat proves only that operating point. Switched or programmable fixtures prove the stack's response within their measured range. Real workloads add the tested software-synchronized profile, never every possible future swing. Record each battery layer's starting state of charge and the source feeding it, because a BBU or BESS result holds only from that starting point. That leaves a risk allocation to make explicit: which layers you require to be demonstrated absorbing a real swing during IST, and which you sign off by analysis and recharge-test, knowing the first true exercise comes during the proxy run.

At least one dynamic scenario should run with the building on generator, not utility. The discipline-level gate for that is in Chapter 13.3. In a vendor lab test published in September 2026, Vertiv worked with ST Telemedia and Baudouin to replay H100 power swings onto a generator. Past about 75% AI load against total IT capacity, generator frequency swung too far for the UPS to synchronize to bypass. With 20% input smoothing, voltage and frequency variation dropped from about ±10% to ±0.5%. The threshold depends on the genset and its configuration, so treat bypass and re-transfer availability on generator, under the swing you specified, as an acceptance item. A flat load bank cannot show it.

Not every failure mode in Appendix F can or should be physically tripped on the real building. Some are too destructive: you do not rupture a coolant line on a live hall to watch the leak-detection-and-isolation sequence. Some are impractical, since you cannot induce a real utility-side 230 kV fault on demand. Some are covered acceptably by a witnessed factory or vendor test plus an installed-and-healthy check. IST forces an explicit triage of the FMEA catalog into four buckets. The allocation is a risk decision the owner signs, not the Cx agent.

Each failure mode in the table needs a coverage record that says what was actually exercised. Thermal ride-through, cooling failover and the alarm hierarchy can be tested with commanded disturbances on liquid banks and injected signals. Residual risk depends on the missing branch, waveform or control interface. That follows from the realism gap: the failure modes most specific to AI factories are the ones a load bank can demonstrate against a fault but not against the workload. Each required criterion still gets pass, fail or hold, and the acceptance package names the later test that clears each hold.

A defensible IST acceptance package does three things the legacy IT version never had to. First, it states what was proven and against what load: every passed scenario is annotated with the load-bank class used, so the residual-risk register shows which results carry the realism caveat. Second, it sets the instrumented thresholds that gate GPU energization: coolant temperature and flow stability under the worst observed transition, critical-bus voltage and timing within the terminal limits, CDU re-energization time inside the throttle window, and a clean, de-duplicated alarm record. Third, it names the deficiencies knowingly carried into the proxy run, meaning the thermal-hydraulic dynamics and workload-synchronized electrical behavior outside the fixtures' measured coverage, so the proxy run is chartered to resolve them rather than treated as a victory lap.

• Gate to energize GPUs: the utility-loss chain, transfers and design-basis fault set have passing results from approved methods at the agreed boundary; thermal ride-through measured inside the coolant limits; critical cooling confirmed on the protected bus with re-energization time bounded; BMS/DCIM/SCADA agreement and a clean alarm hierarchy demonstrated; the transfer and ride-through settings verified against the envelope that applies, and any grid-side response left untested listed in the residual-risk register.
• Carried to the proxy run (Chapter 13.9): CDU and secondary-loop control response to real transient heat flux; the cross-rack-synchronized power swing against the full mitigation stack; worst-case-branch thermal-hydraulics under realistic, not air-rejected, load.
• Carried to staged ramp (Chapter 13.10): validation of power-smoothing and ramp-rate-limit configuration against the actual model, preserving live-block redundancy as load builds.

So the acceptance gate is not 'IST passed' but 'IST passed, residual risk named, and the conditions for safely meeting the real workload written down.' That handoff, from a building proven against faults to one about to meet its first real dynamics, is the seam IST exists to make safe.

Buy the tests that cover the required source, branch and control states, and release only what they measured. A bigger load-bank invoice does not cover an unconnected liquid branch, and a successful GPU run does not clear an untested protective action. Choosing by what each test covers lets the staged workload begin with a finite residual list instead of the building's first experiment.