Continuous & Re-Commissioning on a Live Campus

The commissioning program in Part 13 ended at handover. The L5 integrated systems test passed, the approved utility-loss and system-transfer evidence was accepted, the proxy training run hit its goodput gate, and operations took the keys. That was the moment the building was proven. It was also the moment the proof started to expire. A legacy enterprise hall drifts slowly; an AI campus mutates continuously. Racks get denser, CDUs get re-piped, firmware gets pushed fleet-wide, a UPS module gets RMA'd, a fault domain gets re-cabled, and a setpoint gets nudged to chase PUE. Each change invalidates some slice of the commissioning evidence, and none announces it. Re-commissioning is the discipline of re-proving the building against its own design intent, on a cadence and on triggers, while protecting operating load and people.

It runs under a constraint the original Cx team never faced: you cannot casually take the factory down to test it. The original IST ran in an empty building with load banks and a sacrificial proxy run. The re-test happens on an operating campus, so the evidence comes first from test sources, injection, simulation and empty blocks. That reorders everything: what you test, how you isolate it, when you accept risk, and what you decline to demonstrate because the demonstration is more dangerous than the failure it would reveal.

A commissioning result says "under these conditions, this system behaves this way." It stays true exactly as long as the conditions hold, and on a frontier campus they never hold for long. Three forces erode the proof, at very different rates.

Physical drift is the slow one. Coolant chemistry degrades and biofouls the cold plates, quick-disconnects seep, UPS and BESS cells age and lose ride-through margin, generator fuel polymerizes, CRAH coils foul, and breaker contacts and bus connections develop resistance. None of this shows on a single-line diagram, and all of it moves real failover and ride-through behavior away from the commissioned number. Classic retro-commissioning was invented for this regime. LBNL's meta-analysis of existing-building commissioning found a median 16% whole-building energy saving and a 1.1-year payback (Mills, 2009). But that program was built for buildings that barely change, not for ones that double in density every other year.

Configuration drift is the fast one and the dangerous one. Every change ticket moves the system off the commissioned configuration: a firmware bump, a setpoint edit, a re-cabled rail, a swapped PDU, a BMS logic tweak to silence a nuisance alarm. Each is trivial alone. After eighteen months of a live campus, the running configuration and the as-commissioned one have diverged so far that the original test evidence describes a building that no longer exists. The Uptime Institute's outage analysis keeps finding the same thing: power is the most commonly reported primary cause of impactful outages, human error is a major contributing factor operators can control, and within human error, ignored or inadequate procedures dominate. A change made outside its procedure is also a change that never reached the commissioning baseline.

Step changes are the violent ones. A density step-up from one accelerator generation to the next does not drift the building off its proof. It invalidates the evidence whose electrical, thermal, structural or control boundary changed, and unchanged evidence transfers only after an explicit applicability check. A hall commissioned for 40 kW air racks is a different thermal and electrical machine once it hosts 132 kW HPE GB200 NVL72 racks, GB300 NVL72 positions sized to the OEM's peak or busway basis (up to 142 kW in NVIDIA's reference architecture, 155 kW peak at Lenovo), or Vera Rubin NVL72 cabinets at the 330 kW TDP that NVIDIA's DSX reference design uses as its power and cooling basis (OEM operating profiles 188 kW Max Q, 228 kW Max P). The same holds for a Vera Rubin hall being prepared for the ~600 kW Rubin Ultra / Kyber roadmap point, which NVIDIA targets for H2 2027 and SemiAnalysis reported as 2028 in July 2026. The original IST's loss-of-source and transfer evidence covers only the loads it was tested at; unchanged protection, component and route evidence can still apply after the change-impact review. → Chapter 5.1 on the density wall the step crosses.

Three integrated capabilities decay after handover, and each is re-proved on its own clock.

Loss-of-source and transfer is the utility-to-generator handoff plus the BBU/BESS ride-through that bridges it. It decays with cell aging, fuel degradation and setpoint drift, and its calendar is the one most plants already run.

Under NFPA 110, emergency and legally required sets at Level 1 and Level 2 must run under load every month. Operators, insurers or contracts often extend that to optional-standby IT gensets. The 2019 edition (§8.4.2) sets 30 minutes at not less than 30% of standby nameplate kW, or at the manufacturer's recommended minimum exhaust-gas temperature. Where the monthly run cannot reach that load, add the annual supplemental load-bank test on the sequence the adopted edition prescribes (2019 §8.4.2.3: not less than 50% for 30 continuous minutes, then not less than 75% for 1 continuous hour, 1.5 hours total). At least once every 36 months, Level 1 sets must also run continuously for the full duration of their assigned class; a class over 4 hours can end the run at the 4-hour mark (2019 §8.4.9). It is the only routine test that proves fuel delivery and genset cooling over a long run, so schedule it for the IT sets that carry the campus even where the code does not require it. Lead-acid strings get IEEE 450/1188 capacity tests. Lithium BBU/BESS gets the OEM state-of-health protocol, which those standards do not cover. These clause numbers come from the 2019 edition. The 2025 edition is now current, so check which one your AHJ has adopted before you write the calendar.

Redundancy failover is the check that the N+1 and 2N paths still engage on demand. It decays through stuck static-transfer switches, isolation valves left shut after maintenance, and control cards that drifted. Re-prove it per fault domain on a rolling calendar and after every change that touched the path.

Thermal ride-through is how long the liquid loop holds temperature when a CDU trips or a pump fails at real heat flux. It is the AI-specific one. Go-live proved the liquid path with liquid-cooled load banks or emulator racks on the manifolds and confirmed die-level behavior on the staged proxy run (→ Chapter 13.5; Chapter 13.6). An air-rejecting facility load bank never reached the cold plates, so a re-test that falls back to one proves nothing about the loop. Re-prove it after any material change to the loop and at every density step.

The hard part is running any of them without taking the factory down. One rule governs all three: never re-test the whole campus; re-test one block, one fault domain, one concurrent-maintainability boundary at a time, while the rest of the campus carries load and stands ready to absorb the block under test if it fails for real. Inside that boundary, choose the least hazardous method that still produces valid evidence, and climb the ladder only when the rung below cannot answer the question: inspection and functional checks; relay and secondary injection; simulation or hardware-in-the-loop against the commissioning fingerprint; staged transfer on a test source or load bank; an empty or emulator block for thermal work; and, last, a bounded live-load test under a specific hazard and business-risk approval with named hold points and abort limits. Interrupting a live utility source to prove a handoff is the top of that ladder, not the default. The stakes have inverted since go-live: if the ride-through or the generator start now falls short, you do not lose a test, you drop a live training run and breach an SLA.

Meta's published 'PowerLoss Storm' program shows how to take the top rung on purpose. Meta started in pre-production and shadow regions, moved to its smallest production regions, and reached large regions hosting storage, AI and data-warehouse workloads last. Every stage ran under three limits: no lost data, no permanent damage to the facility, and no sustained impact outside a single region. The runs found recovery dependencies. In one case the power-loss shutdown signals also stopped the control plane that was supposed to orchestrate recovery (Meta Engineering, June 2026).

The thermal re-test needs one more sentence because it is the one where the ladder's bottom rungs are weakest. A pump-drop simulation against the commissioning fingerprint tells you what the controls will do. Only product-representative heat, from an emulator rack or a block between tenants, tells you what the silicon will do. A density step changes the heat duty, and a pump, topology or control change can matter just as much. Plan the thermal re-test into the refresh, on the empty block, before the new racks go live.

Re-commissioning fires on triggers, and the triggers are not equal. Their rank sets the scope of re-validation, the sign-off authority required, and how much live load you are willing to put at risk for the proof. Three dominate, in descending order of how much original commissioning evidence they invalidate. Re-rank by the change-impact matrix when a topology or control change reaches further than the density step.

1 — Density step-up (highest risk). A new accelerator generation that raises rack power can invalidate evidence across several systems at once. It changes the electrical and thermal machine together, at the rack scale where the original proof was most specific. The power chain commissioned for 132 kW racks now sees a different load profile, a different transient (NVL72-class power swings and the BBU→BESS→GPU-capacitance smoothing stack), different heat flux, different CDU branch loading and different floor mass. Retain original IST evidence for unchanged, bounded conditions, and list precisely which tests the new load profile invalidates. Where the changed load goes beyond what was proved, the impact matrix can require broad affected-block re-IST: re-validate the power chain transient behavior (→ Chapter 13.3 electrical acceptance), re-accept liquid duty and flush only where the changed wetted boundary, chemistry or procedure requires it (→ Chapter 13.5), re-run the thermal ride-through, and re-baseline goodput. This is why Chapter 1.1 calls the density ramp the most expensive irreversible mistake of the era. It is also why floor, water and electrical headroom had to be provisioned at scoping time: re-commissioning cannot create headroom that was never built.

2 — Topology change (high risk). Re-cabling a fault domain, splitting or merging blocks, re-piping a cooling distribution, changing a redundancy boundary, or repurposing a hall changes which failure is contained where, and containment is the whole point of the redundancy topology. The original commissioning proved that a specific set of fault domains held. A topology change can silently merge two domains that were supposed to be independent, so one fault now takes both. Re-run the failover tests across every boundary the change touched, and re-baseline the documentation so the as-built single-lines match reality. Skip that and the redundancy claim is paper-true but physically false: the diagram says independent, the copper says shared.

3 — Equipment replacement (managed risk). Swapping a UPS module, a CDU, a generator, a PDU or a pump under RMA is the most frequent trigger and the most procedural. The replaced unit was never commissioned in this building, and the manufacturer's factory test is not your site acceptance. Require a unit-level re-acceptance (the L3/L4 script for that component) plus a re-test of the redundancy path it sits in, because the swap is itself a configuration change that can leave an isolation valve shut or a control setpoint at default. A bounded equivalent replacement can justify a component and path retest; changed ratings or control dependencies can demand wider tests and higher authority. Release the unit to service only if the change-management discipline (→ Chapter 14.12) actually enforces the re-test, rather than treating the swap as done when the unit powers on.

Triggering re-commissioning on the calendar alone is necessary but not sufficient. It catches slow physical drift and misses the fast configuration drift that does the real damage between scheduled tests. The mature program closes a loop. The DCIM, telemetry and observability stack that runs the campus (→ Chapter 14.2) continuously compares live configuration and behavior against the commissioning fingerprint, the baseline captured at go-live (→ Chapter 13.2), and fires a re-validation trigger when the gap crosses a threshold.

The loop has four stages. Baseline: the as-commissioned fingerprint, covering power-chain transient signatures, CDU flow/thermal curves, fabric BER and timing, redundancy-path states and firmware inventory. Observe: the live telemetry stream, including firmware/software lifecycle state from fleet management (→ Chapter 14.8). Diff: the automated comparison that flags drift off the fingerprint, such as a setpoint that no longer matches the SOO, a firmware version that diverged from the approved baseline, a failover path not exercised within its window, or a thermal margin that has eroded. Trigger: the diff escalates into a re-test scope sized to what drifted. A firmware divergence triggers a canary soak, an eroded thermal margin triggers a ride-through re-test, and an unexercised path triggers a failover re-test.

The threshold is the design choice. Too tight, and you re-commission constantly, putting live load at risk for noise and burning out the operations team. Too loose, and you re-discover the gap as an outage. The right setting is workload-aware: checkpointable training and latency-bound inference can set different service-performance thresholds and outage budgets, but neither may loosen protection, OEM, chemistry, dew-point or personnel-safety limits. Price replayed GPU-hours for checkpointable training, and missed latency-SLA responses, service credits and lost contribution for serving. → the goodput-vs-availability framing in Chapter 12.2.

Re-commissioning does not live alone. It is one of three recurring proof activities that share the same machinery and should be planned together; the other two are disaster-recovery drills and FMEA-driven failure-mode demonstration.

DR drills (→ Chapter 12.3) are re-commissioning at campus-and-region scale. A failover drill that moves workload from one site to another, or exercises geographic redundancy, tests the same kind of claim as defined-boundary loss-of-source and transfer evidence ("the redundant path engages on demand") at a larger fault domain. Co-scheduling them is efficient. Do it only when the receiving site's own power evidence is current, because a DR failover that lands on a site whose generator handoff has drifted proves nothing.

The FMEA catalog (Appendix F) is the master register that decides what gets demonstrated, how, and how often. Every failure mode in it needs a re-commissioning treatment, and assigning one is the operator's first pass over the register. Appendix F supplies the mode, trigger, propagation path, detection, blast radius, mitigation, recovery and owner. You add the columns for your own plant: treatment class, retest interval, EOP identifier and evidence reference. The treatment classes are live-demonstrable on a single block (re-test it on a schedule), sub-scale or next-empty-block demonstrable (prove it before that block goes live), and model-only (too dangerous to demonstrate on live load, so prove it analytically and through the quantitative reliability model in Chapter 12.5). Where a random failure and an adversarial abuse share a physical consequence, the catalog links them, but the evidence differs: a fault test proves recovery, while an adversarial test must also prove authorization, detection and containment (→ Chapter 11.10). The demonstration line drawn earlier governs here too: demonstrate what you safely can on live load, model what you cannot, and never let a checklist push you across that line on a running factory.

Returning a CDU to service after isolated maintenance is re-commissioning at the scale of one unit. Chapter 14.5 works one case, a maintenance package stopped at desk review because the OEM manual gave the same valves conflicting identities. The first release is on paper: reconcile the OEM procedure, the site P&ID and the physical tags, then approve the work package under Chapter 14.12. After the work, keep the fluid record and inspect the repaired boundary for leaks. Then prove flow, pressure, temperature sensing, alarms and the affected failover path against the approved profile in Chapter 13.5; temperature sensors that agree with each other do not prove hydraulic capacity. The ASHRAE–PNNL–NEMA AI Data Center Energy Performance Framework (June 2026) gives guidance on coordinating this evidence but sets no test requirement. Record what passed, and which unchanged evidence you kept, before the unit returns to normal duty.