Commissioning Fundamentals, Levels & Program Governance

Commissioning (Cx) is the discipline that refuses to take the design's word for it. Every earlier Part of this guide makes claims: that the power chain will ride through a utility dip, that the cold plates will hold the GPUs under sustained load, that the fabric will carry a non-blocking all-reduce, that the standby plant will pick up the building before the UPS runs flat. Commissioning is the structured process of generating the evidence that those claims are true under the conditions that matter, before the facility carries revenue load. It bridges the design Parts (4–12) and Part 14's day-2 operations. It is also the last point in the project where a latent integration defect costs hours to fix instead of a multi-day outage on a live cluster.

For a conventional enterprise data center this is a mature, well-codified ritual. For an AI factory it is not, and that gap is the subject of this Part. An AI facility is two machines pretending to be one. The facility (substation, switchgear, generators, UPS/BESS, CDUs, chillers, piping, BMS) is something the commissioning industry knows how to accept. The cluster (tens of thousands of GPUs, the scale-up and scale-out fabric, storage and the scheduler) is something the facility commissioning standards barely acknowledge. The two tracks run in parallel and share power and cooling. They must be interlocked at specific gates, or each signs off in isolation against a load the other half cannot deliver. The rest of this chapter sets out the levels, the two tracks, the roles and the governing documents that Part 13 builds on.

Mission-critical commissioning is a ladder of increasing integration, conventionally Level 1 through Level 5. Some programs add a Level 0 design review and a Level 6 post-occupancy or seasonal stage. Each rung tests a wider boundary than the last, and you do not climb to the next rung until the current one is signed and its deficiencies are closed. The rule is that integration faults are found at the lowest level at which they can possibly appear. A mis-wired CT caught in an L3 standalone test is a morning's rework. The same fault found at L5 during a black-building test invalidates the run and resets days of scripted work.

In the usual numbering, L1 is factory acceptance testing (FAT) of major equipment, witnessed at the vendor’s plant. L2 verifies delivery and installation on site: identity, wiring, settings and pre-energization checks. L3 is standalone start-up, with each system energized and run on its own. L4 is functional performance testing of each system under load, through every mode in its sequence of operations. L5 is the integrated systems test (IST), where all disciplines run together through the required failure scenarios. Programs label these rungs differently, so freeze the owner’s level names and integration boundaries in the contract. Each of the two parallel tracks (next section) has its own version of this ladder. Chapter 13.2 turns each rung into a quantitative script with explicit pass/fail gates, and Chapter 13.6 lists the test results that release each one.

Budget the whole sequence on both tracks. Uptime Institute’s 2026 AI in Practice series includes a paper on Level 4 and 5 commissioning, with an accompanying webinar. Both point to load banks built to mimic the swings in GPU power draw, proof that the liquid-cooling loops work, testing that spans IT and facility together, and a decommissioning plan written during Cx. Add those items to the scope before the schedule freezes. This guide adds one recommendation of its own: an independent witness for L5.

Two governance failures are expensive to recover from: pushing a discipline’s own failure tests into the integrated campaign, and cutting that campaign when the schedule slips. Require the UPS transfer and battery ride-through test results before the generator, cooling and BMS handoff test, and name the owner of each dependency in Chapter 13.6’s ladder. A generator can start and a cooling plant can ride through while the BMS still hands the load off in the wrong order. Calling the individual passes “good enough” pushes that integration risk onto day two.

The L5 test matrix follows from the resilience claim the facility was designed to meet. The two claims that matter are the Uptime Institute's concurrent maintainability (Tier III) and fault tolerance (Tier IV). They sound similar and are not, and commissioning is where the difference becomes physical.

Concurrent maintainability (Tier III) means any single capacity component or distribution path can be removed from service for planned maintenance with no impact on the IT load. The IST must prove that by taking down each path in turn, deliberately, while the load never notices. It does not require surviving an unplanned simultaneous fault. Fault tolerance (Tier IV) is the stronger claim: the facility survives any single unplanned worst-case failure, including the loss of an entire distribution path to fire or flood, with no load impact, while staying concurrently maintainable. Tier IV adds compartmentalization and continuous cooling requirements that Tier III does not, and the IST must demonstrate the automatic response to a fault, not just an orderly manual switchover.

The resilience target you committed to in design writes your failure-mode demonstration script. A Tier III program scripts orderly, one-path-at-a-time maintenance scenarios. A fault-tolerant-topology program scripts approved, risk-assessed scenarios for the required fault set, plus continuous-cooling ride-through, which makes for a materially larger, riskier, longer IST. Choosing the topology is a design and capital decision (Chapter 12.1). Commissioning then proves each claim with the safest valid method: inspection, secondary injection, hardware-in-the-loop, simulation, staged load banks, or a specifically approved live fault test.

What sets AI-factory commissioning apart is that two acceptance ladders run at once, on different schedules, owned by different organizations, under different standards, and coupled by shared physical resources.

The facility track (Chapters 13.3–13.6) is the classical mission-critical Cx program: utility energization, switchgear, generators, UPS/BESS, CDUs, chillers, piping and controls, ending in the L5 IST. The owner's CxA and the MEP trades run it. ASHRAE Guideline 0 and Guideline 1.1 govern it alongside Standard 202's process requirements, as do Uptime and BICSI 002. The cluster track (Chapters 13.7–13.9) is the IT validation program: fabric commissioning (BER, link-flap, topology, bandwidth and latency), GPU node burn-in and diagnostics, cluster-scale NCCL and collective acceptance, and a reference or proxy training run. The platform, SRE and ML-infra team owns it. Vendor reference architectures and deployment guides, benchmark methods (NCCL tests, MLPerf) and owner-written criteria specify it. Third-party GPU-cloud ratings such as SemiAnalysis ClusterMAX 3.0 (September 2026) make a useful checklist but are not an acceptance standard. The facility CxA typically touches none of this.

If the two tracks sign off in isolation, you have certified nothing useful. The facility track accepts power and cooling capacity using load banks, usually air-cooled resistive banks. Their step granularity, control bandwidth and air-side heat rejection set the load they can present. The cluster track accepts compute using real GPUs, which draw a violent, synchronized, switching load and reject heat into liquid. Neither track exercises the seam between them. The interlock gates are where that seam gets tested.

The middle row is the defining limitation of facility commissioning for AI. A conventional air-cooled resistive bank is a deliberately boring load: elements switched in steps, drawing steady current at unity power factor and rejecting heat straight to the room air. What a bank can and cannot reproduce depends on its step granularity, control bandwidth and heat-rejection path, not on the word 'resistive'. Liquid-loop rejection and fast, repetitive waveforms need a bank specified for them.

A frontier GPU cluster can exceed a conventional bank's profile. Tens of thousands of accelerators ramp from idle to full and back in milliseconds, in lockstep across a synchronous step. That imposes power transients and harmonic content a step-switched bank does not reproduce, and it dumps the heat into a liquid loop the load bank never touches. The facility track can prove the power chain is wired correctly and redundant. It cannot prove the chain absorbs the dynamic load swing of a real all-reduce, because an air-rejecting, step-switched load bank cannot generate that full waveform, and any other load bank covers only what its measured waveforms show. This is the dynamic-load realism gap. Its canonical treatment is in Chapter 13.6, and the transient physics behind it is in Chapter 4.5. The mitigation is sequencing: liquid-loop emulation followed by the staged proxy training run of Chapter 13.9 narrows the gap until only the production workload itself is left to test, which is why it must come before go-live, not after.

Commissioning is only as good as the requirements it tests against. Those live in a short chain of governing documents, and every acceptance script must trace back through it. Each document derives from the one before, so a pass/fail gate in an L5 script can be followed back to a stated owner intent.

• Owner's Project Requirements (OPR). The owner's intent in measurable terms: the availability target (and so the Tier or BICSI class), the density and ramp the building must accommodate, the environmental operating ranges, maintainability expectations, and the acceptance criteria the owner will hold the project to. Everything downstream answers the OPR. For an AI factory the OPR must also state the workload intent (training-shaped or inference-shaped, per Chapter 1.1), because that decides whether you commission for goodput or for availability.
• Basis of Design (BOD). The design engineer's documented explanation of how the proposed systems satisfy each OPR requirement: the topology, redundancy scheme, cooling architecture, setpoints and assumptions. ASHRAE Guideline 1.1-2025 covers applying the commissioning process to new HVAC&R systems and sets out how the BOD is developed and verified; it is not a BOD-only standard. The CxA verifies that the BOD answers the OPR before construction, not after.
• Sequence of Operations (SOO). The control-logic specification: exactly how the BMS, EPMS and DCIM should behave in every normal, maintenance and failure mode. It is the script-writer's bible, because an L4/L5 functional test is in essence a line-by-line proof that the real controls match the SOO under real conditions. A vague or incomplete SOO is a frequent cause of a failed or inconclusive IST, because there is nothing precise to test against.

The role that owns this chain is the Commissioning Authority (CxA), and the governance decision that matters most is to engage the CxA at the design phase, as an independent party rather than a general-contractor self-check bolted on at the end. ASHRAE Guideline 0 (the process) and ASHRAE Standard 202 (the formalized commissioning process, or Cx-Process) give the process guidance. The data-center-specific Guideline 1.6P, Application of the Commissioning Process to Data Centers, entered its first public review on 2 October 2026, with comments due by 16 November. Its draft scope covers the power chain and building systems “up to the server racks and IT equipment”, so it stays a facility document, and the cluster acceptance criteria remain the owner's to write. Cite the edition you adopt once it is published. This guide recommends an independent CxA who verifies the OPR→BOD→SOO chain across the whole project lifecycle, from design review (L0) through post-occupancy. An owner who engages the CxA only to 'witness the IST' has already lost the design-phase reviews where the cheapest defects are caught.

Give each interlock an evidence owner and a release owner. The electrical lead supplies bus waveforms and relay events. The mechanical lead supplies branch flow, temperature, pressure and the actual protective response. The platform lead supplies device health, workload correctness and recovery. The CxA joins those records against the OPR, BOD and SOO requirements, operations accepts the restored state, and the owner names the person authorized to release load. These are example responsibilities, not signatures.

Name a utility-interface owner as well. Your utility, or the transmission owner you connect to, can require its own commissioning evidence. It can ask for load models and an agreed ramp schedule, and for a demonstration that the computing load stays online when a normally cleared voltage sag hits. Prove both that it transfers cleanly to backup and that it rides through the sag. Under NERC’s May 2026 Level 3 alert, transmission owners should, where possible, commission computational loads with the computing equipment installed. In October 2026 NERC’s Board approved registration criteria for the new Computational Load Owner/Operator functions, pending FERC approval. The companion computational-load reliability standards passed NERC’s final ballot the same month and still need Board adoption and FERC approval. The ride-through test is in Chapter 13.6, and utility energization is in Chapter 13.10.

Record the installed rack interfaces, utility boundary and OT access and restore obligations from Chapter 11.11 in the same commissioning record. A missing vendor result stays open under its named owner instead of disappearing between two green dashboards.

A conventional data-center commissioning program is a single, well-standardized ladder ending in an IST against load banks. ASHRAE, Uptime and BICSI govern it end to end, and an independent CxA signs it. Applied unchanged to an AI factory, it certifies a building that has never seen its actual load and a cluster the standards never mention.

The program breaks in three structural ways. (1) Two tracks, not one: facility and cluster run on different schedules and standards and need explicit interlock gates. (2) Load realism: the air-rejecting, steady load banks of a conventional scope cannot reproduce the dynamic power and liquid-side thermal behavior of real GPUs. Switched and liquid-cooled load banks extend that coverage, and a staged, production-representative run supplies the workload-specific normal-operation results that facility emulators do not. (3) Density ramp: GB300 NVL72 is deploying through 2026 alongside GB200, Vera Rubin has been shipping since June 2026, and the announced ~600 kW Rubin Ultra/Kyber planning point follows (NVIDIA targets H2 2027; SemiAnalysis reported 2028 in July 2026). The IST must therefore validate not only the steady state but the headroom the ramp will consume: floor, water, electrical and cooling-plant turndown. A program that handles all three is a Part 13 program. One that ignores them hands day-2 operations a long deficiency list on top of the normal failure rate. Meta’s Llama 3 405B pre-training run saw 419 unexpected interruptions in 54 days, so a training cluster has no spare tolerance for facility defects.

Choose the interlocked program whose required states each have a valid test method, an evidence owner and a funded slot in the schedule. Finishing facility and cluster checklists separately is cheaper to organize, but then the first synchronized workload becomes an unscheduled test of their shared power, cooling and controls.