Documentation, Scripts & Acceptance Test Plans

Chapter 13.1 set the governance frame: the L1–L5 ladder, the parallel facility and IT tracks, and the governing documents (OPR/BOD/SOO) that everything traces back to. This chapter covers execution: the documents the commissioning agent writes, the scripts the field runs, and the gates that decide whether a system passes. It is the least glamorous material in Part 13. It is also what most reliably separates a building that goes live on schedule from one that slips a quarter while two firms argue about whether a 14-second generator transfer was a pass.

The question at every level is the same: did you write an observable, pre-agreed acceptance gate, or did you leave it soft? A script that says 'UPS shall transfer to battery without dropping the load' is unfalsifiable. What is 'the load' when no real load is connected, and what voltage sag counts as a 'drop'? A script that says 'on loss of utility, bus voltage shall not sag below 90% nominal for more than 10 ms, verified against a calibrated power-quality analyser logging at ≥ 10 kHz, witnessed by Owner and CxA' makes the decision explicit. Those thresholds are illustrative, though. They become a contract only after the owner freezes the terminals, the waveform/RMS convention and the uncertainty rule against the equipment limits. Chapter 13.3 supplies a fully disclosed source-transfer record. That precision decides whether a failing result is a deficiency the contractor fixes at their own cost or a 'disagreement' that becomes a change order at yours. Price commissioning from the witness hours, temporary load, instrumentation and retest scope. The rework and downtime it prevents justify that expense only if the gates are hard enough to enforce. → Chapter 13.1.

A commissioning script (also called a test procedure, ATP step or functional-performance test) is not a checklist. A checklist asks 'did you do X?'. A script asks 'when you do X, does the measured result fall inside the gate?' Every well-formed script has the same eight fields, and a missing field is where disputes start:

• Unique ID and traceability — back to a specific OPR/BOD requirement and a SOO step, so a passing script proves a design intent, not just an action.
• Pre-conditions — the exact system state, isolations and safety lockouts that must hold before the step runs. Skipping these is the field's most common shortcut, and the one that injures people.
• Procedure — numbered actions, written so a competent technician who has never seen the plant can execute them identically.
• Expected result with an observable gate — a number, tolerance and unit, or a categorical result such as the correct breaker position or denied access. For a numerical gate, name the terminals, time window and uncertainty rule; 'within spec' leaves all three undecided.
• Instrumentation and DAQ basis — which calibrated instrument reads the result, its accuracy class, and its in-date calibration certificate.
• Actual result — the recorded measurement, with a timestamp and a reference to the captured waveform or trend, not a tick.
• Pass / Fail / Deferred determination — decided against the gate, not against an opinion in the room.
• Witness signatures — CxA, Owner's representative and the responsible contractor, each signing that they observed the result, not that they trust it.

Whatever else varies, the gate is agreed and frozen before the test is run. The moment you negotiate the acceptance threshold while looking at a failing result, you have lost the leverage commissioning exists to give you. Freeze the gates at script review, a formal owner sign-off of the procedures weeks before energization. After that they are a contract, not a draft.

An AI factory has two acceptance regimes that meet at a seam, and most program risk lives in that seam. The facility ATP (the L3/L4 mechanical/electrical track) proves the building: switchgear, generators, UPS/BESS, chillers, CDUs, pumps and BMS under load-bank load, culminating in L5 integrated systems testing. The cluster ATP / SAT (the IT track) proves the machine: node burn-in, fabric BER and bandwidth, NCCL collectives, storage throughput, scheduler behaviour and a reference workload. It is accepted against goodput, not against load-bank kilowatts. The two regimes run on different schedules, against different standards, witnessed by different parties, and they are not interchangeable. Readiness tools from the vendors can now run much of the cluster ATP, but the pass/fail limits are still yours to set. NVIDIA's Cluster Readiness Engine (open-sourced in September 2026; v0.7.0 announced at OCP, October 2026) runs NCCL collectives, DCGM level-4 diagnostics and a reference pretraining job, and reports the nodes that fail. It ships no thresholds and never cordons a node: you set the limits, freeze them like any facility gate, and own quarantine. Record the accepted software stack as well. NVIDIA's AI Cluster Runtime v1.0 (October 2026) signs every recipe's provenance and, for recipes with published results, attaches a signed attestation of the validation run. When the installed stack drifts from the recorded configuration, re-test. → facility electrical in Chapter 13.3, cooling in Chapter 13.5, IST in Chapter 13.6; cluster burn-in in Chapter 13.8, fabric in Chapter 13.7, benchmarking in Chapter 13.9.

The seam exists because of a physics gap that is central to AI commissioning: the facility ATP exercises the building with a load the real workload does not resemble. An air-rejecting resistive load bank held flat draws a unity-power-factor, thermally steady load. Switched banks add defined steps, and liquid banks put heat into the loop. A synchronous training job draws a load that swings by megawatts in step with each training iteration, with millisecond-scale edges at the facility (microsecond edges stay inside the board and PSU), and it rejects its heat into cold plates and a liquid loop. A flat, air-rejecting fixture can prove a defined steady load (100 MW is an illustrative scope) but leaves UPS/BESS transient response and CDU branch heat untested. On an HPE GB200 NVL72 at 132 kW nominal, about 115 kW leaves through liquid and 17 kW through air, so an air-rejecting bank sized to rack power sends none of those 115 kW through the liquid loop the CDUs serve. A switched fixture adds only its measured waveform coverage, and a liquid-connected fixture adds only its measured branch duty and control bandwidth. Real nodes still confirm the die and contact path and the software timing. That gap makes two tests essential: liquid-loop emulation first, then a staged SAT workload. Treat both as acceptance work in their own right, not as 'IT validation' tacked on at the end. The dynamic-load realism problem belongs to Chapter 13.6; the cooling load-realism limit is engineered in Chapter 13.5.

A pass/fail gate is only as trustworthy as the instrument that reads it, and the recurring sin is letting the building's own BMS grade its own homework. The facility's permanent sensors are installed for control and trending, not for metrology. A BMS temperature point may carry ±1–2 °C uncertainty and a multi-second poll interval, which is useless for accepting a delta-T gate of ±1 °C or a transfer gate of ±10 ms. Acceptance reads against calibrated reference instruments: power-quality analysers, thermal imagers, ultrasonic and Coriolis flow meters, calibrated PT/RTD references and micro-ohmmeters. Each needs an in-date calibration certificate, attached to the script, from an ISO/IEC 17025-accredited laboratory. The certificate states traceability to NIST or another national metrology institute and the uncertainty at the range used. A result without a certificate behind the instrument is an anecdote.

Two DAQ decisions distinguish a serious program. First, sample rate must out-resolve the phenomenon. UPS and static-switch transfers last milliseconds. Sample them at 10 kS/s or more per channel, about 83 samples per 60 Hz half-cycle, or you cannot reconstruct the half-cycle RMS or the sub-cycle transfer you are accepting against. A generator load step is slower to settle: voltage and frequency dip within milliseconds and take seconds to come back. Where a set is specified to ISO 8528-5 class G3, that class permits up to about 4 s of voltage recovery and 3 s of frequency recovery. Log at the same rate until both have recovered. A 1 Hz BMS trend will log 'no anomaly' straight through a transient that broke the limit. Second, capture the full time-series, not the summary statistic: store the waveform and the trend, not just 'min 89.2%'. The captured series lets you adjudicate a disputed result after the fact, and it doubles as the baseline fingerprint discussed below. This matters more on AI factories than on legacy IT halls because the loads are transient. The interesting failures live in the milliseconds, and a DAQ basis that cannot see milliseconds cannot accept against them. → fabric timing acceptance (PTP/IEEE-1588) as its own metrology problem in Chapter 8.7.

Freeze the rule for reading the result at the same time as the gate. For a lower limit, subtract the measurement uncertainty written into the script; for an upper limit, add it. A result that overlaps the limit is held for better measurement or correction, never rounded to a pass. Specify the waveform channels, RMS window, event-trigger timing, clock alignment, missing-sample handling and artifact hashes. The source-transfer record in Chapter 13.3 shows this rule in use, and the installed-fabric record in Chapter 13.7 applies the same ownership of results and restoration discipline.

A tool's default is not a gate you froze. ibdiagnet v2.13.0's --ber_test checks every port against a default BER threshold of 10^-12, but NVIDIA's manual limits that test to SwitchX, ConnectX-3 and ConnectX-4 and sends later devices to --get_phy_info. A fabric script that says 'ibdiagnet BER test passes' on a current fabric therefore says nothing about the links actually installed. Write the BER type, FEC mode and limit for the installed links into the gate, as Chapter 13.7 does.

The binder of passed scripts is the visible deliverable. The deficiency log decides whether you go live. A serious program treats every failed or partially passed step as a tracked deficiency with an owner, a root cause, a corrective action, a re-test reference and a severity class, and the severity class is the lever. In a flat punch list a mislabelled valve sits at the same priority as a failed UPS transfer, so go-live becomes a negotiation about which items 'really' matter, conducted under schedule pressure. Pre-agree the severity tiers and which tiers block.

• Class A (blocker) — a life-safety defect or a failure of a core redundancy or ride-through claim. Go-live cannot proceed until it is closed and re-tested. Examples: a failed automatic transfer to generator; an EPO that does not trip; a leak-detection interlock that does not isolate.
• Class B (conditional) — a real deficiency that does not defeat the design basis. Go-live may proceed only on a written, time-limited concession: a dated corrective-action plan, an owner, and a named person who accepts the residual risk until it closes. Examples: a single redundant pump trending warm; a BMS alarm mis-mapped but functional.
• Class C (warranty/punch) — items with no safety or operational impact, which roll to the warranty list. Classify by impact, not by category. An equipment or isolation label a technician needs to switch safely, and an as-built or single-line a responder needs at 3 a.m., are Class A or B however much they look like documentation. Examples: cosmetic paint damage; a formatting correction on a non-operational document.

Getting the tiering wrong costs you either way. Tier too loosely and you carry a Class-A ride-through gap into live operation, where the first real utility loss finds it. Tier too strictly and you hold a live-block go-date hostage to a paint scratch. Fix the tiering rules and the blocker list in writing when you freeze the gates, before anyone has a result to argue about. Zero open Class-A items, every open Class-B under a signed time-limited concession, and a complete safety-critical label and as-built set are the real go-live gate. They feed directly into the Operational Readiness review and the handover package. → handover and the Operational Readiness gate in Chapter 13.10.

The most valuable artifact commissioning produces has no pass/fail gate at all: the baseline fingerprint. At the moment a system is accepted, it is in its known-good state: clean filters, balanced flows, calibrated sensors, fresh firmware, characterised transients. Capture that state quantitatively and day-2 operations has a reference against which all future drift is measured. Skip it and the first time anyone characterises the plant is the day it misbehaves, with nothing to compare against.

A useful fingerprint is multi-domain and time-stamped:

• the captured transient waveforms from every transfer test;
• the as-accepted pump and fan curves and the flow balance;
• per-rack and per-branch coolant flow and delta-T at known load;
• thermal images of every switchgear connection and busbar joint;
• PUE/WUE at the commissioned load point;
• per-node power-draw signatures and HBM/ECC baselines from burn-in;
• per-port BER and per-link bandwidth from the fabric;
• NCCL busbw and goodput from the reference run.

Together they seed the operational digital twin and the day-2 reliability program. Anomaly detection, predictive maintenance and lemon-node ejection all need a 'normal' to deviate from, and acceptance is the first controlled observation of the system inside the operating range it was tested in. → the operational twin and telemetry handoff in Chapter 14.2; the goodput baseline carried into operations in Chapter 14.1. The fingerprint is distinct from the design-validation digital twin of Chapter 2.7: that one predicts behaviour before the build, and this one records measured reality at acceptance.

Scripts form a dependency graph: a passing downstream script is valid only if its upstream prerequisites passed first. Electrical acceptance (L3/L4) must clear before integrated systems testing can apply real building load. Cooling acceptance and the secondary-loop flush must clear before any GPU draws power into a cold plate. Fabric BER and bandwidth must clear before NCCL collectives mean anything, and node burn-in must clear before a reference training run is interpretable. The program is therefore a sequence of gates, each unlocking the next, with two deliberately overlapping seams that 13.1 flagged. The first is mechanical Cx ↔ GPU burn-in: liquid banks load the loop before real nodes confirm cold-plate and contact behavior. The second is facility IST ↔ first real workload: a staged proxy or reference run gives product-representative workload results, while dynamic AI-emulating and liquid-cooled load banks cover other parts of the behavior without risking production hardware. Treat each overlap as a single coordinated gate with shared acceptance criteria, not a clean hand-off. Otherwise each side accepts to its own boundary and the seam goes untested. → the staged power/load ramp that walks these gates live in Chapter 13.10; the design-basis redundancy definitions the topology scripts validate against in Chapter 0.5.

Release against the frozen requirement, the captured result and the restored configuration together. A signed waveform without its terminals and uncertainty settles nothing. A signed checklist without a retest dependency can describe equipment that no longer exists. Spend the witness time on that join before the revenue date makes every missing field a dispute.