Cooling Acceptance: Air, Liquid-to-Chip & CDU Commissioning

Every acceptance domain in Part 13 can reach its steady design point on surrogate load; what the surrogate misses differs by domain. Electrical acceptance drives the switchgear and UPS with load banks (Chapter 13.3), but no bank reproduces the exact synchronized power swing of a real cluster. Generators and microgrids are paralleled and islanded against resistive and reactive banks (Chapter 13.4). Before any exceptional live high-energy test, integrated systems testing uses secondary injection, approved loss-of-source simulation, HIL, or a controlled transfer at an isolated boundary; Chapter 13.6 is where the guide treats that electrical realism gap. Cooling has the wider gap. An air-rejecting facility load bank is a resistor stack with a fan: it turns megawatts into hot air and rejects that air into the room. It cannot push heat through a cold plate into the secondary liquid loop. Liquid-cooled load banks can drive the CDU secondary loop and the rack or manifold branches before production GPUs are racked, and a staged workload then covers the die, TIM and contact path and the workload-specific behavior.

That limit sets the order of the chapter. Airside acceptance comes first. Next is the secondary-loop work that can be done dry or with surrogate heat: fill and vent the isolated test scope, hydrostatic acceptance, flush to fluid quality, coolant charge and purge. Then come CDU acceptance and the worst-case-branch problem, and finally the leak-integrity, failover and burn-in interlocks that close the gate. Each fork carries a cost that comes due when real silicon arrives. Mechanical commissioning and GPU burn-in (Chapter 13.8) are not adjacent phases with a clean handoff. They overlap by physics, because a staged workload supplies the hardware- and workload-specific results that liquid-loop emulation cannot.

Cooling acceptance spans two physically distinct systems joined at the CDU. The facility water system (FWS) covers chillers or dry coolers, towers, the primary loop, pumps and the airside plant. It is conventional mechanical Cx, and most of it can be driven to design with surrogate load: load banks dump heat into the room for the air handlers to reject, and the CDU's own heat exchanger or temporary process loads can exercise the primary loop. The technology cooling system (TCS) is where the load-realism limit bites. It is the secondary loop the CDU isolates from facility water, plus the in-rack manifolds, quick-disconnects and cold plates. You can fill it, pressure-test it, flush it and run the pumps. Rack- or manifold-connected liquid-cooled load banks then impose controlled steady and transient heat, exercising secondary-loop flow, ΔT, control response and branch hydraulics without production silicon. Chapter 5.6 engineers the CDU/TCS separation and loop architecture; this chapter accepts what was built there.

Even a fully liquid-cooled hall has a residual air load. In HPE's GB200 NVL72, 17 kW per rack stays on air. Expect the power-shelf PSUs and optics in that 17 kW, along with tray parts outside the cold-plate path, for example NICs, DPUs, local storage and management hardware. Storage, networking and any modest-density inference rows may be entirely air-cooled (Chapter 5.2). Lenovo Press LP2357 puts GB300 NVL72 on a 90/10 liquid/air heat split: roughly 13.5 kW on air at its 135 kW rack TDP and about 15.5 kW at the 155 kW peak, with the NVLink switch trays moved fully to liquid. Vera Rubin NVL72 puts its compute and switch trays and the busbar on liquid, but its power-shelf PSUs stay on air, so even that rack leaves a residual air load (Chapter 5.1). Airside acceptance is the conventional, well-understood half of this chapter, and it is provable before GPUs arrive because air is the load bank's native sink. The work is to verify CRAH, RDHx and in-row capacity at design heat, with load banks placed to mimic the rack thermal map. Commission containment (hot/cold-aisle or rear-door) for leakage and bypass. Tune supply-air setpoints against the ASHRAE A1–A4 envelope, and prove airflow balance so no rack starves. For hybrid halls running DLC plus rear-door exchangers (Chapter 5.3), the RDHx water side is part of this acceptance, and its condensation and dew-point margin is set here.

The decision that matters in airside acceptance is how much residual-air capacity you commission relative to the liquid fraction. Over-commission and you pay for air handling that idles as the hall moves more load to liquid through the density ramp. Under-commission and the room plant comes up short when the installed rack mix raises residual-air duty beyond the accepted limit. Liquid-cooled silicon has no alternate air path, so cold-plate and flow faults must instead prove alarm, isolation, power capping or throttling, orderly shutdown, and trip. Commission against the OEM's stated residual-air duty for the racks in scope, and preserve physical and electrical provisions for later airside expansion; total rack power does not define the future air load.

Before any working coolant touches a cold plate, the secondary loop must be flushed and proven clean. This is the most under-budgeted step in cooling acceptance. The microchannels that make direct liquid cooling work are sub-millimeter passages that drive the convective coefficient, and particulate or biological fouling blocks them. Once a cold plate is fouled you cannot clean it in place. You replace it, in a live rack, with the loop drained. The flush protects the most expensive and least serviceable surface in the building.

Your fluid supplier and your TCS pre-commissioning plan set the acceptance number. OCP's pre-commissioning guideline for row manifolds (NVIDIA and Nalco Ecolab, March 2025) shows what a data-center version looks like, for pipework before racks and CDUs are connected. Its procedure water is at most 50 µS/cm. Flushing continues until the outlet reads within 10–20 µS/cm of that water, or no more than 60 µS/cm, and turbidity stays reliably under 5 NTU. Filtration steps down in stages to a final 5 µm, since cold-plate channels run about 100 µm. OCP's PG25 guideline (Rev 2.0, September 2026) is stricter about the water you put in: under 25 µS/cm and 1 NTU, with caps on chloride, sulfate, hardness, iron and bacteria. After bulk debris is gone, PG25 itself is its preferred final flush fluid. Hold the loop to your fluid's own limits for conductivity, particulate class and residual glycol, sampled at the furthest branch rather than at the CDU. Record the wetted-material matrix, water source, fluid batch and concentration with the release signature; a different fluid carries different limits. Chapter 5.7 covers the flush cadence, and Chapter 5.4 the cold plates.

Pressure integrity can be tested before GPUs because it tests the pressure boundary. A cold static hold does not accept hot-state cycling, QD service cycles or surge. The charged-piping code basis is ASME B31.x in North America or the EU PED / EN 13480 fork in Europe (Chapter 5.13). Hydrostatic acceptance begins by filling and venting the isolated pipe scope with a compatible test fluid. The governing code sets the hold pressure. B31.3 and B31.9 differ in scope and testing rules, so use the adopted code's pressure, temperature correction, examination and hold requirements for the exact pipe scope, never a blanket multiplier. Remove or isolate components not rated for the pipe-scope test and prove them under their own OEM-qualified procedures. Hold with a stabilized fluid temperature, because pressure decay needs temperature correction before it can identify a joint or gasket that seated only under initial pressure. The sequence is strict: fill and vent the isolated test scope with test fluid, pressure-test, drain only if repairs or the plan require it, flush to the cleanliness criterion, then charge the working coolant and purge. Reorder it and you either flush a loop you have not proven leak-tight or charge working fluid into a loop you have not cleaned.

The gaps between steps matter as much as the order. OCP's TCS pre-commissioning guideline (2025) sets a limit of 24 hours for untreated water sitting in a loop with no circulation. A passed hydrotest moves directly into cleaning and flushing with no drain and refill, and the guideline's preferred next move is filling straight away with glycol coolant. If the coolant or CDUs are not ready, hold the loop in one of two states: circulating, or purged as dry as you can and sealed under nitrogen. The plan says which.

The decision embedded here concerns the quick-disconnects. A GB300 NVL72 has 27 liquid-cooled trays (18 compute, 9 switch), so it has at least 54 tray-level dripless quick-disconnect connections before counting any inside the trays (Lenovo's product guide; get the maker's schedule for your rack revision). Every one is a potential leak path. A QD inside the test scope gets its static seal checked by the hold, and that is all the hold shows; it says nothing about behavior after service cycles. So cycle a representative sample under the OEM's qualified procedure to catch couplings that seal statically but weep after a service cycle. The sample adds acceptance time. Skipping it buys leak risk during the first board swap, and since serviceability is the whole point of dripless QDs, cycling a sample is the right call.

The CDU is the seam of the cooling system. It isolates the technology loop from facility water, sets secondary flow and temperature, and carries the controls that must respond to load. Uptime Intelligence has flagged it as the component most likely to complicate commissioning, because some CDU makers came from outside the data-center world and have, or used to have, limited experience integrating a unit into a complex fluid network. That makes the factory witness test (FWT) decision consequential: witness the CDU's flow, head, pump-redundancy failover and control response in the vendor's bay, or accept it on a datasheet and discover a defect in your live hall. The cost asymmetry is stark. A pump-curve or PID defect found at the factory is vendor rework; found in the field it is a hall-level schedule hit with the cluster waiting. For any first-of-a-kind CDU model or vendor, FWT is the rational default.

Ask for the capacity basis as well. ANSI/ASHRAE Standard 127-2020 Addendum b, approved in February 2026, adds a laboratory rating method for liquid-to-liquid CDUs. It sets facility water in at 26 °C and TCS supply at 30 °C, with 25% propylene glycol on the TCS side. Capacity is measured at 100, 75, 50 and 25% at constant TCS flow, plus a minimum-capacity point with the automatic flow controls active. The report covers net capacity, input power and the external head available. Ask vendors for ratings on that basis and at your own design temperatures, since a CDU rating in MW with no conditions cannot be compared. A rating test leaves pump failover and control response unexercised, so the factory witness test and on-site acceptance must cover them.

On site, CDU acceptance proves rated flow and head, pump N+1 failover (kill the lead pump and confirm the lag pump holds flow without a thermal excursion), filtration and dew-point control, and the leak-detection integration. Test at the duty the hall will run, not only at nameplate. HPE rates the in-row CDU for its GB200 NVL72 at 1.3 MW and allows at most eight racks per unit; eight racks at 115 kW of liquid heat each put about 920 kW on it. Prove flow, head and approach at that installed duty and at the facility-water temperature you will actually supply. Flow verification follows the approved staged profile, for example 25% → 50% → 75% → 100% as an illustrative sequence, not a portable ramp rule. Log temperature differential, flow rate and pressure drop across the CDU, piping and rack manifolds at coincident heat duty. That staging runs against surrogate or balancing-valve load, which raises the hardest problem in the chapter.

The worst-case branch. A liquid loop balances flow across many parallel branches. The branch with the least available differential pressure relative to its required flow is the hydraulic candidate for starvation at full load, and the hottest branch can be a different one, so instrument both. Acceptance instruments the worst-case branch and verifies that it makes its minimum flow while the whole system is loaded. Connect liquid-cooled load banks at every rack or manifold branch in the acceptance scope to impose controlled, simultaneous steady and transient heat. Use balancing valves to reproduce the branch with the least differential-pressure margin at required flow, and instrument the hottest branch separately. This proves branch flow, balance and control response before production silicon arrives. A staged workload then confirms production-rack and workload-specific operation.

Leak detection in a liquid-cooled hall is a real-time interlock that must throttle or park the GPUs before a coolant loss drives junction temperatures past their limit. Two architectural choices set the acceptance work. First, positive versus negative-pressure operation. A negative-pressure (sub-atmospheric) secondary loop draws air in on a breach instead of pushing coolant out, so a spray onto live electronics becomes an air ingress. That is a much safer failure mode, and some designs adopt it specifically to de-risk leaks. Acceptance must confirm the pressure regime behaves as designed under a fault. Second, the detection-to-action chain: rope and spot leak sensors, flow and pressure anomaly detection, and the logic that converts a detection into a GPU power cap, throttle or park. Acceptance injects faults (manual trips, simulated sensor alarms, a forced CDU pump loss) and confirms the action fires fast enough.

The protection chain is proven by injection: trip the sensor, spoof the alarm, kill the lead pump under liquid-bank heat, and record the BMC command and the time of the actual load reduction. The number to hold is the time to actual heat reduction or shutdown against the local limit in Chapter 5.11. Measure it first on the emulator, then verify it with the installed control path; timing a BMC command alone leaves actuator response unproven. A leak is not staged on energized silicon to prove the logic. If an owner authorizes a bounded live fault test, it is a documented exception with its own hazard analysis, not an acceptance step.

One trace record per injection. P starts from 4×80 = 320 kW liquid and 40 kW air. At t = 0 the lead pump stops. Standby duty at 1.00 s and the P extrema cover every branch through t = 9 s. Lower flow = 112×0.980 = 109.76 L/min (about 110), upper temperature = 36.4 °C, pressure = 0.09–0.51 MPa(g), and standby upper time = 1.05 s ≤1.20 s. P-R restores the original pump assignment and balance at 9 s (upper 9.05 ≤10), then records the baseline through t = 39 s. Restore before the next injection.

D has no failed pump or leak input. It requests 60→80 kW on all four branches over t = 0–1 s, so liquid heat rises 240→320 kW while air stays at 40 kW. Lower flow = 116×0.980 = 113.68 L/min (about 114), upper temperature = 36.2 °C, pressure = 0.11–0.49 MPa(g). D-R returns to the 80 kW/branch baseline at t = 9 s, within the 10 s recovery gate, and holds through t = 39 s. This trace tests the load increase; it cannot be borrowed from P.

L starts again at 320 kW liquid plus 40 kW air. At t = 0 inject a branch-1 leak and request +20 kW there. The simulated control record refuses the increase, so actual branch heat never exceeds 80 kW. Detection at 0.10 s, actual heat off at 1.80 s and isolation at 3.00 s are three separate events. The heat-off upper bound is 1.85 s ≤2.00 s, and the three surviving branches keep P's bounds at 80 kW each. The isolated branch has zero flow by design, which is not a passing loaded-flow result. L-R keeps it off until the deliberate reset at 10 s, after simulated repair, leak clear and restored baseline flow. A 4 s ramp then restores its 80 kW. At 14 s all branches regain baseline, within 10 s of reset, and hold through 44 s. No automatic restart is assumed.

Result and flip. P/P-R, D/D-R and L/L-R pass only their simulation gates. Their baseline lower flow is 117.6 L/min, upper temperature 35.2 °C and pressure 0.19–0.41 MPa(g). Release of the whole CDU waits for the missing records: air duty, sensor failure and supervisory loss, plus actual fluid and pump curves, high-rate surge, calibrated power and thermal traces, and a witnessed reset. Mechanical supplies the branch and valve records, platform and electrical the actual power and command timing, and operations and the CxA sign off restoration and release. Abort on a missed limit or a lost critical channel, using the independent heat-off path. The flip point is an indicated heat-off of 1.95 s. An actuator that indicates 2.40 s has an upper bound of 2.40 + 0.05 = 2.45 s and fails by 0.45 s, even if temperature stays low. Repair the actuator before increasing load. OCP's CDU methodology (cover dated August 2024) and ASHRAE 127 Addendum b (2026) each rate CDUs at fixed test conditions that are published in advance. Local limits belong to 5.11. Skipping the witness slot to save time moves a pump or control defect into the first hot hall.

The recurring cooling-acceptance failures share a root cause: treating an air-rejecting load bank as proof of the liquid loop, or treating mechanical Cx and burn-in as cleanly separable. Four stand out:

• Signing off cooling on facility load banks alone. Facility load banks prove FWS capacity, not the cold-plate path. Put liquid-cooled banks on the manifolds and read the first workload before signing the liquid side.
• Under-flushing to recover schedule. Calling the loop clean before conductivity and particulates have stabilized. The debt comes due weeks later as unexplained throttling and a cold-plate replacement campaign in a live hall.
• Accepting a first-of-kind CDU on a datasheet. Skipping factory witness on a new CDU model or vendor turns a pump-curve or control defect that would have been factory rework into a hall-level schedule hit with the cluster idle.
• Using live silicon as the protective-function test fixture. Prove the interlocks with injected trips while the liquid banks run, and measure actual power reduction as well as command timing on the installed rack. A leak staged on a live rack proves nothing the injection did not, and it costs you the rack.