Thermal Design, Reliability, Leak Detection & Commissioning

Earlier chapters in Part 5 chose a cooling technology and sized its pieces. This chapter makes those pieces work together, reliably, and survive the day something fails. It separates a rack that runs at rated TDP for five years from one that throttles every afternoon because the thermal budget never added up at the worst rack in the row. Liquid forgives less than air ever did. With water (or PG25) inside a 200+ kW rack, an open budget, an undetected leak or an unrehearsed loss of cooling does not degrade gracefully. It can trigger a rack protection response, and on a synchronous job that restarts thousands of GPUs from the last checkpoint.

Four problems follow, each constraining the next. The end-to-end thermal budget comes first: a worked ladder on a specific rack walks the temperature from junction to wet-bulb with the resistance stack from 5.1 and shows the budget adds up at the worst-case branch, not the average. Then the reliability architecture: redundancy topology, UPS-backed pumps, and the rack's validated thermal ride-through. Then leak detection and containment: sensing, zoning and the auto-isolation response. Finally serviceability and commissioning: quick-disconnects, blind-mate, MTTR, the handoff to integrated acceptance in 13.5, and the fluid-analysis program that keeps the loop in spec for life. The cooling-controls transient problem (anti-hunting, setpoint stability, dew-point excursions under a synchronized load slam) is the thermal twin of the electrical transient. It lives in Chapter 5.12, together with pump-power continuity. The consolidated failure-mode catalog is in Appendix F.

The most useful artifact in a liquid-cooled design is a temperature ladder that starts at the silicon junction and ends at the outdoor wet-bulb, carrying every thermal resistance and every approach temperature. If the junction stays under its throttle limit with outdoor air at design wet-bulb (design dry-bulb for a dry cooler) and the worst rack in the hall at full load, the facility works. If the ladder adds up on a spreadsheet only for the average rack, not the rack at the end of the longest manifold run, you throttle, and you spend the first year of operations chasing a designed-in goodput leak.

Follow the heat forward. A Blackwell-class GPU dissipating on the order of 1–1.4 kW has a die-average heat flux of about 90 W/cm² (1.4 kW over roughly 16 cm² of silicon), with local hotspots well beyond that. That is past air's practical socket-level limit, which is why we are here at all. The junction-to-coolant path runs through the silicon, the TIM, the cold-plate base and the convective film inside the cold-plate microchannels. Each is a thermal resistance in K/W, and the sum times chip power is the rise from coolant to junction. The coolant then heats as it crosses the cold plate. That is the cold-plate delta-T, set by flow rate and fluid heat capacity. The warmed coolant returns through the rack manifold (a pressure-drop and mixing penalty) and crosses the CDU heat exchanger (typically a 3–5 °C approach between the technology-cooling loop and the facility water loop). It travels the facility loop to the heat-rejection plant and rejects to ambient across a dry cooler or tower, another approach to dry-bulb or wet-bulb. Each handoff costs degrees. The budget is the bookkeeping of those degrees against the one number that matters: the GPU's throttle threshold. The ladder below does that bookkeeping for Lenovo's GB300 NVL72, whose published flow table fixes the coolant rise, with every other operand an assumption. Chapter 5.1 owns the resistance stack itself; this chapter adds the local limit the protection schedule must hold.

The last two rows hold the money. Only the margin to throttle on the worst branch at design wet-bulb counts. An inexperienced designer chases a colder plant: drop the facility supply and buy back degrees with a chiller. A colder plant does buy margin, but it costs energy and capital, and a warmer supported point can add economizer hours. There is no universal W40 or W45 default. Pick the operating point from your own TCS limits, FWS class, climate bins, heat-exchanger and rejection approaches, load, control range and redundancy. Recover margin upstream of the plant instead: a tighter CDU approach, a better TIM, higher per-chip flow to shrink the cold-plate delta-T, or a balanced manifold so the worst branch is not 3 °C hotter than the best. Spend the degree where it is cheap. Fundamentals and the metric definitions are in Chapter 5.1; the cold-plate and flow design in Chapter 5.4; the warm-water loop in Chapter 5.7; heat rejection in Chapter 5.8.

The ladder assumes an evaporative plant, which rejects against wet-bulb. A dry cooler rejects against dry-bulb, which runs well above wet-bulb on design days. At a 35 °C design dry-bulb and a 5 K approach, facility supply is about 40 °C and TCS supply about 44 °C. To accept 44 °C water, Lenovo's W45 rating wants something close to the rack's 177 L/min high-temperature flow, roughly double the 35 °C row, and even then the supply is only 1 K under the 45 °C inlet limit. Junction margin hardly changes from the ladder above. The warmer water is offset by the doubled flow, which trims the coolant rise to about 10 K and leaves the junction near 80–92 °C, as tight a band as on an evaporative plant. What breaks is the inlet. Stack the ladder's 1–3 K manifold pickup on top and the worst branch reads 45–47 °C on a design day, touching or past the limit while the CDU is still on setpoint. So a dry plant in a warm climate carries a trim chiller or adiabatic assist for its design days.

A direct-to-chip loop's loss-of-flow response depends on the product and its implementation. Establish whether your rack throttles, shuts down in a controlled or emergency way, or trips, and on what timescale, from OEM transient data, controls tests or a professional-engineering calculation. That evidence sets the pump and heat-rejection protection and the redundancy posture. No universal seconds-to-trip value or N+1/2N topology applies.

Redundancy then splits into two layers that fail on different timescales. The pumping layer (CDU pumps, secondary-loop circulation) must be sized and protected against that tested loss-of-flow response, and the same results say whether the pumps need UPS/BESS, protected feeds or redundant circuits. The heat-rejection layer (chillers, dry coolers, towers, the facility loop) has a transient that differs by product and implementation, because loop volume and temperature margin vary. Test that transient on your own loop; do not assume a ride-through time measured somewhere else carries over. The design choice is whether to buy ride-through with thermal storage (a buffer tank, a chilled-water volume, a deliberately oversized loop) or accept thin margin and lean on fast, reliable failover. The electrical spine that powers all of this, and the BBU/BESS layering behind it, is in Chapter 4.5. CDU sizing and N+1 pump topology are in Chapter 5.6; heat-rejection redundancy and the lack of chilled-water inertia in DLC are in Chapter 5.8.

The cooling posture is a continuity decision made state by state. Checkpointing may cut what a trip costs in goodput, and request retry and fleet failover may soften the hit to serving, yet none of them proves N, N+1 or 2N cooling is sufficient. Only working through each state against the requirements listed under the table does. Then compare the incremental topology cost with workload and fleet recovery in Chapter 12.5.

Water and energized electronics are an old enemy, and the industry's tolerance for leaks in a 200 kW rack is effectively zero. The defense is layered. Keep the fluid in the loop. If it escapes, detect it fast enough to cut heat, contain the fluid and close the branch valves before the coolant can reach a live busbar, at a valve speed the surge analysis allows (Chapter 5.13). At every stage keep the response proportional, so a drip pauses a rack rather than a job. Connectors do most of the work. Modern in-rack plumbing uses dripless quick-disconnects (UQD/UQDB-class couplings) and blind-mate floating-tray connections that self-align and seal on insertion, so servicing a node causes a few hundredths of a millilitre of loss per connection, not a spill. OCP UQD/UQDB v2.0.0 caps loss per coupling cycle at 0.02–0.07 mL, depending on size, and OCP's MQD/MQDB 1.0.0 at 0.020–0.025 mL. But connectors fail, hoses chafe, cold plates corrode and gaskets age, so detection and isolation are the backstop.

Detection runs on three complementary modalities, and serious designs use more than one. Point sensors (float switches and conductive pads in drip trays and at low points) are cheap and certain, but they see only fluid that has pooled where you guessed it would. Rope or cable leak detection runs along manifolds, under racks and through cable trays, localizing a leak to a length of cable rather than a point (Project Deschutes 1.0.0 builds three rope zones into the CDU, read by its PLC). Pressure and flow telemetry catches the leak that never reaches a sensor: a slow drop in loop pressure, or a mismatch between supply and return flow, betrays a leak upstream of any pooling. Leak forecasting with machine learning is still research. A December 2025 preprint (Sunkara and Konakanchi, arXiv 2512.21801) forecast leaks 2–4 hours ahead from humidity, pressure and flow signals, but only on synthetic data. Treat forecasting as an extra alert layer, never as the commissioned local detection and isolation response. That response must be zoned: alarm and isolate the affected rack or row by closing its manifold valves in that same heat-off, surge-safe sequence. Do not silently dump the whole hall's cooling, which would turn a contained drip into a hall-wide loss-of-cooling trip.

A liquid-cooled rack will need service: a cold plate fouls, a hose ages out, a node fails. How long that service takes decides your fleet availability, and mean-time-to-repair is a parameter you set at procurement, not a number you discover in operations. Two architectures bracket it. A rack built on dripless quick-disconnects and blind-mate trays lets a technician pull a node, swap the failed assembly and re-seat it through a tray procedure that has been demonstrated end to end: isolation, tools, spill, air inclusion and restoration time. The connectors break and seal cleanly, and the rest of the rack keeps running. A rack plumbed with conventional fittings and shared manifold segments that cannot isolate forces a drain-and-fill of a whole branch to touch one node: hours of downtime, a re-bleed to purge air, and a re-commission of that branch before it carries load again.

Treat serviceability as a set of requirements to extract at procurement. Every fluid connection is a dripless quick-disconnect rated for the cycle count of expected service. Each node uses its qualified hand-mate or blind-mate arrangement. Where you choose blind-mate, inserting the tray seats the intended power and coolant interfaces within their alignment limits. Every branch has isolation valves so a single cold plate can be isolated without draining the row. The CDU carries side-stream filtration (Deschutes-class designs run 0.2-micron filtration) and redundant pumps, so filter and pump service never stop cooling. Name the published OCP interfaces by revision, so a hall is not locked to one vendor's serviceability story. Couplings follow UQD/UQDB v2.0.0 (effective 22 September 2026), which MGX 1.2 compute trays also use, or MQD/MQDB 1.0.0 (OCP Steering Committee approval 8 October 2026) for the MGX busbar coolant connection and fanless I/O bays. Rope leak sensors can follow OCP's base specification 1.0.0 (effective 2 February 2026), which sets their signal interface, onboard power and false-alarm handling.

Then ask what qualification actually covers. UL Solutions launched certification to UL 4501 on 2026-09-29. The document is still an Outline of Investigation and has not yet become a consensus standard. Its tests cover cold plates, manifolds and quick disconnects, and check each for durability, flow, interoperability, coolant compatibility, and pressure and leak integrity. UL named no certified product at launch, so ask which coolant and which part pairings any certificate covers. For the cold plate itself, OCP's single-phase cold-plate test specification 1.0.0 (effective 3 September 2026) is an open test suite covering leak, burst, thermal resistance and pressure drop. After shipping-temperature cycling it requires a helium leak rate at or below 1×10⁻⁵ mbar·L/s. Ask suppliers for their report against it alongside any UL certificate. NVIDIA DSX Ready is a different kind of qualification. A CDU vendor earns it by running NVIDIA's own self-qualification suite against functional requirements, so it is not site acceptance. Treat it as one input. It feeds your site hydraulic sizing, your redundancy design and the commissioning evidence that goes to Chapter 13.5. Vertiv, whose CDU earned the label in September 2026 (Chapter 5.6), says the qualification does not replace site engineering. For couplings, UQD/UQDB v2.0.0 makes every supplier cross-validate against at least three others and publish the list. Ask for the cross-verification report on your exact plug and socket pair. If your manifolds, couplings and coolant come from different vendors, that combination still needs its own test results.

Commissioning is where the design meets reality, and finding a weeping weld before delivery is cheaper than finding it over a live rack. Hand the local cause-and-effect schedule to Chapter 13.5 for integrated cooling acceptance. Chapter 13.1 sequences the program, and 13.6 owns the L1–L5 ladder and the Level 5 integrated test. Before local response tests, finish the pressure test on the isolated loop, flush manufacturing debris to the fluid-release limits, calibrate sensors and synchronize timestamps. Establish commanded heat and flow at the worst branch. Then use approved signal injection or a qualified thermal simulator to exercise pump loss, a leak signal or a control fault. An artificial heater proves its heat-transfer and control response, not an accelerator's no-flow junction transient.

Distinguish command from physical result. Record detection, actual heat reduction, valve position, surviving branch flow, temperature, pressure and reset permission on one time basis. Sensor uncertainty and timing resolution spend margin. A delivered alarm with no required heat or valve response fails, even if the test happened to finish before anything overheated.

Heat-off upper bound = 1.80 + 0.05 = 1.85 s, inside 2.00 s. Flow lower bound = 112 × 0.980 = 109.76 L/min, above 109 L/min. Temperature upper bound = 36.2 + 0.2 = 36.4 °C, below 38.0 °C. Pressure bounds are 0.09–0.51 MPa(g), inside the allowed range. Keep the sequence as a candidate for test.

Repeat the case with the supervisory connection absent and verify that actual local heat-off, isolation and deliberate reset meet the same limits. Record command and actual heater power, valve position and alarm state alongside the thermal signals. A 10 Hz thermal log can be enough for this example's event resolution; choose the pressure sampling rate separately, from the shortest relevant hydraulic wave transit. A slow thermal trace cannot clear surge.

The heat-off crossover is an indicated 1.95 s when ±0.05 s timing uncertainty applies. A failing illustrative trace at 2.40 s has an upper bound of 2.45 s and fails by 0.45 s. Correct the local execution or its validated timing budget before retesting, and do not waive the failed limit because temperature happened to stay low. Get the real OEM transient limits and approval of the test boundary before you run it. Chapter 13.5 owns integrated acceptance, including the other required pump, plant, restart and sensor states.

Commissioning hands you a clean, balanced, leak-tight loop. Operations has to keep it that way for five-plus years, and the loop degrades whether you watch it or not. PG25 and similar glycol coolants oxidize and lose inhibitor. Dissimilar metals in cold plates, manifolds and the CDU set up galvanic corrosion. Microbes grow when the glycol is diluted below its biostatic level, or when the biocide in a water-based fluid is used up. The OCP PG25 guideline (Revision 2.0, September 2026) wants glycol held at about 25% or above to keep bio-growth down, while its fresh-fluid range for PG25 is 24.5–27.5%. So a PG25 fill made at the band's minimum starts under that line, and topping up with plain water can drag any fill under it. Particulate sheds from every wetted surface and fouls the microchannels your thermal budget depends on. A fouled cold plate raises junction-to-coolant resistance, the largest term in the thermal ladder, and eats into the margin you commissioned. The countermeasure is a fluid-analysis program: scheduled sampling for pH, inhibitor concentration, glycol concentration, conductivity, dissolved metals (the corrosion signature) and biological activity, with side-stream filtration (sub-micron on hyperscale CDUs) and corrections the fluid supplier specifies from the sample analysis. The cadence is a real cost and a real decision. Use the supplier-approved interval and adverse-trend triggers from 5.7. It is still cheaper than discovering corrosion by the leak it eventually causes.

This is the last of Part 5's mechanical content. The coolant chemistry and material-compatibility limits are set in Chapter 5.4 (coolant selection) and Chapter 5.6 (CDU filtration and chemistry). Facility-water treatment and the biocide and Legionella program for the heat-rejection side are in Chapter 5.7 and Chapter 5.8. The pressure-system mechanical engineering (pipe code, water-hammer, NDE, and the hydrostatic acceptance behind the installed-loop pressure test) is in Chapter 5.13. The flush, fill and pressure acceptance procedures referenced here are detailed in the construction-execution path at Chapter 13.5.

Pick the local response and capacity arrangement that meets your fault requirements on its own, with no help from the supervisory system. If a test misses the heat-removal deadline, change the protection sequence or lower the operating load. Spare capacity on a steady-state calculation does not, by itself, keep useful output flowing through the transition.