Cooling-Controls Transient Dynamics & Setpoint Stability

Chapter 4.5 told the electrical side of this story. A frontier training cluster is an unusually hard load for the grid, because tens of thousands of synchronized GPUs ramp from idle to full power and back on millisecond timescales, and the power chain must absorb the di/dt with stored energy before it propagates upstream. This chapter is the thermal twin of that problem. The same synchronized slam that stresses the busbar lands within milliseconds as a step change in heat flux at the cold plate. A direct-to-chip loop may have far less usable thermal inertia than the legacy chilled-water reserve it replaces. Local component mass, circulating inventory and temperature margin decide which holds more. The decision here is about controls, not capacity (Chapter 5.1 owns the capacity wall): how fast the controls may move, how they avoid oscillating, and what they do when the load disappears as fast as it arrived.

Four settings carry the chapter: slew-limit tuning, inertia versus response, dew-point margin and capping coordination. Get any of them wrong and the cost is concrete: a throttled GPU (lost goodput), a hunting loop (mechanical wear and unstable die temperatures), or condensation on a cold manifold inside an energized rack (the failure mode catalogued in Appendix F). None of this is exotic. It is the ordinary consequence of running a low-inertia loop under a load that swings harder and faster than any cooling plant was historically asked to follow.

A legacy air-cooled hall is a giant thermal flywheel: room air, the raised-floor plenum, slab and racks, plus stored water in tanks and pipework. But it rides through a load step only on the part of that mass the heat actually reaches during the event, so a large room does not by itself excuse a sluggish CRAH fan. Before you set a delay budget, work out the excess energy coming in, the inventory that takes part, whether circulation continues and the temperature rise you can allow.

A direct-to-chip loop can buffer a maintained-flow heat step for tens of seconds in its coolant and metal mass, but it offers almost no inertia after flow stops. The coolant over the die is a thin film in a microchannel cold plate. Take the TCS inventory from the actual circuit schedule. Transport from cold plate to CDU heat exchanger and back takes seconds, not minutes. Thermal resistance from junction to coolant is deliberately tiny, which is the whole point of liquid cooling, so the die rides a small, power-dependent offset above the local coolant temperature. When GPU power steps by more than half of TDP in milliseconds (measured: power variations exceeding 50% of TDP within a few AC cycles, and current swings from a 5–10 A idle baseline to 20–25 A in under 200 ms on a single inference node), that offset jumps first, within milliseconds, buffered by the silicon and cold-plate mass. The loop responds on the tens-of-seconds scale, and how far it moves depends on the usable inventory and the excess-energy integral below. At maintained flow the first loop-level symptom is the return temperature climbing; supply stays wherever the CDU holds it.

A different boundary is loss of flow. Stop circulation and only the locally coupled fluid and solid mass can absorb heat on the immediate timescale. Whether the junction throttles, shuts down or trips, and how fast, depends on the product and how it is built; Chapter 5.11 sets out how to establish it for your rack. Flow loss calls for redundancy, a protected local response and a demonstrated outcome for the GPU load that stays energized, not for controls tuning. Treat the two transients as one and you design the controls against the wrong time constant.

Redundant CDU pumps that share an unprotected feed can trip together in a grid sag, and pump redundancy does not cover that. The OCP grid-resilience specification (version 1.0.0, effective 17 September 2026) holds computational load to fast recovery but leaves cooling out. It notes that cooling drives can take minutes to restart and relies on facility thermal inertia to carry compute meanwhile. ERCOT's NOGRR282, effective 1 August 2026, takes cooling load that trips below 0.35 pu out of the baseline its recovery requirement is measured against. A direct-to-chip loop has little inertia once secondary flow stops, and stored facility water does not keep the cold plate cool. So any GPU load you keep running through a sag needs one of two things: protected pump power with its controls, or a tested bridge of thermal inventory and load shedding that caps or parks the GPUs before the junction limit and holds until cooling returns. That does not mean a UPS on every pump. NVIDIA's DSX reference design (August 2026) backs only the Core portion of its central utility building with UPS and leaves GPU-side plant cooling unbacked, but that one design does not set the power boundary for every CDU pump. Chapter 4.10 covers the grid ride-through limits.

This inversion catches operators migrating from air: a delay the old hall survived can throttle the new rack when local heat rises faster than the measured control and transport response. Measure those timescales, because calling a hall air or liquid does not tell you which one is faster. The HPE GB200 NVL72 loop is unforgiving because the coupling is tight. It needs ~140–220 L/min of secondary-loop flow per rack on the guide's water-property heat balance across OCP's 7.5–12 °C rise (PG25 at OCP's 1.25–2.0 L/min per kW gives ~144–230 L/min). Vendor reference designs run roughly 12 °C rise and specify ~130–140 L/min for the same ~115 kW liquid share (QCT up to 130 L/min, nVent about 138). QCT's 110 L/min figure is its sidecar's rating at 75 kW, not a rack requirement. Flow and delta-T trade directly, so never quote one without the other. The die sits a small, power-dependent offset above supply, and the rack accepts supply up to ~45 °C, so the closer you run to that limit the less headroom you hold. A loop that lags a load step costs you throttled GPUs. The control system now lives in the goodput path.

The CDU is where the control decision physically happens. Its two actuators are the pump VFD, which sets flow, and a modulating valve on the facility-water side of the heat exchanger, which sets how much heat the TCS loop rejects and therefore the secondary supply temperature. Read the installed valve arrangement, two-way or three-way, from the manufacturer’s piping schematic. A two-way valve varies facility flow with demand, whereas a three-way arrangement bypasses instead and can keep that flow roughly constant. Chapter 5.13 covers hydraulic sizing. Both actuators have slew limits, the maximum rate at which they may change. Get from the CDU manufacturer the allowed pump-RPM change, valve travel per minute, flow, pressure and steady-state supply-temperature band. Ask too for the conditions that band is rated at and a demonstrated transient excursion and settling response. Tune against those limits and against the actual transport delay and duty response. A predictive-control experiment on one vendor’s unit does not set another unit’s actuator limits. The limits also decide which layer can answer a step. ProphetStor's predictive optimizer chooses setpoints inside what it describes as vendor (OCP) slew limits: pump speed may change by ±3% RPM per minute and the valve may travel 10% per minute. At those rates the valve covers under 2% of its stroke during the 10 s that follow a step. Control held to those limits can only pre-position the loop ahead of a load change it can predict. The step itself must be soaked up by stored thermal mass or met with a power cap. The local PID can take it only on a unit whose own actuators are rated to move faster, and even then its correction arrives late, behind the loop's transport lag.

The slew limit cuts both ways. Set it too slow and the loop falls behind a sustained load step: the return comes back hotter, the heat exchanger rejects less than the racks produce, and over tens of seconds the accumulated energy drags secondary supply above setpoint until the die's headroom is spent and the GPUs throttle. That is lost goodput, the thing a frontier training run can least afford. Too fast a limit risks the reverse: valve and pump can overshoot the correction. Seconds later the sensor sees that over-correction, and the controller swings back the other way. The loop hunts: a sustained oscillation in flow, temperature and pressure that wears valve seats and pump bearings, fatigues quick-disconnects and hoses with pressure cycling, and swings die temperatures around the throttle threshold so the GPUs oscillate between full and capped. Hunting is the cooling-controls failure mode catalogued in Appendix F. Diagnosing it means separating tuning faults, such as excessive gain and transport delay, from hardware faults such as valve stiction or a failed sensor, and from unstable hydraulic authority.

The transport lag in the callout above is why this is hard: any loop with a long dead time between actuation and feedback tends to oscillate when its gain is too high. The classical fixes apply. Lower the proportional gain, lengthen the integral time, add a setpoint deadband so small excursions do not chase the actuator, and enable anti-windup so the integrator does not saturate while the valve is slew-limited at its travel cap. The next lever is to stop relying on feedback alone. Add feedforward where a timely load signal improves the response, in tests, by enough to repay the telemetry it needs and the failure modes it brings.

The three strategies are a spectrum, and real facilities blend them. Where a dense training cluster's load steps come large and fast, and are synchronized and predictable, feedforward answers a timing problem: it acts before delayed temperature feedback arrives. That holds only while its load signal is timely and its measured response beats the tested feedback-only case. A training step is a repeating waveform. The cluster slams to full power during compute, drops during a collective or a checkpoint, and slams again. If the controls can see that waveform, through GPU power-management telemetry, the scheduler or even the BMC power signal, they can feed the valve and pump positions forward so the coolant is already cold when the heat arrives, instead of chasing it. In ProphetStor's disclosed integration test, a 20-job mixed workload on a single eight-GPU Supermicro server, mapping coolant flow to measured GPU power cut pump energy 16–18%. That test did not establish the supplier's separate 22–28% figure for total CDU energy. The bottom row, buying your way out with thermal inertia, is the fallback when even feedforward cannot keep up. It is also where the cooling and electrical problems meet.

The cleanest way to make a cooling-controls problem tractable is to shrink the heat step at its source. GPU power capping does that, while the BBU/BESS spine in Chapter 4.5 smooths the upstream electrical boundary without changing chip heat. Coordinate the levers by what each actually changes.

GPU power capping is the fastest and cheapest. The accelerator’s own power-management firmware can clamp or ramp-limit board power, smoothing the di/dt at the source. NVIDIA’s Vera Rubin NVL72 power shelves hold about 400 J of capacitor storage per GPU at the rack boundary (NVIDIA design figure, March 2026), and that storage smooths electrical power, not on-die heat. Capacitors, BBU and BESS can flatten upstream draw, but the silicon still makes the same heat at the same power. Chapter 4.5 decides where they sit and what they must do. A cap is also the graceful-degradation lever for cooling. When a thermal excursion is developing, throttling compute a few percent to hold die temperature is almost always cheaper than letting the loop chase it into instability or, worse, letting a chip trip. That is the goodput decision: a small, deliberate cap costs a sliver of throughput, and an uncontrolled excursion costs a restart.

Battery-backed units (BBU) and facility BESS absorb the electrical transient so the grid sees a smoother draw. NVIDIA's production-BESS guidance for AI factories names transient absorption and power smoothing as first-class BESS roles alongside ride-through, with closed-loop state-of-charge control sized to the workload's transient signature. Be precise about what that buys thermally. Storage changes where the electrons come from, not how much heat the silicon makes: the rack's heat flux follows GPU power exactly, wherever the power is sourced, so only the power cap above flattens the thermal waveform. What the BESS buys the cooling plant is indirect but real. It keeps upstream electrical disturbances from becoming compute disturbances, so the thermal profile stays the workload's own.

The coordination decision is who arbitrates. If GPU capping, BBU smoothing and CDU controls each react independently to the same slam, they can fight: the cap reduces heat just as the CDU finishes slewing colder, and the loop overshoots cold toward the dew-point floor (below). The mature pattern is a supervisory layer that knows the load forecast and sequences the responses, capping and smoothing first and feeding the cooling forward to the post-cap heat profile, rather than three reactive loops racing each other. Chapter 4.5 owns the electrical-spine design; Chapter 14.7 owns the in-operation power-and-thermal management policy.

Minimum commanded supply = 20.0 + 0.5 + 0.2 + 0.8 + 0.5 = 22.0 °C. At that command the budget holds. If the local dew point rises while the command stays at 22.0 °C, the crossover is 20.0 °C: above it the 0.5 K reserve is no longer met, so raise the permitted command or switch to the tested fallback before colder surfaces are exposed. Check the coldest surface and the startup state, not only the CDU sensor. The data-center chapter of the ASHRAE Handbook—HVAC Applications (2023, Chapter 20) says liquid circulating in a liquid-cooled rack must stay above the dew point. Chapter 5.7 sets the fluid and equipment limits and 13.5 the installed tests.

The dew-point floor interacts awkwardly with the warm-water trend. Everything in Part 5 pushes coolant warmer, with 30 °C-plus facility water to maximize free cooling and heat reuse (Chapter 5.1), and warmer supply sits comfortably above any realistic dew point, so condensation risk falls as supply temperature rises. But warmer supply also leaves a smaller margin between coolant and the die's throttle limit, so the loop forgives a transient overshoot less: there is less headroom to absorb a momentary lag before the GPU throttles. The warm-water decision is therefore a transient-stability decision. It trades condensation risk down for throttle-margin risk up, and the control loop has to be tuned for the regime you actually run, near the middle of the operating band, not at a corner.

Hunting is a sustained oscillation, and it is preventable: it comes from unstable control or hydraulics. It can show up at constant load as well as after synchronized load changes, so passing a steady-state capacity test does not show the loop is stable. Commissioning traces should show it holding steady at fixed operating points and through load steps. The mechanism is dead-time-driven oscillation: a loop with several seconds of transport delay between actuator and sensor oscillates if its gain is high enough that a correction returns to the sensor still large enough to provoke an opposite correction. Three knobs and one architecture decision keep it stable.

• Gain and integral time. Lower the proportional gain so a single correction does not overshoot, and lengthen the integral time so the controller does not wind up faster than the loop can respond. The classic symptom of aggressive tuning is a regular swing in supply temperature, often roughly sinusoidal. Compare synchronized actuator and sensor traces to find its period and cause instead of assuming a universal multiple of transport delay.
• Deadband. A small setpoint deadband (for example ±0.3–0.5 °C, where measured noise and allowable excursion justify that band) stops the loop chasing sensor noise and micro-excursions, which drive high-frequency valve dithering and seat wear. The cost is a slightly looser steady-state band, an easy trade.
• Anti-windup. Because the actuators are slew-limited, the integrator will try to demand more correction than the valve can deliver during a fast transient. Without anti-windup the integrator saturates, and when the transient passes the loop dumps the accumulated error as a giant overshoot, straight toward the dew-point floor on a load drop. On a slew-limited loop, anti-windup (clamping or back-calculation) is not optional.
• Architecture: feedforward can lead a dead-time loop. Feedback cannot erase physical transport lag. A timely load signal can feed the expected correction forward while the local PID trims residual model error. Accept that path when it improves the bounded response and falls back safely on stale data. It is the same idea as the predictive-supervisor row in the table: the difference between chasing the load and leading it.

One more anti-pattern: controls fighting across layers. If the CDU PLC, the facility BMS and a GPU-side thermal manager each run their own loop against overlapping setpoints, they interact like coupled oscillators and the whole system hunts even when each loop is stable on its own. Decide which layer owns the secondary-supply setpoint and let the others observe, not actuate. It is the thermal analogue of the electrical arbitration problem above.

With no incremental rejection, excess energy = 60.0 × 10.0 = 600 kJ. Required effective volume = 600/(4.18 × 2.00) = 71.8 L; additional volume = max(0, 71.8 − 50.0) = 21.8 L. Add storage only if a real tank can deliver that effective volume within the interval. The result does not cover a loss of circulation.

With linear rejection recovery, the excess-energy triangle is 0.5 × 60.0 × 10.0 = 300 kJ. Required volume is 35.9 L, so the existing 50.0 L suffices and added volume is zero. The crossover is 50.0 × 4.18 × 2.00 = 418 kJ of excess energy; below it, a new tank adds no capacity you need. Measure the imbalance and the participating volume before you choose a tank over control changes. The CDU's rating will not settle it either. Addendum b to ASHRAE Standard 127-2020 (February 2026) rates liquid-to-liquid CDUs at steady operating points, with an energy balance as its check on the instruments, so it says nothing about the transient. Chapter 5.1 covers conservation and 13.5 the acceptance test for the installed transient.

A cooling plant that passes a steady-state capacity test can still fail badly under the workload's real transient profile, because the two are different physics. A Level 4 capacity proof says the plant can reject the rated heat at the rated flow indefinitely. It says nothing about whether the loop hunts when the load steps, whether the controls catch a slam before the die throttles, or whether the setpoint clamps above dew point on a fast drop. Those have to be demonstrated, which is why the integrated-systems-test (IST) for a liquid-cooled hall must include transient cases, not just the load-bank-at-full sequence.

The transient acceptance set is short and specific. A synchronized step test drives the load from low to full and back as fast as the load banks or a real workload allow, and confirms the controls catch it without throttling or overshoot. An anti-hunting demonstration holds the load at the worst-case operating point and confirms the loop settles inside its band without sustained oscillation. A dew-point excursion test drops the load fast and confirms the setpoint clamp holds secondary supply above the measured dew point throughout. A common-source pump-power-loss test cuts power to the CDU pumps while the GPUs run at load. Record flow, junction temperature, how fast the system detects the loss and caps or shuts down, and the restart through the full bridge duration. Pair these with the metering that makes them observable in operation. Per-rack power comes from the electrical metering of Chapter 4.12; the flow, supply-temperature and dew-point signals are a thermal instrument schedule you specify here, covering sensor placement, accuracy, sample rate and timestamp alignment to the power stream, because no electrical meter delivers them. Cooling acceptance and CDU commissioning are in Chapter 13.5; the full Level 5 IST and failure-mode demonstration, including the pump-power-loss test, are in Chapter 13.6.

Set the control bandwidth and the buffer from the actual delays and excess-energy history, and keep the local limits in force. Extra inventory helps only while the heat can reach it. A predictive supervisor has to beat feedback alone on a matched test to justify its cost, and it loses authority when its inputs fail.