CDUs & the Secondary Loop

Direct-to-chip liquid cooling (Chapter 5.4) gives you a closed loop of clean, conditioned coolant running through cold plates a millimetre above a 1.2 kW die. The building supplies facility water, chilled or warm, conditioned for its own equipment. An open-tower circuit can carry treatment chemicals and dissolved solids the cold plates would not tolerate, and a closed circuit has its own pressure and chemistry limits. The two fluids must exchange heat without ever mixing. The box that enforces that separation, and also pumps, filters, monitors and controls the clean side, is the Coolant Distribution Unit (CDU). It is the most operationally consequential box in a liquid-cooled hall, and nobody outside the mechanical team thinks about it until a pump trips and a training run loses a thousand GPUs to a thermal throttle.

The CDU defines two loops. The primary or facility water system (FWS) is the building side, running out to the chillers, dry coolers or towers (Chapter 5.8). The secondary or technology cooling system (TCS) is the clean side, running from the CDU heat exchanger to the in-rack manifolds and cold plates (Chapter 5.4). The CDU is the membrane between them: a brazed-plate or shell-and-tube heat exchanger, a redundant pump set, a filtration package, make-up and expansion provision, and a controls stack that holds TCS supply temperature, flow and pressure to the accelerator vendor's limits. ASHRAE TC 9.9 and OCP both codify this FWS/TCS split, and it makes everything downstream tractable. The facility side can be warmer, more variable and less clean than the cold-plate circuit, within its own equipment limits, while the technology side stays tightly controlled because the connected GPU hardware and its warranty require it.

The membrane raises four decisions: where you place it (the four CDU architectures), how big and how redundant it is, what fluid and filtration run on the clean side, and how you control dew-point margin and detect leaks. Each carries a downstream cost in stranded megawatts, blast radius or throttled GPUs.

A CDU performs five functions, and you select it against all five, not just the headline kW.

• Isolate. The heat exchanger keeps facility water (FWS) and technology coolant (TCS) physically separate. While the plates are intact the two inventories stay apart, though a leak or pressure loss can still interrupt heat transfer. The real failure is a perforated or fatigue-cracked plate: a cross-leak joins the two circuits in whichever direction the differential pressure pushes. Conductivity, level and dP trending are therefore the cross-leak alarm, and what the facility side carries (a closed treated circuit or open tower water) decides what a breach costs you. This isolation is the box's reason for existing.
• Pump. The CDU drives the secondary loop. It supplies the flow and head to push coolant through the in-rack manifolds, the tray-level quick-disconnects (at least 54 on a GB300 NVL72, which has 27 liquid-cooled trays) and the cold plates, against their pressure-drop budget. For the HPE GB200 NVL72, a guide water-property heat balance gives ~140–220 L/min through the secondary loop at OCP's 7.5–12 °C design rise across its 115 kW liquid load. PG25 at OCP's 1.25–2.0 L/min per kW gives ~144–230 L/min. A row or central CDU multiplies that across many racks.
• Control temperature. The CDU modulates a facility-water control valve and pump speed (VFDs) to hold the TCS supply temperature at setpoint inside the GPU vendor's acceptance limits. For GB200-class racks, QCT's QoolRack acceptance maxima are a 45 °C liquid inlet and a 65 °C return, separate limits and not an HPE figure. Subtract a 3–5 K CDU approach from that inlet and the facility water can be no warmer than ~40–42 °C (Chapter 5.7). NVIDIA's own Vera Rubin POD design feeds the CDUs 41 °C facility water and supplies the racks 45 °C coolant (NVIDIA, March 2026). Select the TCS supply setpoint from the rack vendor's approved inlet-temperature and flow limits, verify the required dew-point reserve, and model annual plant energy; higher setpoints use thermal margin but can extend economizer or dry-cooling hours. The exchanger's approach temperature, how close the TCS supply can get to the FWS supply, is a first-order selection criterion. Several vendors list units built to the OCP Project Deschutes design (contributed by Google) at 2 MW with a 3 °C approach, which buys warmer facility water and more free-cooling hours.
• Filter and condition. Full-flow filtration protects the connected equipment at its required cleanliness, and a separately specified side-stream stage can polish the inventory. The CDU also hosts make-up, de-gassing and often chemical dosing for the TCS.
• Monitor and protect. Flow, supply and return temperature, differential pressure, conductivity and leak sensors feed the BMS/DCIM and, increasingly, the cluster scheduler. The CDU is the sensor platform that tells the cluster whether its coolant is in spec before a chip throttles.

There are four common CDU placements. L2L separates FWS from TCS, while L2A transfers TCS heat to air, and the choice is the defining decision of this chapter. The placements differ on whether facility water reaches the rack at all, on how many racks share one CDU (the blast radius), on serviceability, and on how well they absorb the density ramp from 132 kW to 600 kW racks.

In-rack liquid-to-air (L2A). A self-contained CDU sits inside or atop the rack and rejects the captured heat back into the room air through an air-cooled heat exchanger. There is no facility water at the rack: the only utilities are power and room CRAH/CRAC capacity. That makes it the brownfield option, since you can land a liquid-cooled rack in a hall with no building-water connection at the rack, though its local liquid circuit still needs containment and commissioning. The cost is capacity and efficiency. An L2A unit is bounded by what the room air can absorb (commonly ~22-100 kW depending on the unit), it dumps the heat back into a hall that now needs more air cooling, and at a ~15 °C approach it is the least thermally efficient option. Row-format L2A exists too: Vertiv's CoolChip CDU 350 delivers 350 kW at a 15 °C ATD, with 27 °C room air, from a row-based unit piped to nearby racks. L2A is a bridge, not a destination (it overlaps with AALC in Chapter 5.3).

In-rack liquid-to-liquid (L2L). A CDU in the rack exchanges heat to facility water. Facility water now reaches the rack, but the blast radius is one rack: a CDU failure throttles only its own GPUs. Early units were limited by the few U they could occupy, at ~50-110 kW. Nidec now lists 4U units at ~200–250 kW, and its 250 kW model is specified for a single GB300 NVL72. It has also shown a 300 kW prototype aimed at mass production in Q1 2027 (all vendor ratings). Pick an in-rack unit by its duty at your rack's liquid load, flow and available external head, not by the nameplate. The 300 kW prototype rating sits under the 330 kW cabinet TDP that NVIDIA's DSX reference design assumes for Vera Rubin NVL72 (Chapter 5.1). Cabinet TDP is not CDU liquid duty, so check the rack's liquid load against the unit's rating at your flow, approach and external head before you rule it in or out. Capacity is only half the in-rack case. The other half is blast radius, service access and how many facility-water drops you will run, and you pay for every small CDU and every drop.

Row-level (in-row) L2L. One larger CDU serves a row of racks. This is the 2026 mainstream for dense AI rows: an MW-class cabinet feeds 10-20+ racks through a row manifold. Row and end-of-row units on the market span ~600 kW to 2–3.5 MW. Vendors rate them at different flows and approach temperatures, and the sizing section below shows how to compare them. The OCP Project Deschutes design, contributed by Google, is a 2 MW unit. You amortize one high-quality, internally redundant CDU across the row and keep the CDU-to-rack run short. The allowable length follows from the CDU's available external head, as the worked example below shows. The cost is blast radius and concurrent maintainability: lose the row CDU without N+1 and you lose the whole row, so the redundancy posture below is non-negotiable.

Central / plant-level L2L. A few very large CDUs (or a CDU plant) serve a whole hall or pod from a mechanical room, distributing TCS through a building-scale secondary loop. This maximizes amortization, redundancy pooling and serviceability, because the CDUs sit outside the white space. The costs are the largest blast radius, the longest secondary runs (more pumping head, more fluid volume, harder hydraulic balancing) and the heaviest commissioning. It blurs into the facility-loop design of Chapter 5.7.

The L2L-versus-L2A trade comes down to capacity at scale. Vertiv rates its CoolChip CDU 2300 L2L at up to 2.3 MW and its CoolChip CDU 350 L2A at 350 kW at 15 °C ATD. That 6.6× compares two nameplates rated for different applications, and the L2A unit is capped by the room air it rejects into. It does not show a 6.6× advantage at matched temperature, flow, head or energy boundary. L2A wins only where its one advantage, no facility water at the rack, is worth more than capacity and PUE, which is the brownfield retrofit case (Chapter 5.10). For a purpose-built training hall whose room cannot accept the L2A duty but whose water path can, the decision narrows to row-level vs central L2L. That is a blast-radius-versus-amortization argument, not a thermodynamics one.

Sizing a CDU is a four-variable problem, not a matter of picking the kW that matches the rack. The four are heat load, flow, approach temperature, and pressure/head, and you must satisfy all of them at once at the worst-case facility-water temperature, not the design-day average. A CDU rated '2 MW at 18 °C facility water' may deliver far less on the hottest day, when the tower can only supply warmer water. Treat the rating as a curve. Size against the corner of that curve your climate and heat-rejection plant actually deliver (Chapter 5.8).

Line up CDU quotes at one shared operating point rather than by nameplate. All of these are vendor ratings. Ask for the duty at your flow per kW and approach. Ask for the duty in the redundant state, which can fall below maximum duty. Motivair's WCDU shows why. Its rating is a maximum of 3.5 MW at 1.5 L/min/kW, or 2.5 MW at 2.0 L/min/kW: 5,250 against 5,000 L/min, almost the same flow, yet 1 MW apart. The pumps move nearly the same coolant. The extra megawatt comes from letting it rise about 10 K instead of about 7.5 K on PG25, which hands the exchanger a hotter return, so your rack's flow per kW decides which rating you actually get. Motivair offers N+1 only up to 2.5 MW. Ask for flow at the external head you will actually have; LiquidStack specifies its CDU 2.X as 3,750 L/min at 3.5 bar, with no MW rating.

Ask for the approach at your design facility-water temperature, where Project Deschutes-design units are 2 MW at 3 °C. Check the flow basis behind that 2 MW too. The spec pairs it with 500 GPM, or 0.25 GPM/kW: about 0.95 L/min per kW, a rise of about 15 K on water. OCP's cold-plate guideline is a 7.5–12 °C rise, about 1.25–2.0 L/min per kW of PG25, typically 10 °C at 1.5 L/min/kW. At 1.5 L/min per kW, 500 GPM serves only about 1.26 MW of racks; giving 2 MW of such racks their flow takes about 3,000 L/min (about 790 GPM) at the head they need. The Deschutes 1.0.0 spec also budgets 74 kW of electrical input at that duty and 6,900 lb wet on a 65 × 42 in footprint, so the CDU belongs in the power and floor-loading schedules too. Ask for all four quoted to ASHRAE 127 Addendum b, described below, and get the ship date too, since several units launched in 2026 ship in 2027.

The accelerator sets the flow. Its approved fluid, its liquid duty and the temperature rise you choose fix the number, and it must stay above the OEM's minimum-flow curve. The head side is set by the worst-case hydraulic path, the longest and most restrictive run from CDU through row manifold, in-rack manifold, UQDs and cold plates. Get it wrong and the far racks starve (the branch-flow requirement in Chapter 5.7 is a CDU-plus-manifold co-design problem, not a manifold-only one). Approach temperature is the efficiency lever: a tighter exchanger approach (3 °C vs 5 °C) lets you run warmer facility water for the same chip inlet, which buys free cooling and heat reuse (Chapter 5.9).

The expensive call is the oversize factor against the density ramp. Size a CDU plant exactly to HPE GB200 NVL72 rows at their 115 kW liquid duty and nothing is held back for VR200 NVL72, whose cabinet TDP NVIDIA's DSX reference design puts at 330 kW, or for the ~600 kW Rubin Ultra / Kyber planning figure on NVIDIA's roadmap (Chapter 5.1). Kyber's date is contested: NVIDIA targets H2 2027, while SemiAnalysis reported 2028 in July 2026. Buy the headroom now and you carry idle capital until your load ramp fills it, plus whatever part-load energy penalty your module and control curves show. Buy it later and you re-plumb a live hall mid-life. There is no clean answer: whether you can economically future-proof from 120 kW to 600 kW loops, or a 2-3 year re-fit cycle is now structural, is open. The defensible move is to oversize the irreversible substrate (pipe risers, valve stations, mechanical-room footprint, facility-water capacity) and keep the reversible CDU modules matched to the current generation, the same reversible-vs-irreversible discipline as the slab and power chain in Chapter 1.1.

The February 27, 2026 Addendum b to ASHRAE 127-2020 supplies a CDU test and rating method. Specify the application temperatures, fluid, flow, available external head and electrical boundary together, and say how internal heat and losses enter the selection. A rating done to this method still does not count as acceptance for your project.

Total exchanger duty is 320 + 3.00 = 323 kW. With a heat-capacity rate of 32.0 kW/K, the CDU hot inlet is 35.0 + 323/32.0 = 45.1 °C and the FWS return is 30.0 + 323/32.0 = 40.1 °C. At the exact unrounded ports both terminal approaches remain 5.0 K. Each 330 kW thermal selection exceeds the duty by 7 kW. The auxiliary heat is counted once, and neither the 320 kW IT load nor the 330 kW selection is the CDU's electrical draw.

A fails hydraulic duty: 200 < 213 kPa. B has about 27 kPa of head margin, and its VFD is trimmed to the required operating point. Select B for the current piping, and carry the full 323 kW rejection requirement to 5.8. Do not subtract internal CDU loss from a rating that is already external head. Two B units establish steady capacity only; their transfer, power and shared headers still need 13.5 acceptance.

Flip. A becomes hydraulically eligible at an external duty of 200 kPa or less, if its thermal and control conditions still hold. The 80.0 mm common-header alternative in 5.13 screens at 191 kPa and crosses that boundary, trading pipe space and capital against CDU selection. Use the ASHRAE CDU rating method to keep the comparison at identical ports.

Because a CDU sits in the goodput path, its redundancy has to be justified from its maintenance and fault states. Where you put the redundancy is a fork with different costs and a different blast radius.

Internal pump redundancy (N+1 within one CDU). Most quality CDUs ship with dual or N+1 pumps, often with redundant power feeds per pump circuit (the Project Deschutes 1.0.0 spec puts each of its two N+1 pumps on its own VFD and asks for at least 1N input-feed redundancy, with 2N preferred), so a single pump or VFD failure does not stop flow. That covers the most common failure, a pump. Pumps are not the whole box, though. Most commercial CDUs still have one exchanger, one cabinet and one controller. Duplicating only the pumps leaves every part that is not duplicated exposed, and that can include the feed and the isolation boundary. Google says its Deschutes design duplicates both pump and exchanger, and that its in-row CDUs run on UPS. Google's April 2025 report claims roughly 99.999% availability for those CDUs since 2020. It does not say how it defines availability, and no outside party has audited the number. Where the box is single, lose it and you lose the row.

Unit-level redundancy (N+1 CDUs). A spare CDU per row (or per N rows) covers whole-box failure and allows concurrent maintenance: you can valve out and service a CDU without dropping the row. That holds on the cooling side only if isolation, controls, power and the surviving flow still deliver the required service, and where the row is one training job they must. A Tier III or IV claim still needs the complete topology and certification boundary in 12.1. The cost is a redundant MW-class cabinet, its facility-water drops and the floor space.

UPS-backed pumps for thermal ride-through. This is the subtle one. A DLC loop coasts only on the usable thermal inventory that maintained circulation can reach. When facility power is lost the coolant stops moving, and the rack's throttle, controlled-shutdown, emergency-shutdown and no-response states and their timing must come from OEM transient data, controls tests or an engineering calculation. Size the protection path, including whether CDU pumps or the rejection plant need UPS/BESS, to that validated response and to the power-transition case. Skipping this is the cooling-side equivalent of skipping ride-through on the GPUs. See the ride-through treatment in Chapter 5.8 and Chapter 5.12.

The most important CDU control setpoint is the secondary-supply temperature relative to the white-space dew point. Coolant runs through manifolds, hoses, cold plates and quick-disconnects, much of it exposed to room air. If any wetted surface drops below the room's dew point, moisture condenses on it: water beading on hoses above energized rack busbars and GPUs, including the 50 V busbar in Lenovo's GB300 NVL72 rack (Lenovo Press LP2357, 30 August 2026). The rule is absolute: hold the TCS supply temperature above the white-space dew point at all times. That keeps the cooling 100% sensible (no condensation, no latent load), and it is one of the reasons warm-water cooling is winning. A ~45 °C supply has a wide margin in a room whose measured dew point is well below it, provided startup and control undershoot preserve the gap.

This couples the CDU to the room. Dew point depends on white-space humidity, so the CDU's minimum supply setpoint is a function of air-side environmental control and any humidification policy. Run the room too humid and you raise the dew-point floor, which forces a warmer minimum coolant supply and gives up cold-plate margin. Run a chilled-water trim loop too aggressively and you risk dipping below dew point on a transient. The margin is therefore a negotiated setpoint between the mechanical (CDU) and environmental (CRAH/RH) controls, the kind of cross-loop interaction that Chapter 5.12 treats as a stability problem.

Leaks and pump trips are the fast failure modes. The slow one is the secondary loop fouling itself over a 10–15 year life, and it is the one most likely to be under-engineered at commissioning because it takes years to show. There is no universal TCS fluid. PG25 is one choice, made per project, and its name gives a concentration, not a full chemistry specification. Your fluid specification must name the qualified product, the percentage basis, water quality, the inhibitor and biocide or biostat package, wetted-material compatibility, the operating-temperature range, sampling limits, the replenishment or replacement plan, and the OEM's warranty requirements. Compared with water at the same temperature, glycol generally raises viscosity and lowers specific heat, which raises pumping power or required flow. Pick the concentration and controls from the fluid, equipment and maintenance program you have approved, not from a universal default.

Three chemistry failure modes work on the loop over its life, and the CDU is where you manage all three:

• Galvanic corrosion. Mixed metallurgy (copper cold plates, aluminium components, steel pipe, brazed-plate HX) plus a conductive fluid sets up galvanic cells. Corrosion inhibitors are dosed into the fluid but deplete, so this is a maintenance cadence, not a one-time fill. Project Deschutes 1.0.0 limits its CDU's wetted materials to stainless steel, copper and EPDM and bans nickel-plated brass from the secondary side, a list worth writing into your own CDU purchase specification.
• Biofilm. Warm water is a microbial habitat, and biofilm fouls cold-plate microchannels and degrades heat transfer. The warm-water roadmap raises this risk just as it lowers the chiller bill, so biocide and biostat management matters more as inlet temperatures climb.
• Particulate / additive depletion. Particulate from manufacturing residue, wear and corrosion clogs the tightest cold-plate channels. Full-flow filtration (commonly ~25–50 µm) catches bulk particulate, while a separate side-stream stage can polish to sub-micron on premium units. Both stages load and must be serviced, and additives are adjusted only through the supplier-approved program in 5.7.

The industry has not answered with public data how often TCS fluid must be replaced over a facility life, or what the long-run material-compatibility failure rates are. Vendor claims need independent validation. The practical consequence is that filtration spec, fluid-sampling cadence and inhibitor replenishment are commissioning decisions, and they decide whether you are servicing clogged cold plates in year three. Get the fluid program wrong and the failure shows up as a slow, fleet-wide rise in coolant delta-T and a quiet loss of goodput, the hardest kind to diagnose. Chapter 5.7 defines fluid acceptance; 13.5 accepts the installed cooling system.

A modern CDU is a controls node, and how it integrates decides whether the cooling can defend the compute or only report on it afterward. At minimum it modulates pump speed (VFDs) and the facility-water control valve to hold TCS supply temperature, flow and differential pressure at setpoint, and it streams supply and return temperatures, flow, dP, conductivity, filter state, coolant level and leak status to the BMS/DCIM. The VFDs matter beyond control. Ultra-low-harmonic drives (Project Deschutes 1.0.0 names ABB's ACH580-31 and keeps current THD under 20% per IEEE 519 guidance) keep the CDU from polluting facility power quality, a real concern in a hall with dozens of CDU pump and fan drives beside the rest of the mechanical plant. A CDU's MW rating is thermal duty, not its drive's electrical input (Chapter 4.12 covers power quality).

What separates a goodput-aware facility from a facilities-aware one is whether CDU telemetry reaches the cluster scheduler, not just the BMS. GPUs report their own temperature and clock-event reasons. Feed the scheduler the row's coolant-inlet temperature as well. It can then see the drift toward the throttle threshold and drain or de-rate the job before a throttle slows the synchronized steps, or a thermal event interrupts the run. If the CDU data dies in the BMS, the scheduler sees the problem later, from GPU temperature and clock telemetry alone. This is the cooling-side case of the DCIM and observability argument in Chapter 14.2 and the autonomy ladder in Chapter 14.13: the CDU's value is holding setpoint and also being a sensor the cluster can act on.

The CDU is the hinge of the thermal stack. Upstream are the cold plates, in-rack manifolds and quick-disconnects (Chapter 5.4), whose pressure-drop and flow-balance budgets the CDU must meet. Downstream are the facility water loop and its temperature class (Chapter 5.7), then heat rejection (Chapter 5.8) or heat reuse (Chapter 5.9). The CDU's approach temperature is the dial that connects them: a tighter approach lets the facility loop run warmer, which makes year-round free cooling and high-grade heat capture possible. Choose the architecture and approach badly and you have capped your PUE, free-cooling hours and heat-reuse grade before the first GPU boots.

One dial is outside the design: manufacturing capacity. Liquid-cooling plant is now reserved like switchgear. nVent leased its third liquid-cooling factory expansion in three years (160,000 sq ft in Blaine, MN; production 1H 2027), and reports >2 GW of liquid cooling already deployed (Jul 2026). A CDU selection that ignores the vendor's build slots is a schedule decision made by accident.

Ownership has moved too. Several CDU makers now belong to large power or HVAC groups: Motivair to Schneider Electric, LiquidStack to Trane (completed 3 March 2026) and Boyd Thermal to Eaton (a $9.5 billion deal closed in March 2026). Schneider and Eaton can bundle switchgear, UPS and CDUs under one warranty, leaving one supplier accountable for power and cooling together, and Trane can sell CDUs alongside its chillers. The bundle buys coordination but gives up price leverage, and because power and cooling now ship from the same delivery queue, one slip holds up both. If you take it, keep a second CDU vendor tested on your rack (Chapter 2.3 covers lead times and owner-furnished equipment).

Choose the CDU whose thermal duty, external head, controls and isolation all work at the required operating points. A nominally larger exchanger cannot make up missing pump head. Preserve the failure-state duty in the topology you choose, then hand the installed response to Chapter 13.5. NVIDIA's DSX Ready label (Vertiv's 2.3 MW and LG's 2.6 MW CDUs, September 2026) shows that a CDU passed NVIDIA's self-qualification suite for functional requirements. It does not size the unit for your flow and head, prove the redundant state, or replace integrated acceptance under 13.5.