Capacity, Power & Thermal Management in Operation

By the time a facility reaches steady-state operation, its most expensive and least reversible decision is already made and paid for: how many megawatts it can energize. The interconnection slot is won or lost, the transformers are humming and the cooling plant is sized. What remains is a discipline the chip-bound era never needed: run a fixed power budget hot enough to pay back its capital, but not so hot that a workload transient trips it. This is the day-2 face of the power-bound thesis. The constraint has moved from how many GPUs you can buy, or even cool, to how completely you convert energized megawatts into goodput, request by request and step by step.

This chapter treats four operational functions as one problem because they share a single ceiling: live power management and oversubscription (running the budget full), transient management in operation (keeping the swings inside the budget), stranded-capacity and thermal operations (recovering megawatts the design left unused), and capacity planning, workload-aware operations and demand response (matching the ramp and the grid). The physics of the transients is covered in Chapter 4.5 (storage and ride-through) and Chapter 5.12 (cooling-controls dynamics). This chapter manages them in operation.

A facility provisioned for 100 MW of IT load rarely draws 100 MW. Nameplate is the sum of every device’s worst-case rating, and real workloads almost never peak together. The gap between nameplate and realized peak is power headroom. The central operating question is whether to sell it: to oversubscribe, deploying more racks than the budget supports at nameplate, on the bet that they will not all peak at once. The bet can pay. In a simulation on Microsoft’s inference traces, the POLCA study fit roughly 30% more servers under the same power budget with negligible performance loss. The bet can also fail: the budget overruns, the protection scheme fires, and a statistical near-miss becomes an outage.

Whether to take the bet depends on whether your workload leaves headroom to sell, and here the two archetypes diverge sharply. In a tightly phased synchronous job, the compute, collective and checkpoint phases can correlate demand across ranks, though all-reduce does not always coincide with peak GPU board power. That correlated swing leaves almost nothing to overbook: POLCA’s measured training fleet had only ~3% headroom. Inference behaves differently when request arrivals are dispersed. Across nodes, compute-bound prefill and memory-bound decode then fall into different phases, so the aggregate can stay below the sum of peaks; POLCA measured ~21% headroom across its inference fleet. Correlated prefill or synchronized arrivals can erase that margin. The same overbooking ratio can add usable capacity under one measured profile and breach a feeder budget under another.

The table applies two tests: measured peak correlation sets the headroom, and interruption and recovery cost set the price of using it. Batch inference can absorb deferrable work when deadline slack exists, but a synchronized batch launch can still create a sharp peak, so price the actual queue and waveform before you assign it oversubscription risk. Sophisticated operators do not pick one posture. They run a heterogeneous budget where interruptible, uncorrelated workloads absorb the overbooking that synchronous, latency-bound workloads cannot. → The disaggregated RL picture is in Chapter 1.4; the inference burst profile in Chapter 1.3.

Oversubscription is a bet, and every bet needs a stop-loss. In a power budget the stop-loss is the power-cap hierarchy: layered enforcement points that hold realized draw within the workload range you have validated. The hierarchy has four levels, each slower and broader than the one below it. At the GPU/node level, the accelerator’s hardware and firmware power management reduces clocks within its specified measurement and actuation limits; a microsecond response must be documented for your platform, not inferred from a BMC command. At the rack/PDU level, intelligent rack PDUs and the BMC enforce a rack budget across nodes. At the row/lineup level, the DCIM control plane allocates a shared budget across racks on a common busway or RPP. At the facility level, the EMS holds campus draw under the interconnection limit and any demand-response obligation.

The design question is where the binding cap sits and how fast it engages. Cap too low and you leave goodput on the table, with GPUs throttled that the budget could have fed. Cap too high, or too slow, and a correlated transient breaches the budget before the control loop catches it. Speed is the hard part: participating ranks can produce a millisecond-scale step as compute resumes after a collective. Measure the overlap, the step size and the response deadline of the installed controls. Time-critical response belongs in validated local device controls and the designed power and energy buffers.

The rack BMC issues supervisory cap commands on an out-of-band loop that runs in seconds to tens of seconds. The DCIM and EMS layers set budgets for the fast layers to enforce and do not react to transients themselves; an EMS that senses and acts more slowly than the transient must leave that duty to the local path. For the platform you actually bought, specify measurement delay, command latency, actuator response and behavior after control-plane loss. Independent electrical protection still covers faults and loss of control. Keep oversubscription inside the equipment’s time-current and ride-through limits: feeder safety is a protection setting, never an assumed BMC reaction time.

Trace. The supervisory response takes 1 s + 2 s = 3 s. The energy required is 4 MW × 3 s = 12 MJ, which exceeds the 8 MJ usable buffer by 4 MJ. On this rectangular-step model the longest delay the buffer can cover is 8 MJ / 4 MW = 2 s. So don’t sell the headroom: keep it until a local control and buffer design meets both the power and the energy limits. A successful BMC API call is not proof of actuation, and a breaker trip is not normal budget control.

Flip. The energy test passes only if the validated total delay is 2 s or less, or if usable energy is at least 12 MJ at the 3 s delay, with discharge power still sufficient. A local response bounded at 0.5 s would need 2 MJ, but it earns operating authority only after tests of latency, stale data, saturation and control loss. Time-current coordination and protective devices still constrain operation on top of either answer. The arithmetic is the energy integral ∫P_excessdt and the SI watt–joule–second relationship. Storage duty is sized in Chapter 4.5 and protection coordinated in Chapter 4.11.

The power transient is what makes oversubscription dangerous. A tightly phased synchronous job can pulse: compute raises demand, while exposed communication or checkpoint phases lower it. Overlap and staggered jobs reshape that waveform, and a job start, kill or reload changes the participating load, not automatically the whole facility. Workload-driven swings at fleet scale are large and abrupt, but the grid’s recent load-loss events do not show their size; they record sites transferring to backup power during voltage sags. On 2026-07-22 in Ashburn, a single normally cleared fault saw about 3.8 GW of data-center demand leave the grid that way. The earlier July 2024 Virginia case, a six-fault, 82-second 230 kV sequence, saw about 1.5 GW leave and helped drive NERC’s Level 3 alert (May 2026). Together they made grid ride-through, staying connected through a sag instead of transferring to backup, a first-order planning input.

Ride-through is already a binding rule in parts of the US, and nowhere nationally. Under ERCOT’s NOGRR 282, in force since 2026-08-01, a new Large Computational Load of 75 MW or more has to stay connected through specified voltage sags and then restore demand inside set time limits. SPP imposes comparable rules on new high-impact large loads requested from 2026-01-15. PJM has so far only proposed rules (2026-09-08), after the Ashburn event, with a FERC filing planned for November. FERC’s RD26-7-000 order (2026-07-16) tells NERC to submit its first standards for computational loads, plus registry changes, by 2026-12-31. Those standards (CLO-001-1, CLO-002-1 and CLO-003-1) cleared NERC’s final ballot, a result announced on 2026-10-15; they now await adoption by the NERC Board and approval by FERC. Neither the order nor the standards set a ride-through curve. A new campus in ERCOT or SPP that fails a qualifying sag, or restores demand too slowly afterward, is already out of compliance. Everywhere else, expect the requirement to arrive in your interconnection agreement before it arrives in a national standard.

The physics of absorbing these swings (UPS, BESS, ride-through and the controls that damp them) is engineered in Chapter 4.5 and Chapter 5.12. The question this chapter owns is narrower and continuous: how do you run the live facility so the transients you generate stay inside what your absorption layer and your grid contract can take?

The answer is a set of operating levers that shape the workload’s power profile before it reaches the power chain. Power smoothing holds a floor under idle phases (a synthetic GPU load during the all-reduce trough), so the swing the grid sees is smaller than the swing the GPUs produce. It trades a little energy for a lot of transient amplitude. On GB300 NVL72 racks, smoothing is built into the platform, so no custom job is needed. Each GPU has three programmable controls: a minimum power floor, a timer before it ramps down, and its ramp rates, set through nvidia-smi or Redfish. NVIDIA tested those controls on GB200 hardware and says the energy-storage shelf is also coming to GB200 NVL72, so confirm what your GB200 firmware and shelves support. GB300 power shelves hold 65 J of stored energy per GPU. In NVIDIA’s own test, a GB200 power shelf without storage and a GB300 shelf with it ran the same training workload in the same rack, and the storage version cut peak grid demand by about 30% (a vendor measurement). The floor is not free: in Microsoft’s simulation, a floor at 90% of TDP added 10.5% to total energy use. Treat these settings as controlled changes, and cost them in goodput-per-MWh.

Ramp-rate limiting staggers job starts and checkpoint reloads so a 100 MW cluster does not step its entire load in one second; the scheduler launches ranks in waves. Job-start ramp limits and post-disturbance recovery are different jobs, so configure them separately. After a sag, ERCOT’s NOGRR 282 gives a new large load 2 s from voltage recovery to get back to at least 90% of its prior demand. A load that transfers to backup or stabilizes its controls during a sag below 0.8 pu must start returning within 0.25 s and reach 90% within 0.5 s. SPP gives 1 s. The OCP grid-resilience specification is an industry document, not a regulation; it has been in effect since 2026-09-17 and asks for 90% of computing load back within 2 s. If the job-start ramp limit also governs recovery after a sag, the site can miss that window and fall out of compliance. Work out the recovery ramp rate with your transmission owner, build it into the platform and UPS controls, and test it during commissioning (Chapter 13.6).

Stagger and phase-offset scheduling spreads correlated peaks across time on purpose. The storage layer, UPS and BESS, absorbs whatever swing the workload levers leave. Suppressing the transient in software is cheap in energy and free in capital, but it taxes goodput and is the operator’s job. Suppressing it in hardware, with BESS sized to the swing, costs capital up front and carries no goodput tax. That split is the design decision. Most facilities do both, and the ratio is where they differ.

Stranded capacity is energized, paid-for power or cooling that cannot become compute because some other resource runs out first. In a power-bound world that is capital spent on megawatts that earn nothing. It comes in three strands, and the diagnosis matters because each one recovers differently. Confusing them wastes the recovery capital too.

• Cooling-limited. The power chain can feed the racks, but the cooling plant cannot remove the heat, so racks sit half-populated or throttled. This strand dominates in halls built for air and asked to host liquid-class density: the cooling cliff shows up as a utilization ceiling. Recovery is mechanical: more heat-rejection capacity, warmer water or a cooling retrofit. → Chapter 5.4.
• Power-chain-limited. Cooling has headroom, but a transformer, busway, RPP or breaker is the limit, often because the design reserved fault margin or redundancy capacity that nameplate planning never released to load. Recovery is electrical and partly statistical: oversubscription releases reserved headroom, and rebalancing phases and circuits recovers fragmented capacity. → Chapter 4.6.
• Fragmentation-limited. Power and cooling both have aggregate headroom, but it is scattered, a kilowatt free here and two there, in pockets too small to land a 132 kW rack. This is the most insidious strand because the DCIM dashboard shows spare capacity the placement engine cannot use. Recovery is operational: workload-aware placement, defragmentation campaigns and consolidating partial rows.

Misdiagnose the strand and you spend in the wrong place. Add chillers to a fragmentation problem and you have bought cooling you still cannot fill. Oversubscribe a cooling-limited hall and you trip the thermal protection you were already brushing against. The operational core of stranded-capacity management is reading from telemetry which resource is binding, row by row, before committing recovery capital. → The telemetry that reveals it is in Chapter 14.2.

Thermal operations is the cooling-side twin of power management: the continuous job of matching heat rejection to a load that swings as hard as the power draw. The same all-reduce that pulses the power chain pulses the heat load, and the cooling plant’s thermal mass and control loops respond more slowly than the electrical ones, which is its own hazard. While flow is maintained, coolant and metal mass may buffer a heat step. If a cooling-loop failure stops flow, what happens next is implementation-specific: for your rack, the throttle, controlled-shutdown, emergency-shutdown and no-response states need OEM transient data, controls tests or an engineering calculation. That data sets the pump and rejection protection and the redundancy topology you need.

The day-2 levers are setpoint and flow management across four separate limits: the facility-water supply class, the CDU approach, your rack’s rated TCS inlet and return temperatures, and its OEM flow and pressure-drop limits. Keep the secondary supply above the white-space dew point so cooling stays 100% sensible. The steady-state choice is how warm to run the loop. Warmer water expands free-cooling hours and crushes the cooling share of PUE (warm-water DLC plants approach ~1.1 versus ~1.3–1.5 for typical air halls, though well-optimized air can also reach ~1.1), but it shrinks the thermal margin to the throttle threshold. A GB200 that deviates from its inlet spec throttles, on a product- and workload-specific curve you measure rather than assume. Run cold and you waste compressor energy buying margin you may not need. Run warm and you operate closer to the cliff, where a transient that would have been absorbed becomes a throttle event and a goodput loss. Cooling-controls stability under these transients is covered in Chapter 5.12, the piping and CDU mechanics in Chapter 5.13, and predictive maintenance of the cooling plant in Chapter 14.5.

Capacity planning in a power-bound facility means never letting demand for megawatts outrun the supply you have energized, and never leaving so much energized capacity idle that the capital decays unused. The forecast has an asymmetric loss function. Under-provision and you turn away revenue while depreciation runs whether or not the racks are full. Over-provision and you strand energized megawatts, the costliest resource on the site. The forecast must track the density ramp, not just steady-state load. Plan the substrate for the generation you will deploy next, because transformer capacity, busway ampacity, cooling-plant tonnage and floor loading are costly to retrofit and cannot change as fast as rack density climbs. Racks went from 40 kW air-cooled to ~132 kW for GB200 and GB300 NVL72 (HPE’s nominal rating), and are heading to Vera Rubin NVL72 and ~600 kW Kyber-class. OEM datasheets rate Vera Rubin NVL72 operating profiles at 188 kW Max Q / 228 kW Max P, while NVIDIA’s DSX reference design plans each cabinet at 330 kW for facility power and cooling. A hall whose power and cooling were sized to today’s density strands its own future.

The operating choice is the reserved-capacity buffer: how much energized headroom you hold back against the next ramp step and against failures. Hold too little and the next GPU generation arrives with nowhere to land: the density-ramp trap, now an operating reality rather than a scoping one. Hold too much and you pay to idle megawatts. Forecast the ramp generation by generation, size the hard-to-change substrate to it, and keep the reversible IT fit-out matched to current draw. Reserve the headroom that is costly to retrofit (floor, water, ampacity; strengthening a slab is possible but disruptive, see Chapter 6.2) and defer the spend you can. The interconnection and energization sequencing that feeds the forecast (50–100 MW tranches, energized capacity leading commissioned load) is in Chapter 3.2; the scoping-time version of the ramp decision is in Chapter 1.1.

The most powerful lever an AI factory has over its own power profile is the scheduler. A large share of the fleet’s load is interruptible (batch inference) or schedulable (RL rollouts, off-peak training), so the operator can shape when and where power is drawn, not just react to it. Workload-aware operations treats the power budget and the job queue as one optimization: placing interruptible work where stranded capacity hides, throttling low-SLO tenants first when a cap engages, smoothing the aggregate profile by phase-offsetting correlated jobs, and shifting deferrable load to cheap or low-carbon hours.

That same flexibility makes demand response and curtailment economically real for AI loads, and makes it dangerous. Flexible load has modeled system value. Duke’s Nicholas Institute modeled how much new load the grid could take if data centers accepted limited curtailment. Across 22 US balancing authorities it found room for 98 GW if only 0.5% of annual energy is curtailed (February 2025). A follow-up in February 2026 put the drop in retail electricity prices at about 0.5–2.8% over the next decade, in stylized scenarios that run from skipping 1% of peak hours to also moving load between hours and regions. Both are model results, not observed rate cuts.

For a facility, participating means accepting curtailment events (averaging ~2 hours) in exchange for a better interconnection deal or a demand-response payment. In some markets curtailment is a condition of service instead: Texas SB6 requires new large loads to accept emergency curtailment, unpaid. Treat paid demand response and mandatory curtailment as separate obligations when you set eligibility. Everything then rides on which workloads you make eligible. Curtail batch inference or defer a checkpointable training run only after pricing deadline slack, lost work, restart, rebound demand and the contract’s baseline and notice requirements; checkpointability does not make the cost zero. Curtail an online-inference SLA or a synchronous run mid-step and you destroy goodput value that dwarfs the demand-response payment: a breached latency SLA, or a forced checkpoint-and-restart that throws away hours of correlated compute. Online inference qualifies through spare headroom or admission control provided it keeps its SLO; synchronous training qualifies only once its checkpoint-and-restart budget has been tested. Write each workload’s eligibility into the contract.

The final function folds power, thermal and workload management into one objective: useful work per megawatt-hour. PUE measures only facility overhead, and at warm-water DLC’s ~1.1 there is little overhead left to chase. The frontier has moved inside the white space, to metrics like work-per-watt and goodput-per-MWh that count whether the IT load is doing useful computation or burning power on idle GPUs, retries, throttled clocks and failed jobs. All else equal, a facility at PUE 1.1 running GPUs at 30% MFU uses about 37% more energy per unit of useful work than one at PUE 1.2 running at 45%. Under matched hardware, peak-per-watt and workload, useful output per facility MWh follows productive-time fraction × conditional MFU ÷ PUE. Multiply a separate active-duty factor only if Chapter 14.1’s clocks show it is not already included. The numerator moves over a far wider range than the denominator. Energy operations therefore reaches past the cooling plant into the cluster: raising MFU, cutting failure-induced rework, and timing flexible load to low-carbon, low-cost grid hours. By 2026 the efficiency levers that matter most are scheduling and reliability, not mechanical plant. → The full KPI and reliability-economics treatment is in Chapter 14.1.