Predictive & Preventive Maintenance of Power and Cooling Plant

A traditional enterprise data center can take a maintenance window. Drain a hall, fail traffic to the other availability zone, swap the UPS batteries on a Sunday at 03:00, and bring it back. The workload never noticed, because it was designed to be moved.

An AI factory in a pre-training run is the opposite kind of machine. It is one tightly coupled job spread across thousands of accelerators that all move at the speed of the slowest straggler. It draws tens of megawatts through a power chain and rejects the heat through a liquid loop that cannot be paused. There is no other zone to fail to. The job does not migrate; it checkpoints and restarts, and every restart is goodput you will never recover.

The maintenance problem is therefore whether you can service the plant without dropping the job. That answer was largely decided at commissioning, by whether you bought a redundancy topology that lets you transfer the load to a tested alternate path, isolate the equipment being maintained, and establish a safe work condition on it before anyone opens a cabinet (Chapter 6.9).

This chapter covers the power and cooling plant that keeps the accelerators fed. It assigns each asset class to a rung of the maintenance-strategy spectrum (run-to-failure, time-based/preventive, condition-based, predictive) and attaches the cost of getting it wrong. It then goes deep on the predictive program for the power chain (transformers, switchgear, UPS, batteries, generators) and the cooling plant (chillers, pumps, CRAHs, cooling towers), and on the liquid-cooling-specific failure modes that have no precedent in air-cooled operations. It closes with running maintenance in a synchronous world and with the spares forecasting that, when you get it wrong, turns a fast repair into a slow one. The failure taxonomy and rates that feed this program are in Chapter 14.3, and the telemetry stack that carries the signals is in Chapter 14.2. This chapter is about what you do with them.

Maintenance strategy is a per-asset decision, and the work is matching each asset to a rung by its failure economics. The four strategies trade setup and monitoring cost against the consequence of a missed failure, avoidable service work and restoration options. None is cheapest over life for every asset.

Run-to-failure (reactive). Run the asset until it breaks, then fix it. This is rational only where the asset is cheap, redundant and non-critical, and its failure is benign and instantly detectable: a single redundant fan in an N+2 bank, a corridor light. Apply it to anything in the critical path of a synchronous job and you have chosen to be surprised at 02:00.

Time-based / preventive (PM). Service on a fixed calendar or runtime interval regardless of condition: the generator load-bank run (NFPA 110 requires it annually only for diesels that miss the monthly loading rule), the quarterly switchgear inspection, the CDU filter change every N hours. This is the regulatory and warranty baseline, and for many assets it is correct. Its weakness is that it is blind to actual condition: you replace healthy components on schedule (waste) and still miss the component that degrades faster than the interval (surprise). PM intervals also generate outages, because every scheduled task on a non-redundant path is a planned hit you must negotiate.

Condition-based (CBM). Service when a measured parameter crosses a threshold: coolant conductivity past a setpoint, a UPS cell's internal impedance up 20% from baseline, a bearing's vibration RMS over the ISO band. You intervene only when the asset tells you it needs it, on your schedule rather than the failure's. CBM is the workhorse rung for AI-factory plant because it converts most surprises into scheduled interventions. It requires the sensors and the baselines, which is the up-front cost most operators under-budget.

Predictive (PdM). CBM plus a model that projects remaining useful life from the trend, so you schedule the intervention against a forecast failure date and a lead time rather than a static threshold. This is where 2026 DCIM and digital-twin tooling is heading (refrigerant-loss detection before cooling degrades, battery RUL across a whole string, generator-readiness prognostics), and where the agentic-ops vendors are concentrating. The payoff is real, but this is the most data-hungry and most over-promised rung. Treat vendor 'AI predictive' claims as CBM-plus-a-trendline until proven otherwise.

The power chain is the most mature predictive-maintenance domain, because the failure physics are well understood and the instrumentation has been proven for decades. AI loads change the stress profile, though, and that changes what you watch for. Walk it from the utility inward.

Transformers are the canonical predictive asset. Dissolved-gas analysis of the insulating oil is a chromatograph reading the fault before it propagates: acetylene signals arcing, ethylene signals overheating, hydrogen signals partial discharge. The trend over months gives weeks to months of warning. The AI twist is the load profile. AI halls run at a sustained high load factor, so transformers sit close to their thermal rating, and insulation ages with winding hot-spot temperature (IEEE C57.91). Iteration-scale training swings (Chapter 4.5) are filtered out thermally, because winding time constants are measured in minutes, but they still stress voltage regulation and protection. Events that last minutes do cycle the windings: job starts and stops, restart stalls, drained halls. At these duty cycles, online monitoring of hot-spot temperature, top-oil temperature and load history is no longer optional.

Replacement runs on three clocks. An on-site spare or mobile transformer can restore service in days to weeks, and repairing the failed unit takes months. Replenishing the spare, or installing a new US substation unit, faces lead times above 160 weeks in Wood Mackenzie’s May 2026 assessment. Predictive warning is what keeps a component failure from becoming a multi-year capacity shortfall.

Switchgear and breakers fail two ways. Insulation breakdown shows up first as rising corona current, which partial-discharge monitoring catches before the flashover. Connection degradation shows up as a loose or corroded busbar joint that glows hot under load relative to its neighbors, which infrared thermography catches within seconds of a scan. Online PD sensors plus periodic IR surveys make up the standard CBM pair. Skipping them risks something worse than a trip: a bus fault or arc-flash event takes an entire switchgear lineup and is a personnel-safety event, not merely a goodput event.

In the US, NFPA 70B sets the frame. Since the 2023 edition it is a Standard, not a recommended practice. It requires a documented electrical maintenance program, with intervals driven by an equipment condition assessment of physical condition, criticality and environment; the worst of the three ratings governs. An alert from monitoring or prediction shifts the equipment into a shorter interval. The 2026 edition, now current, extends the scope to predictive and corrective maintenance and lets permanently installed temperature sensors that monitor continuously satisfy its thermal-inspection requirements. A jurisdiction, insurer or contract makes it binding by adopting it, so find out which edition yours cites.

The upstream electrical plant is also now a grid-scale reliability concern. NERC's May 2026 Level 3 alert came after events such as the July 2024 Virginia 230 kV disturbance, in which roughly 1,500 MW of data-center demand left the grid as sites transferred to backup during six faults over 82 seconds. Later, on 22 July 2026, a single normally cleared 230 kV fault near Ashburn sent about 3.8 GW to backup, roughly 2.5 times the 2024 scale (PJM). ERCOT (NOGRR282) and SPP now oblige new large loads to stay connected through faults like these. Keep each site's transfer and protection settings under the same maintenance and re-test regime as the rest of its protection (ride-through, and the rules that now govern it, are covered in Chapter 4.10 and Chapter 13.6).

UPS and batteries are where predictive maintenance pays for itself fastest. The battery is the most failure-prone element in the power chain, and its failure is invisible until the moment it matters. The discipline is per-cell internal impedance trended against the commissioning baseline. A repeatable rise opens an investigation and the capacity or condition test the maintenance program prescribes, whatever float voltage reads. A rise of about 20% is a widely used first screen, and some programs reserve “serious” for 50% or more. The acceptance limit itself belongs to the chemistry, the instrument and the battery OEM. Impedance flags a suspect cell; it does not measure the capacity left in it. Layer on float-voltage drift, per-cell temperature spread (thermal imaging across the string reveals the differential signature of a degrading cell), and discharge-event depth and count. For the UPS itself, the wear items are the DC-bus and filter capacitors (ESR and temperature trend) and the cooling fans (runtime hours). Chemistry sets the program. VRLA strings degrade silently on a 3–5-year replacement clock, so they demand aggressive impedance-based CBM. Lithium strings carry their own BMS telemetry, which shifts the program toward consuming BMS state-of-health rather than instrumenting cells yourself. Ride-through and energy-storage roles are in Chapter 4.5.

Rack-level energy storage moves part of this program into the IT rows. Where racks carry Open Rack V3 battery backup units (BBUs), the hall holds many small lithium-ion modules instead of a few central strings, and you maintain them as a fleet. OCP's ORV3 BBU specification (v1.4, September 2023) has each 3 kW module test its own capacity every 90 days by a forced discharge. Tests run one module per shelf at a time, and each test discharge is capped so the module under test could still back up a 90-second outage, which preserves the shelf's 5+1 redundancy. The timestamped result sits in a register the rack monitor reads over Modbus. A module reports end of life when it can no longer carry full power for that 90-second default; the spec sizes the pack to hold four minutes at full power for at least four years at 35 °C. The spec queues the test while any module in the shelf reports end of life or the shelf is not fully populated, so a dead module left in place, or an empty slot, stops its neighbors from being tested. Pull the results into the DCIM, trend them by purchase cohort, and stock hot-swappable replacements against the cohort curve (Chapter 14.6).

Power-shelf capacitors are the other new storage layer, about 65 J per GPU on GB300 NVL72 (NVIDIA, July 2025). They are an electrolytic wear item like the UPS DC-bus capacitors, and the smoothing that draws on them is governed by per-GPU power controls the driver applies, so verify those settings again whenever firmware or configuration changes (Chapter 14.7). The HPR V2 shelf's PSU-to-BBU signalling is in Chapter 4.5; the lithium fire consequence, including off-gas detection, is in Chapter 6.5.

Generators are a readiness problem, not an uptime problem: they spend almost all their life not running and must start on the one occasion the grid is gone. Predictive practice is oil analysis (wear-metal spectroscopy reading bearing and ring wear), coolant and fuel-quality testing (microbial contamination in stored diesel is a classic no-start cause), vibration on the genset, and a logged start-reliability statistic. Standby diesels have their own trap, wet-stacking: a diesel repeatedly exercised at light load sends unburned fuel and carbon into its exhaust. NFPA 110, where adopted, has each diesel exercised monthly for at least 30 minutes, at 30% or more of standby nameplate kW or at the OEM's minimum exhaust temperature. A diesel that meets neither condition needs an annual supplemental load-bank run, 30 minutes at 50% load followed by an hour at 75%. That makes the load-bank test a PM task and a health check at once. Spark-ignited gas engines do not wet-stack, as NFPA 110's committee notes explicitly, so for behind-the-meter gas units follow the OEM's minimum-load guidance. Black-start and load-bank testing detail is in commissioning practice; behind-the-meter generation economics are in Part 3.

Cooling-plant predictive maintenance is rotating-machinery maintenance plus heat-transfer maintenance, and the AI factory raises the stakes on both because the thermal margin has collapsed. Cooling-fault response in a liquid hall is set by the installed rack, CDU, loop and controls, the initial conditions and the failure state, not by rack power alone (cooling-controls transient dynamics are canonical in Chapter 5.12). Validate maintained-flow, degraded-flow and loss-of-flow transients with OEM data, a heat balance and controls tests. Then size pump and heat-rejection redundancy and shutdown timing to those results.

The rotating equipment (chillers, primary and secondary pumps, CRAH/CRAC fans, cooling-tower fans) is classic vibration-analysis territory. Bearing-defect frequencies appear in the vibration spectrum weeks before the bearing seizes, and rising RMS, bearing temperature and motor-current signature analysis triangulate the failure. The heat-transfer side adds its own indicators. A widening approach temperature (the gap between the refrigerant and the water it is cooling) signals fouled tubes. Rising differential pressure across filters and strainers signals a clogging element. A refrigerant charge trending down signals a leak, and 2026 predictive tooling increasingly flags refrigerant loss before cooling capacity visibly degrades. Cooling towers and evaporative systems add water chemistry (scaling, Legionella control, blowdown) to the program.

Running this rotating equipment as run-to-failure carries the worst consequence in the cooling program. A pump or chiller that fails uncaught in a dense liquid hall does not give you a clean trip and a restart. It gives you a thermal excursion that throttles or trips accelerators while you scramble, and in a synchronous job that is a job-wide restart.

Direct-to-chip liquid cooling is a common 2026 choice for dense GPU halls, where the installed OEM platform requires or supports it (Chapter 5.4). It introduces a maintenance surface that did not exist in air-cooled operations, and most of it is chemistry and fluid mechanics, disciplines the average DC ops team had never owned. Three enemies dominate. All three are slow, cumulative and invisible to a temperature alarm until they have already cost you: corrosion (galvanic and general, attacking cold plates, manifolds and the wetted metals), scaling/mineral deposition (narrowing the sub-millimeter cold-plate microchannels), and biological fouling (microbial growth in the warm, nutrient-bearing loop). Each reduces effective channel cross-section, raises thermal resistance and pressure drop, and ends in hot spots and GPU throttling. On GB200-class equipment, the size of the loss depends on the OEM-qualified coolant range and the measured workload response, so correlate chemistry, flow and GPU throttling to quantify the silent goodput tax.

The predictive program is built on coolant chemistry monitoring: pH (corrosion onset), conductivity (ionic contamination and inhibitor depletion), ORP/oxidation state, and turbidity (particulate and biological load). Per 2026 CDU-sensor practice, these four parameters reveal corrosion, scaling, fouling and contamination long before any temperature alarm trips. On top of chemistry sit the mechanical CBM points: filter differential pressure (the best clogging indicator, and the most common scheduled service), CDU pump vibration, coolant flow and supply-temperature trends per loop, and the leak-detection and level-sensing layer that turns a slow seep into an alarm before it becomes a puddle on a live busbar.

The CDU itself is the unit of concurrent maintainability here. A CDU with internal N+1 pumps and redundant heat-exchanger paths may keep its load supplied while the pump you are working on is stopped, isolated, depressurized and made safe under the OEM procedure. Redundancy alone does not authorize pump work, and a single-pump CDU needs another supported cooling path or a load outage. That is why the CDU redundancy decision at design time is a maintenance decision (Chapter 5.4).

The conflicts are mundane and easy to miss on a desk read. A figure legend calls a valve isolation while the numbered step calls the same item bypass, or a drain step names a different item from the hose step before it. No one should have to guess a route through pressurized coolant from a document like that, and unresolved paperwork strands a maintenance window as surely as a missing spare part does. Release the work for preparation only when four things hold: no identity conflicts remain, the OEM has confirmed the procedure for the installed unit, a P&ID and tag walkdown matches it, and an energy-isolation and restoration plan is approved. The electrical isolation authority takes part in setting the service boundary. That release still does not authorize anyone to touch a valve before field isolation is proved. After the work, leak, flow, pressure, alarm and redundancy-path tests follow Chapter 14.14, and the procedure itself is written from the controlled MOP/SOP/EOP skeleton under Chapter 14.12.

Instrument health belongs in the same check. Vertiv's XDU450 manual, for example, expects its three secondary supply-temperature sensors to read within 0.3 °C of each other. That cross-check exposes a drifting sensor, but it proves neither absolute calibration nor cooling capacity.

In a traditional data center you fail traffic away and take the window. In a synchronous AI cluster there is nowhere to fail to, so the window must be created without dropping the job. A pre-training run is one job, and you cannot move 40% of it to another availability zone for an afternoon. Three levers exist, and they form a decision tree.

Lever one: concurrent maintainability. If the facility is Tier III/IV-class, the preventive task happens on the redundant path while the job runs. This is the cleanest answer, and the one bought at design time. The maintainability choice above therefore determines whether maintenance is invisible to goodput or a negotiation with it.

Lever two: coordinate with the checkpoint cadence. When a task genuinely requires the load down (a non-redundant-path job, an N-topology hall, a fault that breaks concurrent maintainability), you do not pick a clock time. You pick a checkpoint boundary. The job is already writing checkpoints at the Young/Daly optimal interval (Chapter 9.4). The marginal cost of a planned stoppage at a checkpoint is the checkpoint commit and stall, plus drain, shutdown, restart or reconfiguration and reload, plus any work done after the checkpoint and before the stop, which is lost. The maintenance calendar and the ML platform's checkpoint schedule must therefore be planned together, which is an organizational requirement, not a technical one (below).

Lever three: batch the work. If the load is coming down anyway, for a firmware campaign (Chapter 14.8), a fabric upgrade or a planned migration, pull every deferrable preventive task into that same window. A drained hall is the scarcest resource in an AI factory, and wasting it on a single task is a planning failure.

An operator that runs an N-topology training hall on a time-based PM calendar (fixed-date transformer inspections, fixed-interval battery service) without coordinating those dates with the workload will either (a) defer the PM and accept rising failure risk, or (b) take the dates and bleed goodput on the platform team's behalf without telling them. Both are common and both are avoidable. Both trace to treating the maintenance calendar as a facility artifact rather than a joint facility/ML-ops artifact.

Predictive maintenance buys you warning; spares strategy decides whether the warning matters. A CDU pump you can see failing three weeks out via vibration trending is only useful if you have the replacement pump, or can get it inside three weeks. The value of a predictive program collapses if the spare is a 16-week procurement. Spares forecasting for plant is therefore the necessary complement to the predictive program, and it obeys a different logic than IT-component sparing (the FRU/RMA machinery of Chapter 14.6).

A spares service level needs fleet population, demand and failure distribution, repair-and-return and replenishment distributions, a target fill probability, common-mode exposure, cannibalization or shared-spare options, and the consequence of consuming redundancy. Failure rate × replenishment time is an expected-demand input, not the stocking answer.

The plant-specific wrinkle is the lead-time asymmetry. Commodity items (fans, filters, pumps, capacitors, contactors, sensors) are days to weeks and cheaply over-stocked. Some long-lead items may justify an on-site, mobile, manufacturer-held or shared regional spare despite cost and space; Wood Mackenzie reported US substation-transformer lead times above 160 weeks in May 2026. That forces a different hedge: predictive catch with maximum warning, manufacturer hold agreements, designed-in redundancy so the unit's loss is survivable, and in the extreme a shared regional spare across sites. Warning and spares are complementary rather than interchangeable, and which one is available depends on lead time and unit cost, not preference.