Day-2 Operations, Upgrades & Lifecycle

Goodput, useful work delivered within quality and service limits, sets an AI factory's day-2 economics. Justify each reliability dollar by the useful output it buys, once contracted availability is met. High uptime can hide a tenth of compute doing no useful work.

IT health data sampled in seconds and OT plant telemetry polled from seconds (CDU flow) to a minute (plant trends) can explain a goodput drop, if timestamps, data quality and asset identities reconcile across both planes. Specify that correlation before buying the DCIM.

A large GPU fleet accumulates hard, transient and silent failures. Meta’s Llama 3 run, interrupted every few hours, shows one operating regime. Measure each component’s exposure and the whole-job trace before you design detection, spares and recovery.

When a frontier training job is interrupted every few hours, as Meta's Llama 3 run was, operations comes down to detecting, isolating and recovering faster than failures pile up. Every minute of recovery is goodput the job does not get back.

Whether you can service the plant without dropping a synchronous job is decided years earlier, when you buy concurrent maintainability and build a condition-based program. That program lets you intervene on your schedule rather than the failure's.

On-site spares depth and supported swap speed decide how much goodput a fleet of hundreds of thousands of accelerators gets back when the wait for a replacement is the binding constraint. The OEM warranty and replenishment channel decide whether the local pool stays usable.

The megawatts you energized are a fixed, capital-intensive ceiling. Steady-state operations trade how full you run that budget against how violently the workload can swing it.

Firmware and software changes can reach 100,000 GPUs on a synchronized estate. Roll out a validated version tuple through bounded cohorts, so goodput holds and a bad bit never travels past what your tested recovery path can undo.

Refresh turns the Chapter 1.8 depreciation assumption into a physical operation. Pull timing, cascade routing, sanitization and the ITAD channel decide whether the residual value you underwrote is actually realized.

The facility outlives its silicon by decades, and its end state (repower, demolish or convert) is decided years early. In a power-bound market, the energized shell is often the most valuable asset.

The org chart, talent bench and incident-command model are reliability infrastructure. Qualified shift coverage and executable procedures decide whether operators keep goodput up under stress; thin staffing or unusable procedures can turn a fault into a long outage.

Tested MOP/SOP/EOP procedures and change control make human work on live plant a reliability control: prove the intended safe state before you touch anything, and prove service is restored before you close the change. Skip either proof and routine work can become an outage.

An agent may act unsupervised on a control loop only as far as you have bounded its action space, tested its override path, and assigned the liability for when it is wrong.

On a live campus, every change ticket and density step can invalidate part of the commissioning evidence. A standing re-commissioning program keeps the proof that still applies and retests what changed while protecting revenue-bearing load.