Operations Organization, Workforce, Talent & Incident Command

Human action, and the organization that shapes it, can turn an equipment fault into prolonged downtime. Every earlier chapter in Part 14 covers a thing: telemetry, spares, firmware, maintenance, refresh. This one covers the people who operate those things.

Uptime Institute's 2025 survey shows where the human failures sit. 58% of human-error outages involved staff not following established procedures, and roughly 85% of human-error outages involved either that behavior or flaws in the procedures themselves. About 40% of organizations had a major human-error outage in the prior three years. And 80% of operators believed better management, process or configuration would have prevented their most recent downtime incident. That last figure is the operators' own judgment, not a measured share of causes.

The org chart is therefore a reliability control. A GW-scale AI factory costs on the order of $30–40B to build (Epoch's May 2026 model: about $38B up front for 1 GW of IT, excluding financing; Chapter 3.1) and earns its return only on sustained goodput. Size the organization from the loss modes you measure in your own service, not from a generic goodput target. How the operating organization is structured, staffed and commanded under stress decides whether that return is delivered or destroyed.

This chapter covers organization design, workforce, talent and incident command: the org seams and what falls through them, the shift and on-call models and what they cost in fatigue and error, the unified incident-command model (this is its canonical home; Chapter 11.12 references it from the security side), and the automation tradeoff that lets a small team run a gigawatt fleet while enlarging the blast radius of a single mistake. The procedure framework that traps human error is Chapter 14.12; here we build the org that executes it.

The team that builds the facility and the team that runs it are often treated as one continuum. They are two organizations with different incentives, different skill profiles and a hand-off, the turnover, that is among the most error-prone moments in the facility's life.

The development-phase organization is a temporary project machine: an owner's development team, EPC (engineer-procure-construct) project managers, design engineers, long-lead procurement and, late in the phase, the commissioning agents (CxA) who prove the plant performs to design basis through Levels 1–5 commissioning and integrated systems testing. Its incentive is schedule and cost to first power. It disbands when the building is energized.

The steady-state operations organization is a permanent reliability machine: facility engineers, BMS/EPMS operators, mechanical and electrical technicians and, on the IT side, the platform, fleet-reliability and on-call engineers who own the cluster. Its incentive is goodput over a 10–20 year operating life. The two teams optimize opposite objectives. On the day the project team hands over the keys, the as-built drawings, the sequence of operations, the alarm setpoints and the institutional memory of why a thing was built that way must transfer intact. Otherwise the operating org inherits a facility it does not understand. Chapter 6.6 gives the construction-phase view of this org, and Chapter 6.9 covers the EHS thread that spans both phases.

Inside the steady-state org there is a fault line that did not exist in the enterprise-IT era, and it is the most consequential boundary in AI-factory operations: the seam between facility operations (the people who own megawatts, cooling water and the building) and ML-platform operations (the people who own the cluster, the scheduler and the job). In a traditional colo the two worlds barely touched. The facility delivered power and cooling to a meet-me room, and the tenant's IT was a black box. In an AI factory they are mechanically coupled.

A synchronized GPU load can swing by tens of megawatts as the workload changes. Meta's Llama 3 405B run, on up to 16,384 H100s, is a case in point. Voltage sags move load by a different mechanism. In the July 2024 Virginia event, NERC's review found that about 1,500 MW of data-center demand left the grid over six faults in 82 s as sites transferred to backup power. About 1,260 MW of it stayed off for hours. On 2026-07-22 a single normally cleared 230 kV fault took about 3.8 GW off the grid in Ashburn. Neither was a trip or a workload swing: the sites transferred to backup as designed. Commissioning should prove both: clean transfer outside the ride-through envelope, and staying connected inside it (Chapter 13.6; regulatory status in Chapter 4.10).

Within seconds of a coolant-loop excursion, the GPUs throttle or shut themselves down to protect the silicon. A scheduler decision to pack a hall changes the thermal and electrical load the facility must absorb as well. The facility and the cluster are one thermodynamic system run by two organizations that, by default, do not share a pager.

Goodput, the useful work delivered within the declared quality and service limits (as a time fraction, the share of wall-clock time that advances the job; Chapter 14.1), is a cross-seam metric. A GPU stalled because a CDU tripped is a facility failure that shows up as a platform metric. A breaker that trips because the scheduler oversubscribed power past the design headroom is a platform decision that lands as a facility event. If no single role owns goodput across the seam, each side optimizes its own KPI (facility uptime, cluster utilization) and the cross-seam failures fall into the gap. The fork is organizational: keep two orgs with a coordination layer, or build an integrated operating model with a single accountable owner of goodput.

The integrated model puts goodput under one accountable owner, which can shorten cross-domain decisions. It also demands cross-trained people, who are among the scarcest in the industry, and it is organizationally heavier, so weigh the benefit against those staffing and contractual costs. The operator-managed model buys speed-to-power and a ready bench. But the SLA you sign almost always covers facility availability (the nines), not goodput (the useful work). You can comply fully with the contract and still lose the GPU-hours you built the place to sell.

Chapter 14.1 is the canonical accounting of useful output. Chapter 12.2 defines the continuity obligation that must also hold, and Chapter 12.1 treats the colo/operator boundary as a redundancy and blast-radius question.

The workforce question is not a soft HR topic to defer to staffing season. For the build, it is a hard physical constraint that belongs in site selection alongside power and water. The skilled trades that build and energize an AI campus (journeyman electricians, mechanical pipefitters, controls-and-instrumentation technicians, high-voltage technicians, commissioning engineers) are in acute shortage. Unlike software talent, they have near-zero geographic mobility: a pipefitter must be physically on site. In its January 2026 forecast, ABC says US construction as a whole needs about 349,000 net new workers in 2026. That figure covers all construction, not data centers alone, though ABC points to data-center demand for electricians.

Power infrastructure is the biggest single cost line in the build. Cushman & Wakefield's 2026 cost guide gives it about 21% of all-in greenfield development cost, against 17% for core-and-shell and sitework combined. That base includes land, sitework, shell and contingency and leaves out chips and servers. It is a cost share, not a labor share, but the electrical scope behind it is what draws on the scarce trades. If labor is the binding input, a GW campus can use up a regional trade pool the moment it arrives. JLL's 2026 midyear outlook shows how tight that is: contractors with data-center work average 12.2 months of backlog against 8.3 for those without, and labor supply is constrained in 61% of US metro markets, a share JLL expects to reach 72% by 2027.

This reaches back into siting. A site with cheap stranded power and a cold climate but no local electrical workforce, no nearby training pipeline and a semiconductor fab competing for the same journeymen is not as cheap as the power spreadsheet suggests. It carries a schedule-risk premium and a wage premium that can outweigh the energy savings. Chapter 6.6 covers the construction-phase side of this constraint. On the operations side the operating org meets the same squeeze: qualified plant operators are scarce too, not only trades.

Construction hiring needs are large at national scale. ABC's January 2026 estimate is 349,000 hires needed across all US construction, mostly to replace retirees, after 439,000 for 2025. No operator hires its way out of that in a tight local market, least of all for qualified plant operators, whom that forecast never counted. The operators who win the labor race build a pipeline, and the decision is the same build-vs-buy fork that governs every other scarce input.

Buy means poaching experienced operators and trades at a wage premium. It is fast, expensive and zero-sum against your neighbors: it works for the first crew and fails at scale, because the pool is finite and your competitors are bidding for the same people. Build means apprenticeships, partnerships with IBEW/UA locals and community colleges, and an internal certification ladder from technician to senior facility engineer to shift lead. It is slow (a journeyman electrician is a 4–5 year apprenticeship), but it is the only path that expands the pool rather than redistributing it, and it produces operators who learned your facility's failure modes rather than someone else's.

The certification ladder matters beyond morale: it is how you encode and verify competency on safety-critical tasks. An operator who may execute a switching order on live medium-voltage gear, supervise a LOTO (Chapter 6.9) or take a concurrently maintainable plant out of a redundant configuration must be qualified to a documented standard. That qualification, not seniority or availability, is what the shift roster must respect. The shortage is no longer only in trades. In Uptime's 2026 survey, more than half of operators reported difficulty finding qualified candidates for open jobs, and staff turnover remains a persistent problem. Protect the shift-lead and operations-manager bench first: neither seat should go to someone still learning the job when an incident hits.

An AI factory never stops, so the org must cover 168 hours a week, and the shift model you choose is a direct trade between coverage cost, fatigue and error rate. 12-hour shifts (for example a Pitman 2-on/2-off/3-on pattern; DuPont is a different rotation) cover the week with fewer hand-offs, and hand-offs are themselves an error source. But long shifts degrade decision quality in the back half, exactly when a 3 a.m. incident is most likely. 8-hour shifts keep operators fresher but add a third daily transition: three hand-offs a day against two, 50% more, each one a chance to drop context.

Either way, minimum staffing is load-bearing. How many qualified operators must be physically present to respond safely to the worst credible event (an EOP execution, an evacuation, a switching operation that requires a second qualified person for safety)? Staff below that floor to save cost and you have, on paper, an operating facility that cannot actually be operated safely under fault.

On-call is the release valve: a thinner overnight crew backed by senior engineers who can be paged. It has a hidden cost the org must price, which is response time. For a coolant excursion that throttles or trips GPUs within seconds, the on-call engineer's 20-minute drive is irrelevant. Automation and the on-site crew must handle that failure, and on-call covers only events whose clock runs in tens of minutes or hours. The rotation is also where burnout concentrates and where your scarcest senior people churn. An org that runs a healthy on-call (bounded page frequency, follow-the-sun for global fleets, a blameless culture so the pager is not punitive) retains the bench it spent years building. Software teams have a published yardstick in Google's SRE on-call model: no more than two incidents in a 12-hour shift, and a single-site rotation of at least eight engineers when a primary and a secondary are always on call and nobody carries the pager more than 25% of the time.

Trace. Two posts need 2 × 168 = 336 qualified post-hours a week. Each operator supplies 40 × 0.80 = 32, and 336/32 = 10.5, so at least 11 operators pass this average-hours screen. Ten would supply only 320 post-hours, leaving a gap of 16 post-hours every week. Plan for eleven, then build the actual roster: it must cover every shift, break and concurrent emergency role within fatigue limits and local working-time rules. An on-call engineer driving in cannot fill an on-site post.

Flip. Ten operators pass the hours screen only if availability reaches 336/(10 × 40) = 84%, and even then the roster and qualification mix may not work. Chapter 14.12 sets which procedures need a qualified person.

Break-fix bench. Facility posts are not the whole roster. Hands-on IT break-fix (tray, NIC, optic and cable swaps, reseats, RMA shipments) is sized from the failure rate. Meta's Llama 3 405B run logged 419 unexpected interruptions in 54 days on up to 16,384 H100s and attributed about 78% of them to confirmed or suspected hardware faults: about six a day. Scaled to a 100,000-GPU campus, that is about 37 a day. Read that as a rough count of physical touches, neither a ceiling nor a floor. A reboot or an automatic drain clears some faults (Chapter 10.7), while a dead fan, a failed power supply or a flapping optic may never stop a job yet still need a technician.

If each touch takes 1.5 technician-hours, 37 touches need about 55 hours a day, or about 390 a week. At 32 post-hours per technician that is 12.2, so budget roughly 13 technicians before planned work, RMA logistics and the burst after a bad firmware push. At that rate the campus needs its own break-fix crew on every shift. A smart-hands contract billed per ticket and queued behind other tenants turns each touch into hours of idle GPUs. Failure data is in Chapter 14.3, and spares and RMA in Chapter 14.6.

When an incident crosses the facility/platform seam, and the consequential ones always do, the org needs a single command structure, decided before the incident rather than improvised during it. Most mature operators adopt a model derived from the Incident Command System (ICS): one incident commander (IC) with clear authority, explicit roles (operations lead, communications/scribe, liaison to vendors and the utility), severity tiers that set the escalation and notification path, and a single source of truth for incident state. The IC owns the decision and the specialists own their domains; nobody is both flying the plane and talking to air traffic control. This is the canonical home for the model, and Chapter 11.12 references this structure for cyber-physical events from the security side.

The AI-factory wrinkle is the converged cyber-physical escalation trigger. In a traditional plant, a cooling alarm and a security alert run on separate tracks to separate teams. In a facility where the BMS, the EPMS and the cluster control plane are networked, and increasingly agent-operated (Chapter 14.13), an anomaly can be a failing pump or an attacker manipulating the pump's setpoint, and the first alarm does not say which. The org therefore needs a trigger that forces both the facility incident track and the security incident track into one command room the moment an event is ambiguous between physical fault and cyber cause.

The fork is between dual independent IC structures (cleaner in peacetime, dangerous in a converged event) and a single IC with a facility lead and a security lead reporting in (a heavier standing structure, correct under a converged event). For a networked, automated AI factory in 2026, the converged single-command model is the defensible choice. Write the trigger into the EOPs (Chapter 14.12) rather than leaving it to operator judgment at 3 a.m.

Resolve command before the alarm. The IC sets incident objectives, coordinates restoration and controls communications. The electrical authority controls switching and isolation, the mechanical lead controls the coolant boundary, and the platform lead controls drain and restart. The security lead contains a cyber cause without unilaterally disabling a safety function. Any specialist can call a hold when a proposed action violates the safe operating boundary; the IC records that constraint and chooses a supported recovery path rather than overriding it to meet an ETA. This division follows the Google SRE incident-response roles, and cyber response in Chapter 11.12 joins this command structure.

Rehearse the recovery order, not just the roles, and write down in advance where you will stop. Before each larger PowerLoss Storm drill, Meta wrote down the outcomes it would not accept: data loss, permanent facility damage, or impact beyond one region. The drills also showed that the scheduler and allocator that remediation depends on could fail to restart on their own (drill staging: Chapter 12.3).

Every recurring operational response should be a runbook: a written, tested, versioned procedure. Improvisation throws away the procedural controls, and skipped or flawed procedures sit behind most human-error outages. The decision that defines the modern ops org is not whether to write runbooks but how much of the runbook to automate, and that choice sits on a sharp tradeoff curve.

Automation can let a small team run a gigawatt fleet, though no operator publishes what share of its incidents automation resolves fleet-wide. Operators automate bounded detection, triage and remediation classes, while novel or high-consequence incidents escalate to accountable humans. Self-diagnosing, self-optimizing DCIM and digital-twin tooling (Chapter 14.2) is subject to those same boundaries. Automation also takes the human out of the loop precisely where the human is the error source. That is the upside.

The downside is blast radius and de-skilling, and they compound. An automated runbook that executes a wrong action does so across the whole fleet in seconds: no human pause, no "that doesn't look right", just a fast, wide failure. And the more the automation handles, the less the on-shift operators practice the manual response. When the automation hits a case it was not designed for, the humans who must take over are out of practice on exactly the skill the moment demands. Outages become fewer but, when they happen, larger and stranger, and human recovery is slower because the muscle has atrophied.

So automate deliberately. Keep humans in the loop for irreversible and high-blast-radius actions, require a human confirmation gate on anything that touches the whole fleet, and run regular manual-failover and EOP drills so the bench keeps the skill the automation usually exercises. Chapter 14.13 formalizes the autonomy ladder for how far to let an agent act unsupervised.

Almost no AI factory is operated by a single org under one roof. Steady-state operation is a web of boundaries: the OEM whose GPUs are under RMA (Chapter 14.6), the cooling-plant vendor with a service contract, the colo landlord whose SLA stops at the rack, the specialist operator running the plant, and the utility on the other side of the meter. Each boundary is a place where accountability can be dropped during an incident. The org's job is to make every boundary explicit: who detects, who decides, who acts, and who is on the hook when the cause sits ambiguously across the line.

The classic failure is the orphaned incident. A coolant CDU faults, the operator says it is the vendor's CDU, the vendor says it is the operator's facility water, and the GPUs throttle while the two argue. The fix is contractual and procedural: name the IC who has authority across vendor lines during an incident, write vendor response-time SLOs that match the failure's clock (a seconds-clock failure cannot have a four-hour vendor response), and rehearse the cross-boundary escalation before it is live.

The throughline of every section above is that the operating organization is a learning system, and its rate of learning is the real reliability differentiator over a 10–20 year operating life. The fleet's failure environment changes under it. In at least one large training run, accelerators caused the largest share of unexpected interruptions. In Meta's 54-day Llama 3 405B run, GPUs and their HBM3 memory caused 220 of the 419 unexpected job interruptions and CPUs caused 2. NCSA's Delta study, covering 1,056 A100 and H100 GPUs, projects that about 5% spare capacity would absorb GPU failures at larger scale. Density steps up every generation, and each step-up requires retest scope matched to changed power, flow, floor loading and controls (Chapter 14.14).

An org that runs blameless postmortems, feeds them into procedures, drills the procedures and grows its own bench compounds competence. An org that treats operations as a static cost center to be minimized accumulates fragility and pays for it in the outages it could have prevented. The 80% figure is the most optimistic number in this chapter: most operators think their last incident was preventable, which is good news only for an organization built to act on it.