Disaster Recovery, Business Continuity & Geographic Failover

Most of this Part covers keeping a single facility running: redundancy topologies (Chapter 12.1), the goodput-versus-availability rethink (Chapter 12.2), and the thermal and electrical paths that fail inside the fence. This chapter covers the failure the fence cannot stop, the one that takes the whole site, the whole region or the whole control plane at once. Examples include a transformer fire, a substation lost to weather, a fiber cut that isolates a campus, a cloud region's metadata anchor going dark, a wildfire-evacuation order and a ransomware event in the orchestration layer. The question stops being "is my UPS healthy?" and becomes where does the work go, how fast, and did I pay to have somewhere for it to go?

The decisions run in sequence. First come RTO and RPO targets, set per workload class and priced by tier. Next is active-active versus active-passive across regions, which fixes fleet count and cost. Then comes the capacity constraint: failover capacity is power-bound wherever activation cannot finish within the recovery deadline. After that come the single dependencies (control plane, DNS, weights store, key personnel, single-vendor loops) that turn a local fault into an estate-wide one, the contractual DR obligations you owe tenants versus what the topology delivers, and the drills, runbooks and FMEA tie-in that prove the plan works before you need it.

Two numbers govern every continuity decision. RTO (Recovery Time Objective) is how long you can be down: the wall-clock time from failure to restored service. RPO (Recovery Point Objective) is how much work you can afford to lose: the gap back to the last durable state. They are independent. You can want fast recovery (low RTO) while tolerating some lost work (looser RPO), or the reverse. Cost rises steeply and non-linearly as either target approaches zero, so the first act of continuity engineering is to refuse a single estate-wide target and tier the workloads.

AI workloads split the RTO/RPO table more sharply than any traditional enterprise estate, because the archetypes have opposite tolerances (the same split that drove the redundancy logic in Chapter 1.2 and Chapter 1.3). Interactive inference is the revenue surface and the tightest tier. A user is waiting, an SLA is running, and a regional loss that takes minutes to absorb is a visible outage. AWS's Platinum resilience example uses a 15-minute RTO and 5-minute RPO for mission-critical workloads; set your own targets per workload and recovery architecture.

Training continuity is checkpoint-bounded, but RPO is not the snapshot-start cadence. At failure, select the newest valid checkpoint whose durable copy and dependencies survive the event. Lost work reaches back to its captured progress, not to its commit-completion time. Capture time, background drain, commit completion, replication scope and overlapping checkpoints all move that boundary. RTO is bounded separately by state discovery, capacity reacquisition, restore, validation and resume.

Batch inference can queue and retry when its delivery deadline permits; deadline-bound batch needs capacity that finishes the remaining work on time. Giving a batch pipeline the strongest DR posture is the same mistake as buying fault-tolerant power for a training hall whose checkpoints already absorb a maintenance outage or a single component fault. Either way you pay for continuity the work does not need.

Zero RPO has no universal cost percentage. Declare the protected state and the disaster first. Then measure capture, blocking snapshot, background drain, durable commit, restore and contention under the reference workload. Select the newest valid checkpoint that has committed and survives the event; its captured progress sets the lost work. The link to the second region bounds how quickly a new state can become durable there: copying an assumed 100 TB of checkpoint and weights state in one hour takes about 220 Gbit/s of sustained payload, before protocol and replication overhead. If the workload writes that much new state each hour, a link that cannot sustain the rate leaves the surviving copy staler than the snapshot cadence implies. Chapter 9.4 owns the checkpoint timing boundaries. The worked example below shows how a recent commit can still hold state older than the RPO target.

The newest valid surviving capture is S. Its state age is 09:10 − 09:00 = 10 minutes. The two minutes since its remote commit are not the lost-work age. Reject the five-minute RPO claim: either secure a newer recoverable state or change the recovery objective. Flip: at the same failure time, a valid surviving capture at or after 09:05 passes the age screen, provided commit and recovery dependencies also survive. Keep protected older versions for corruption recovery, because replication can copy deletion or corruption. PyTorch distinguishes staging and upload completion; this example applies the captured-state boundary from Chapter 9.4.

AI-factory DR differs from a classic continuity playbook because failover capacity cannot be conjured on demand: a tight recovery deadline turns compatible capacity into an advance procurement decision. In a traditional cloud estate, failover capacity is fungible. A region fails, you spin up instances elsewhere, pay the burst rate and move on. In a power-bound AI estate that escape hatch is closed. You cannot rent 200 MW of GPUs in a neighbouring region on the morning your primary burns, because that capacity does not exist as slack. Every megawatt is contended, interconnection queues run years (the queue framing from Chapter 3.1), and a high-demand part like a current-generation accelerator is allocation-gated, not catalogue-stocked. Capacity must already be energized and racked when energization, delivery or activation would otherwise exceed the deadline. A reservation helps only if the activation path has been tested.

That collapses the comforting cloud-era distinction between "reserve capacity" and "pay for it later." For the inference surface, the spare region is a real second factory: pre-paid, drawing real power, and depreciating on the same contested 2–3-year bear-case economic clock as the primary (the depreciation argument in Chapter 1.8). The continuity decision is therefore capital allocation, not availability: how many megawatts of idle-until-needed capacity will you underwrite, and can you make them earn while they wait?

The most important mitigation is to stop treating the spare as idle. Reverse-arbitrage the standby: run interruptible, RPO-loose work on the failover fleet during normal operation (batch inference, evaluation sweeps, synthetic-data generation, low-priority fine-tunes), and pre-empt it within the tested recovery budget when the primary fails. This is the continuity analogue of the curtailable-load fast lane in power procurement. The spare region pays part of its own carry by doing displaceable work, and the failover becomes a scheduler pre-emption rather than a cold start. The pre-emptible work must drain fast enough to hit the inference RTO, and that is an orchestration property, not a hardware one.

The lower rows of the table rest on an assumption that fails exactly when it is needed. Pilot-light and cold standby look attractive because their steady cost is low, but their RTO assumes you can acquire the missing capacity when disaster hits, and in a power-bound market that acquisition is what falls through. A cold-standby plan that depends on renting GPUs from a neocloud during a regional outage competes for scarce capacity with every other operator whose primary just failed in the same correlated event (the wildfire, the heatwave-driven grid event, the regional storm). Cold standby is defensible for a workload that can wait through its measured acquisition and restore path, days if that is what the path takes. An interactive endpoint with a shorter deadline needs active-active, hot standby or another prepared path that restores service in time. A long-latency batch service can buy less readiness.

For r regions, GPUs per region = ceil[18,000 / ((r−1) × 100 × 0.8)]. Two regions need 225 GPUs each, 450 total. Three need 113 each, 339 total. The one-region capacity equivalent is 225 GPUs, and the ideal unrounded reserve factors are r/(r−1), or 2 and 1.5. Rounding makes the three-region factor 339/225, about 1.5. These are capacity factors, not total-cost factors.

With three regions, normal demand is 6,000 requests/s each. After one loss, each survivor takes 9,000: 9,000/(113 × 100) = 79.6%, below the assumed 80% limit. Held to the 80% limit, the two survivors can carry 18,080 requests/s between them. The capacity screen passes at the assumed peak; a larger displaced peak or a slower destination requires a new calculation.

Select three regions on the capacity screen: 339 installed GPUs instead of 450, saving 111 GPUs, while the assumed nine-minute recovery fits the 15-minute objective. Treat the choice as provisional until measured throughput and a drill with primary-region access denied confirm it. Flip: at 113 GPUs per surviving region, the throughput crossover is 18,000/(2×113×0.8), about 99.6 requests/s per GPU. Use the unrounded threshold for the screen. At an assumed 95 requests/s, each region needs ceil[18,000/(2×95×0.8)] = 119 GPUs, so the 113-GPU design fails. The timeline flips too: any extra delay above six minutes on the sequential drill path misses the RTO. AWS’s recovery-pattern guidance supplies the method boundary; Chapter 13.9 owns the recovery acceptance evidence.

Geographic failover protects against losing a place. It does nothing against losing a shared dependency: a component that fails everywhere at once and makes your second region useless because it depended on the same thing. Apply the blast-radius lens from Chapter 12.1 across regions. List every system common to all sites and ask what happens when it breaks. Outage data shows the pattern is common. Uptime Institute's nine years of data on publicly reported outages put third-party providers (cloud, internet, telecom and colocation) at about two-thirds of them. Its 2026 analysis adds that external-infrastructure failures are becoming more prominent. Public reports skew toward provider outages, since those hit many customers at once, yet shared providers are exactly the dependencies to enumerate here.

A concrete warning is the AWS US-EAST-1 outage of October 19–20, 2025. A latent race in DynamoDB’s DNS-management automation produced an empty regional endpoint record. Over the next 14 hours 32 minutes, dependent services lost connectivity and recovered on different clocks, and existing EC2 instances stayed healthy while new launches failed. A multi-region data plane still depends on a single-region control plane for operations that require it.

Four 2026 events add the physical-plant counterpart. Coinbase’s June 1 postmortem reports simultaneous chiller failures on May 7 in one AWS use1-az4 data hall. A thermal rack shutdown followed, with roughly eight hours of unavailable or degraded core services and further recovery after that. Surviving zones did not automatically make the application available. Google’s July 25 report describes a July 15 3 ms utility voltage drop at a third-party facility provider's site in europe-west4-a, a failed DRUPS transfer, and a chiller controller that did not restart the pumps while redundant cooling was unavailable during construction. The hall reached 44 °C. The report summarizes 14 hours, 55 minutes of service interruption, with separate timelines for VMware Engine, Bare Metal Solution and NetApp Volumes. This is the shared-control and unavailable-reserve combination in Chapter 12.1.

Nebius’s August 27 post-mortem shows the alarm path failing with the plant. On August 19 a storm took the facility’s building management system (BMS) offline and stopped the chilled-water loop under its us-central1 region. The halls overheated within about two hours, air inlet temperature peaked at 58.5 °C, and thermal protection tripped servers, switches and rack power, control-plane racks included. Because the BMS was also the alarm path, neither the facility operator nor Nebius received a facility alert. Cooling came back after about six hours and full mitigation took about 24 hours.

In both the Google and Nebius cases a facility operator, not the cloud provider, ran the hall. Require your colocation provider to disclose any construction or maintenance state that removes cooling or power redundancy, and adopt the corrective actions Nebius listed: environmental alerting independent of the BMS, a dependency-ordered procedure for cold-starting the whole region, regular recovery drills at region scale, and a written heat-shed procedure for cooling loss. Keep thermal excursion, cooling restoration and service restoration on separate clocks, and derive your liquid-cooled rack’s maintained-flow and stopped-flow response from OEM data and tests, not from these room-air timings.

The fourth event took out a whole region permanently. Drone strikes in March 2026 hit AWS facilities in the UAE and Bahrain, and IRGC-linked media claimed them. In April a second availability zone in Bahrain was also disrupted. In status updates on September 15, AWS said it could not restore Bahrain-only resources and data. The damage went past what regional and multi-AZ designs are meant to survive. In the UAE the gap was limited to data held solely in zone mec1-az2, and recovery continued in the other zones. Multi-AZ covers the loss of a zone, not a region. Back up checkpoints and weights to a second region; a service credit will not bring lost data back. Chapter 11.2 covers the attack itself.

Your inference can span three regions, but a scheduler, discovery layer, secrets store or model-registry metadata in one place still bounds any operation that needs it. Test the recovery region with those primary dependencies unavailable before counting its free GPUs as usable failover capacity.

The dependencies worth enumerating for an AI estate are specific. The control / orchestration plane (scheduler, fleet management, health checking): anchor it in a single region and you have re-created US-EAST-1. The model-weights store: if the failover region cannot serve the current weights because replication lagged, your spare fleet boots into a stale or empty model. Weights replication must be versioned, immutable and ahead of the failover need, not behind it. DNS and global load-balancing: the mechanism you rely on to drain a failed region is itself a global dependency, and it must not share fate with that region. Shared firmware and single-vendor loops: a common-cause defect in a CDU controller, a BMC firmware revision or one liquid-cooling vendor's pump logic can take down every site running that revision together (the thermal-path reliability concern from Chapter 12.2). That is the beta-factor, common-cause problem the quantitative model in Chapter 12.5 exists to size. And people, covered next, because the bus factor on a novel liquid-cooled estate is realer than most plans admit.

Geographic failover answers "what if the site is gone." Business continuity answers the harder, slower questions: what if the people are gone, the supply chain is gone, or the operating environment changes for months rather than hours. These low-probability, high-consequence tails are rarely drilled or funded, and the 2020-2022 period taught a generation of operators to take them seriously.

Key-personnel continuity is sharper for AI factories than for legacy halls because the technology is new. Few engineers can safely intervene on a direct-to-chip liquid-cooled NVL72-class rack mid-fault, or read a specific cluster's NCCL-level failure signatures: often a handful per site, sometimes one. That bus factor is a continuity risk equal to any transformer. The mitigations are unglamorous and effective: runbooks a competent on-call engineer can execute cold (not tribal knowledge), cross-training and rotation so no single person is the only path to recovery, vendor field-service contracts with guaranteed response windows for the loops you cannot self-service, and a refusal to let the on-call roster narrow to one name.

Supply continuity is the slow disaster. AI-factory recovery depends on parts that are themselves allocation-gated. HBM and advanced-packaging supply is contracted a year or more ahead (Micron said on September 30, 2026 that its agreements cover the vast majority of its calendar-2027 HBM bit supply; that is a supplier statement, not an allocation to any buyer; Chapter 2.3). CDUs and quick-disconnects are specialized. High-voltage power transformers carry multi-year lead times, and switchgear takes most of a year (Chapter 4.4 has the lead-time evidence). A spare-parts strategy sized for a legacy hall, a few PSUs and fans on a shelf, does not cover a liquid-cooled estate where the failed part may be a long-lead transformer or an allocation-gated accelerator whose quoted replacement time exceeds the service restoration deadline. Stock the critical-spares list deliberately, hold vendor SLAs with teeth, and accept that some failures are recovered by reconfiguration (shrinking the cluster, re-routing the fabric) rather than replacement.

Pandemic / access-denial continuity generalizes the personnel question. Can the site run lights-out for an extended period with no on-site staff? Can remote hands and zero-touch provisioning carry the load? Can people who cannot physically enter the building execute the runbook? Sites already designed for remote operation and remote hands had the least to change in 2020.

DR is an engineering posture and a set of promises in a contract, and operators get hurt in the gap between the promise and the topology. A colocation or capacity provider's master service agreement carries continuity obligations: availability commitments, maintenance-window rules, sometimes explicit DR or geographic-redundancy clauses. The customer's own SLA to their users sits on top. When a real correlated failure exceeds the topology, the service-credit ladder fires and the reputational and renewal damage compounds. The contract has to describe what the topology can deliver under a realistic failure environment, not under the optimistic independence assumption.

Three contractual failure patterns recur. First, promising an availability number the single-region topology cannot reach: committing to four or five nines on a facility whose control plane, weights store or sole liquid-cooling vendor is a single point of failure. The promise breaks when that common-cause event’s frequency and complete restoration time exhaust the contracted window’s loss budget. Second, silent capacity assumptions: a DR clause that implies failover capacity exists without saying it is pre-energized and reserved, so the obligation is met on paper and impossible in a power-bound outage. Third, mismatched RTO/RPO tiers: selling every workload the same continuity grade, which either over-charges the customer for batch-grade work or under-protects their inference surface. Chapter 12.4 covers availability-versus-goodput SLAs, the penalty and service-credit ladders, and how goodput shortfalls are measured and attributed. Chapter 10.9 covers productizing these commitments to customers, and Chapter 10.11 the serving-side SLOs. One rule connects them: never contract a continuity grade your drills have not proven.

A DR plan that has never been executed is a hypothesis. Regular, adversarial drilling turns it into a capability: fail a region, a control-plane component or a dependency deliberately and on schedule, measure recovery against its RTO/RPO target, and fix the gaps before the real event. This is chaos-engineering discipline applied to the facility and cluster layer. You do not wait for the wildfire to discover that your weights-replication lag exceeds your inference RPO, or that the failover load-balancer shares fate with the region it is supposed to drain.

Scheduled failovers with draining test the easy case. Meta's June 2026 engineering write-up describes "PowerLoss Storm" drills that cut power to whole production regions (co-located buildings sharing power and network) with no preparatory drain, after working up from shadow regions and small regions. They exposed what smaller tests missed: control-plane services with circular start-up dependencies, and shutdown signals that took down the orchestrator itself. Stage your own drills the same way, and include a cold restart of the failed region, not only the failover away from it.

Runbooks make drills repeatable: step-by-step recovery procedures written to be executed cold by a competent on-call engineer who was not in the room when the architecture was designed. A good runbook names the trigger, the decision authority (who declares a failover, since premature failover during a transient is its own outage), the exact mechanical steps, the verification that the failover succeeded, and the fail-back procedure once the primary returns. Fail-back is the step everyone forgets to rehearse, and it is frequently where the second outage happens. Each runbook should map to a specific failure mode in the consolidated FMEA catalog in Appendix F: every mode the FMEA enumerates as estate-significant should have a rehearsed recovery, and every drill should exercise a mode the FMEA flagged. The emergency operating procedures that tie operations to that same catalog live in the commissioning and operations parts. The documentation and acceptance-test discipline that captures the go-live baseline recovery is measured against is in Chapter 13.2.

Choose the least costly posture whose surviving state, compatible capacity and complete drill path meet the workload’s RPO and RTO. A cheaper standby that misses either target buys a recovery promise the service cannot use. More live capacity is justified only when it removes the deadline or loss constraint. Map physical initiators to Appendix F and adversarial initiators to Chapter 11.1, then exercise the same recovery authority and evidence.