Resilience Standards, Redundancy Topologies & Fault-Domain Engineering

Resilience is the one domain where the industry's vocabulary misleads the AI buyer. The standards everyone quotes (Uptime's Tiers, TIA-942's Rated levels, EN 50600's Availability Classes) answer a building question: what infrastructure keeps power and cooling reaching the servers through the maintenance and fault states the scheme defines? Uptime rates the site infrastructure topology for power and cooling; TIA-942 and EN 50600 rate wider facility scopes. None of them rates the servers or the job. They say nothing about whether a 50,000-GPU synchronous training job survives a single bad NIC, nothing about the thermal response of a specific rack at its operating profile, and nothing about goodput. An operator who treats a Tier rating as the complete resilience design basis has answered the building question and left the job-recovery question unanswered.

This chapter does three jobs. First, it maps the standards landscape and says where each standard stops being useful for an AI factory. Second, it prices the redundancy topology ladder, from N through 2(N+1) and the block-versus-distributed fork that governs hyperscale efficiency. Each rung is priced in capital and in capacity held back from normal use, and a worked power package compares two topologies at equal usable load. Third, it installs the lens that recurs through all of Part 12: fault-domain and blast-radius engineering, which means deciding on purpose how much of your cluster the worst single failure may take down.

Availability versus goodput is defined here in one line, then forwarded. Chapter 12.2 carries the full rethink, with redundancy migrating out of the building and into silicon and software. The redundancy vocabulary comes from the primer (Chapter 0.5), and the quantitative math that turns a topology into a number is in Chapter 12.5.

Three classification families dominate globally. They certify different scopes against different criteria, so they are not interchangeable, and a project that conflates them ends up overpaying for one rating while under-specifying the property it needed.

Uptime Institute Tier I–IV is the de-facto global resilience language. Its real content is two properties, not a percentage. Concurrent maintainability (Tier III) means every capacity component and distribution path can be taken out of service for maintenance without dropping the IT load. Fault tolerance (Tier IV) means the topology sustains a single unplanned worst-case failure in normal configuration without dropping load, and is also concurrently maintainable. The two properties are cumulative, not simultaneous: Uptime is explicit that with a redundant component or path shut down for maintenance, the site carries a higher risk of disruption should a fault then occur. Uptime certifies in three flavors: Design (Tier Certification of Design Documents), Constructed Facility, and Operational Sustainability (the people-and-process layer). For years it has actively disavowed the retired availability table that vendors still quote. That table was never the standard; the topology properties are.

ANSI/TIA-942-C (2024) rates the whole facility across four subsystems (Telecommunications, Architectural/Structural, Electrical and Mechanical) on a Rated-1 to Rated-4 scale. Unlike Uptime, it explicitly covers cabling, pathways and the building envelope. The 2024 (C) revision added accommodations for AI-driven density growth and sustainability. A facility is rated to its weakest subsystem, which is a feature: it stops you buying Rated-4 electrical and forgetting the structural slab.

TIA opened Addendum 1: Artificial Intelligence to 942-C in March 2026 (TR-42.1). It covers high-density and high-speed cabling, liquid cooling and electrical for AI/HPC, and targets mid-2027. TIA’s companion quality-management-system standard for data-center physical infrastructure is DCE 9000, run through the TIA QuEST Forum. TIA sent the first draft to participating organizations for formal review on 14 October 2026, with comments due by 30 November, and targets publication in the first quarter of 2027. As of October 2026 neither DCE 9000 nor the addendum is issued. Both remain facility and supply-chain initiatives, and neither guarantees cluster goodput. Buy against the issued standard and bring later revisions in through change control.

EN 50600 / ISO/IEC 22237 is the international and European modular family. Read its availability classes from the parts you select and their national adoption; do not translate them from a Tier numeral. It adds separate Protection Classes for physical, fire and environmental security, and folds in ISO/IEC 30134 efficiency KPIs (PUE/WUE/REF). EN 50600 is a common choice in European procurements; elsewhere buyers often write Uptime Tier or TIA-942 ratings into their contracts. Whichever you pick, put the scheme, edition and parts in the contract text.

US specifications often cite a fourth document, ANSI/BICSI 002-2024. It is a design-practice standard, not an assessment scheme: BICSI credentials designers but awards no facility rating. Its Availability Classes F0 to F4 run from a single path with no redundancy to a fault-tolerant design, and F3 and F4 correspond roughly to concurrently maintainable and fault tolerant. Treat an "F3" line in an RFP as a design requirement to turn into maintenance and fault states, not as a rating anyone will certify (Appendix A lists the edition).

Attach an applicability record to the design basis. The owner selects the scheme, issued revision, assessed phase and load, and award type. The design lead supplies the component and path state matrix, and the assessor evaluates the evidence the scheme requires. The facility, CDU and IT owners identify the handoff at each power and cooling interface. Treat an OEM reference architecture as an interface input, not an as-built certificate. Appendix A owns document identities, Chapter 11.10 owns OT restoration access, and Chapter 12.3 owns continuity authority and exercises. Reopen this record after changes to density, cooling, controls or topology. Required fire and emergency shutdown actions still limit what uptime you can promise.

A Tier rating is achieved by a redundancy topology, and the topology, not the rating, is what you buy, install and pay to operate. The ladder is a sequence of decisions, each trading capital and stranded capacity for a different failure-survival property. Every rung up costs real money and idle equipment. The right rung is the cheapest over its life that still carries the load through the maintenance and fault states you must survive, with the post-event capacity and recovery time your contract requires.

N is exactly enough capacity to carry the load and not one unit more; any failure or maintenance event takes capacity offline. N+1 adds one redundant unit to a set (one spare CDU, UPS module or chiller). It absorbs a single component failure or lets you service one unit, so it keeps full capacity through one component loss or one maintenance event, but it does not keep a distribution path alive by itself. N+2 tolerates two concurrent failures in the same set, which matters where repair logistics are slow or the component population is large. 2N is two fully independent systems, each able to carry the whole load. It is clean and simple as a two-path capacity diagram, but its fault tolerance depends on independent controls, protection, load connections and successful transfer, and its capex and TCO are project-specific because it duplicates the full power path. Quantify the premium from the actual bill of materials, and do not conflate 2N with N+1. 2(N+1) stacks a spare into each of the two halves, for contracts that require each mirrored path to stay maintainable while it still holds a spare.

The block-redundant vs distributed-redundant fork trades idle reserve for better equipment utilization and a different service boundary. Both escape the mirror premium of classic 2N, which at full load holds half its installed capacity in reserve: with the load shared evenly, each side runs at no more than 50% (Chapter 0.5 defines the vocabulary). Block redundancy, the catcher topology, dedicates one reserve block that backstops several active blocks through static transfer switches, so any one block can fail to the shared 'catcher' without every block carrying its own full mirror. Distributed redundancy (3N/2, 4N/3 and similar load-sharing ratios) spreads the load across three or more systems sized so the survivors absorb a failed system's share.

The payoff is utilization: each unit runs far closer to its rating, which recovers capital otherwise parked in reserve equipment. Any PUE improvement must come from the actual equipment loss curves and operating schedule. Whether the design matches 2N's fault tolerance depends on the specific design (the transfer scheme, control independence and the post-event load each survivor is rated to carry), not on the ratio. The cost is switching complexity and a heavier protection-coordination and commissioning burden: more transfer events, more failure modes to test and more ways to mis-wire.

For an AI campus measured in hundreds of MW, distributed redundancy wins wherever the 2N utilization penalty, multiplied by AI density, exceeds the cost of the extra transfer paths and their commissioning. That is a project-by-project calculation from block size and rated post-event loading. The deciding estimate must include transfer equipment, control independence, maintenance access and the extra witness scope.

At full required load, normal aggregate utilization is 3/6 = 50% for 2N and 3/4 = 75% for 4N/3. One lost string leaves 3.0 MW in either design. A maintenance isolation plus another full-string loss leaves 0 MW and 2.0 MW respectively, so both fail a 3.0 MW continuity requirement for that combined state. A shared controller that disables all delivered power also fails; additional rated capacity does not repair that path.

Initial cost is 2×$4.200m + $0.600m + $0.300m + $0.600m = $9.900m for 2N, and 4×$1.600m + $1.100m + $0.450m + $0.800m + $0.250m = $9.000m for 4N/3. Distributed redundancy saves $0.900m up front, as the utilization argument predicts, and gives it back in operation. Add ten years of the annual costs and the totals are $11.850m versus $11.900m. A $50,000 difference on about $11.9m is a cost tie at the precision of these assumed inputs. The operating states decide: choose the topology whose isolation, selective clearing, controls, dual-cord behavior and load-step test results prove the states you need. Until that proof exists, buy neither; if both pass, these costs do not separate them.

Flip: before the extra transfer witness scope, 4N/3 costs $11.650m, so a $200,000 distributed-only addition is the exact arithmetic crossover. Crossing it changes which total is lower, not the test result or the practical tie near that point. A topology that fails a required state is out at any price until a test proves that state. The trade is transfer complexity against installed reserve, not a Tier premium. Uptime’s outcome-based method supplies the topology boundary; Chapter 13.3 owns power acceptance and Chapter 12.5 prices interruption risk.

The most useful idea in the Tier ladder is that concurrent maintainability and fault tolerance are different properties, and the AI operator should unbundle them. The ladder stacks them cumulatively: Tier III delivers concurrent maintainability on its own, and Tier IV adds fault tolerance on top of it. A rung name therefore buys a bundle of outcomes, not the specific maintenance and fault states your service has to survive.

Concurrent maintainability (the Tier III line) means you can take any single capacity component or distribution path out of service, for a firmware update, a pump rebuild or a breaker swap, without dropping the IT load. It is about planned events, and over the facility's life the approved maintenance program can create more interventions than the unplanned-fault count, so price their actual frequencies separately. Fault tolerance (the Tier IV line) means the topology survives a single unplanned worst-case fault, such as a transformer that explodes or a controller that hangs, with no load loss, and it must also be concurrently maintainable. The jump from III to IV is the jump from 'no downtime for maintenance' to 'no downtime for maintenance, and none for a worst-case fault in normal configuration'. It is not a promise that both hold in the same instant.

Savills’ May 2024 report relays Dgtl Infra’s 25–40% Tier IV-versus-Tier III construction-and-fit-out heuristic, while Schneider’s White Paper 122 illustrates how fully redundant 2N power can more than double ten-year power-system TCO versus 1N. These rest on different cost boundaries and source assumptions, so they are not interchangeable project multipliers. Price the actual strings, transfers and operating costs at equal usable load, as in the power-package case above.

Checkpointing changes the economic consequence of a training interruption, but recovering a job after a node loss does not prove that a whole-hall power or cooling interruption is acceptable. Work the maintenance states and faults through first (the callout below lists them). That analysis may select concurrent maintainability without full-path fault tolerance; N+1 or distributed power is then its conclusion, not something the checkpoint supplies. → Chapter 12.2.

A fault domain is the set of equipment that fails together when a shared element fails. A blast radius is how much of your useful capacity the worst single failure takes with it. You choose blast radius at design time rather than discovering it on the worst day: by deciding where to draw the boundaries (electrical block size, cooling-loop isolation, fabric pod, scheduler placement domain), you decide in advance how bad that day is allowed to be.

The four boundaries that matter most for an AI factory, and the fork each presents:

• Electrical block. How many racks share a transformer, a switchboard, a UPS, a generator? A larger block is cheaper per MW and simpler to wire, and it is also a larger blast radius. Block size is the first and most physical blast-radius decision. It interacts directly with the redundancy topology, because a catcher block or distributed-redundant margin only helps if the blocks it backstops are sensibly sized. → Chapter 4.1.
• Cooling loop. How many racks ride a single CDU, facility-water branch, valve or control function? A CDU failure can spend the rack’s thermal margin before a human reacts, whenever the interruption the rack's OEM rates it to survive is shorter than the response. N+1 pumps or N+1/2N CDUs are therefore a first-order reliability investment where they preserve the required cooling path. Test loss of flow separately from loss of heat rejection, then size the surviving path from the specific rack and loop. → Chapter 5.6; the thermal reliability boundary is Chapter 12.2.
• Fabric pod. How many GPUs sit behind a single leaf/spine group or a single rail? A fabric cut that spans electrical and cooling boundaries can create correlated-but-misaligned failures that the scheduler cannot route around if the surviving paths lack the bandwidth or ranks the job needs. Map those cuts against actual placement before deciding the domains are independent. → Chapter 8.5.
• Scheduler placement domain. The software that decides which GPUs run which job is the last line of blast-radius defense. Placement that is aware of the physical fault domains can keep a single job off a single failure boundary. Where the training stack can keep making progress on a surviving replica group, placement can also spread a job so one block's loss costs a fraction of the run rather than all of it. Under a restart-all synchronous job, spreading does not bound the loss; it only changes how many jobs a single block event interrupts. This is where facility fault-domain engineering and cluster software meet.

Make these four boundaries visible to placement and recovery. Align and size them as commensurate blocks where that bounds a job’s loss, but keep replica domains independent where survival requires it. The failure mode to engineer against is the single point of failure that silently spans domains, such as a shared controller bus, a common firmware image or a single make-before-break busway. It turns what you thought were four independent blocks into one large correlated one. Common-cause failure can matter more than component MTBF once redundancy has suppressed independent losses: after you N+1 everything, the residual risk concentrates in the things every redundant unit shares. Shared firmware that updates all your CDUs on the same night, a single SCADA controller behind both UPS halves and a common cooling chemistry are blast-radius-spanning dependencies to represent explicitly. Use a residual beta term only for common causes not already counted. → the math is Chapter 12.5; the failure catalog is Appendix F.

Prove the boundaries by removing them. Meta describes catching circular bootstrap dependencies early, in its Belljar CI tests, and then running zero-notice power-off tests up to whole production regions without pre-draining. At region scale the tests found weaknesses that testing one fault domain at a time inside a single data center had not exposed. The largest fault domain may also be the whole site. When drone strikes in March 2026 damaged AWS's UAE and Bahrain sites beyond what its regional and multi-AZ services are built to withstand, some data could not be restored, so recovery state belongs outside the region (→ Chapter 12.3).

This chapter owes you one-line definitions before Chapter 12.2 spends a whole chapter on them. Measured availability is the fraction of time the facility or service stays inside a defined operating state. Uptime Tier criteria describe topology properties rather than predict an availability percentage. Goodput is useful work delivered within the agreed quality and service limits; as a time fraction, it is the share of wall-clock time in which the cluster advances the job (effective training time, or SLA-conforming inference), net of failures, restarts, checkpoint overhead and stragglers. It is not model FLOPS utilization (MFU). Availability and goodput diverge. A facility that meets its contracted topology outcome can host a training run achieving 70% goodput if the cluster fails every few hours and the checkpoint cadence is wrong. A facility with fewer redundant paths can still host high-goodput work when its measured interruptions and recovery times stay inside your uptime and recovery targets. Meta's Llama 3 405B run, interrupted about once every three hours, still kept more than 90% effective training time (Chapter 12.2 breaks down the causes). Facility-state availability and workload goodput are separate decision inputs; neither substitutes for the other.

The commissioned facility outcomes feed those targets, but goodput is the number that governs return on a multi-billion-dollar cluster. The standards landscape, the redundancy ladder and the fault-domain lens in this chapter are the inputs to that rethink: the design-basis facts about what the building can do. What you should target, and where redundancy should live, is Chapter 12.2. How goodput becomes a contractual term with penalties is Chapter 12.4.

Buy the Tier label your customers or lenders need, then buy redundancy path by path from the maintenance and fault states you must survive, and test it. Compare package costs at equal usable load. A cheaper capacity ratio that lacks either tested transfers or independent controls has yet to prove it can carry those states. The certificate stops at the facility, so set the workload's recovery boundary separately (Chapter 12.2).