Quantitative Reliability & Availability Modeling (RBD / FTA / Monte-Carlo)

Chapter 0.5 gave you the vocabulary: series and parallel, N / N+1 / 2N, the Tier ladder, availability versus goodput. This chapter turns it into a method. You feed in component failure rates and repair times and roll them up into a cluster-level number, with a traceable event, exposure and recovery parent for every availability target and every percent of goodput. The worked case supplies an executable script, a complete synthetic input list and a sensitivity calculation. "We built it 2N, so it's reliable" stops being an assertion and becomes a calculation someone can check, attack and improve.

The techniques stack. Series, parallel and k-of-n algebra extends into real availability arithmetic. Markov state-space models handle repairable systems where crews are finite and degraded operation is a real state, not a binary. Fault trees and minimal cut sets surface the dominant failure path that a redundancy diagram hides. Beta-factors put a number on common-cause failure, where idealized parallel redundancy fails to deliver. Monte-Carlo takes over when the failure environment is correlated, time-varying and non-exponential, and the service's observations require those distributions. A model is only as good as its inputs and its treatment of correlation. The two most common ways an availability model misleads are assuming independence and modeling uptime when goodput is what pays the bills.

Before any algebra, name the target. A traditional IT availability model computes the fraction of time the service is up, the complement of downtime under the state definition you choose. A contractual uptime floor is a number; Tier requirements concern assessed infrastructure outcomes. An AI training cluster cares about goodput instead: useful work delivered within the declared quality and service limits. As a time fraction, it is the share of wall-clock GPU-time that advances the job rather than being lost to failure, detection, restart and recompute-from-checkpoint. It is not MFU, which measures utilization of peak FLOPS. Availability and goodput, in turn, are different functions, and Chapter 12.2 exists because of the gap between them. A facility can be 99.99% available while its cluster achieves 85% goodput. Those are illustrative percentages for the same resource and time boundary: GPU failures, checkpoint restarts and replay eat progress while power and cooling mostly stay present. The two models have different loss terms, but shared facility events still enter both calculations.

Each investment moves a different set of states. Duplicated facility power can prevent or ride through modeled source and distribution-path events. It does not prevent GPU, HBM, optic, software or checkpoint failures. Checkpointing, hot spares and lemon-node ejection reduce workload losses, but they do not keep a hall energized or prove that recovery stays inside the contract. A defensible model declares the target and state set, then compares each mechanism's incremental lifecycle cost with the frequency and consequence it removes. → Chapter 12.2.

The RBD is the workhorse and the right place to start, because it makes the topology's logic explicit before any number goes in. Each block has long-run availability A = mean uptime / (mean uptime + mean restoration time). Check what the source means by MTBF: a failure-to-failure interval that already includes repair must not have repair added again. Independent block states compose by two rules, and these product identities do not require exponential lifetimes. Series (every block must work): availabilities multiply, A = A₁ × A₂ × …, so a chain is always less available than its weakest link and unavailabilities roughly add. Parallel (any one block suffices): the unavailabilities multiply, A = 1 − (1−A₁)(1−A₂)…, so redundancy multiplies small failure probabilities into a much smaller one. That is where the nines come from.

The general case is k-of-n: the system works if at least k of n identical units work. It is the honest model for an N+1 or N+2 power or cooling plant: you need N running, you installed N+1, so it survives any one failure. For independent units each with availability p, system availability is A = Σ(j=k…n) (n choose j) pʲ(1−p)ⁿ⁻ʲ, the probability that k or more of n survive. Shared crews require joint-state probabilities instead. In the high-availability independent-unit regime, moving from N (k=n, all must work) to N+1 (k=n−1 for that installed set) buys a large drop in capacity-loss probability, while further spares yield diminishing absolute reductions. Compare those purchases with faster repair and removal of shared causes. For independent identical units at high availability, the diminishing-returns curve is not intuition: it falls straight out of the binomial, and it supplies the loss-probability side of the Chapter 12.2 tradeoff curve.

The table is an escalation ladder, not a menu. Start with the RBD: it is cheap and forces the topology to be explicit. Move to Markov as soon as repair is shared or the system has a meaningful degraded state, such as a cooling plant that limps at reduced capacity instead of failing outright. Build a fault tree when the RBD is complex enough that the dominant failure path is no longer obvious by inspection. Reach for Monte-Carlo when the thing you care about, the distribution of goodput under a correlated, scale-dependent failure environment, outruns the closed forms; an analytical expected effective training time ratio (ETTR, the share of wall-clock time spent on productive training) remains a useful check on the simulation's mean. Each rung exists because the one below it made an assumption the cluster violates.

An independent-unit RBD often hides a repair assumption: every failed unit gets its own crew the moment it fails. A shared crew couples the repair states and breaks the product, though an RBD can still represent the whole pool as one block using its joint-state availability. Real facilities have finite crews, parts depots with turnaround time, and components that fail into a degraded state rather than a clean binary down. A Markov model captures all three. It represents the system as a set of discrete states (all-up, one-down-repairing, two-down, degraded-running) with transition rates between them: failure rates (λ) push the system toward worse states and repair rates (µ) pull it back. For n equal units and one crew, let i count failed units: i→i+1 has rate (n−i)λ and i→i−1 has rate μ for i>0. With independent unlimited crews the latter rate is iμ. Solve the stationary weights w₀=1, wᵢ=wᵢ₋₁(n−i+1)λ/μ, normalize to πᵢ, and sum the acceptable states n−i≥k. You can then see how much of your unavailability is queueing for a repair crew rather than waiting on a part.

This is where MTTR stops being a single number and becomes a policy. A 2N plant whose second unit is down for 24 hours awaiting a part is running N with no margin for that window, an operating state that disappears when the RBD is reduced to a single availability scalar. The Markov model prices that exposure. If the repair rate µ is slow relative to the failure rate λ, the probability of sitting in the vulnerable degraded state climbs, and so does the second-failure-during-repair term, the one that actually takes the plant down. The decision it surfaces is concrete and expensive: does the next dollar buy another redundant unit, or a faster repair (on-site spares, a larger crew, a depot SLA)? For a plant already at N+1, shrinking MTTR often buys more availability per dollar than adding N+2, because it attacks the dominant second-failure-during-repair term directly. The model tells you which; intuition routinely gets it backwards. → repair-time and sparing inputs in Chapter 14.3.

An RBD is built bottom-up from components. A fault tree is built top-down from the thing you are trying to prevent ("cluster job halted," "hall loses cooling," "tenant SLA breached") and decomposed through AND/OR gates down to basic events with assignable rates. The two are duals, but the fault tree answers a question the RBD does not phrase well: what are the smallest sets of simultaneous failures that take the system down? Those are the minimal cut sets, and ranking them by probability is the output practitioners reach for first. A first-order cut set, a single basic event that alone causes the top event, is a latent single point of failure inside a topology you thought was redundant. A model that surfaces one shared, un-redundant component buried under two layers of N+1 has earned its keep. An RBD can represent that element too; the fault tree’s job is to challenge whether the event inventory left it out.

The recurring AI-data-center example is the cooling distribution unit. The RBD shows N+1 CDUs and reports comfortable nines. The fault tree, traced to basic events, reveals that all of them share a single secondary-loop isolation valve, a single facility-water supply or a single controls PLC: a first-order cut set that the parallel block diagram drew right over. The same pattern recurs in a shared busway feeding ostensibly independent power paths, a single BMS that gates every chiller, and a single firmware image across every BMC. Push the tree down to basic events (explicit hardware, software, control or procedural events with a defined occurrence and recovery basis) instead of stopping at the convenient block boundary, because that boundary is exactly where the shared dependency hides. → the FMEA basic-event catalog in Appendix F.

Common cause is the term that most often separates an overstated availability number from a defensible one. Parallel redundancy delivers its nines only if the redundant units fail independently, and they never fully do. A shared cause defeats redundancy by failing the units together: a firmware bug pushed fleet-wide, a coolant chemistry problem, a single controls fault, a power transient that hits every unit at once, a maintenance error repeated on each "redundant" path. The standard model is the beta-factor: a fraction β of a component's failures is assumed common-cause and hits all redundant units simultaneously, while the remaining (1−β) fail independently. IEC 61508-6:2010 Annex D maps hardware common-cause scoring to β bands of 0.5% to 5% for logic subsystems and 1% to 10% for sensors and final elements, as described in SINTEF’s 2015 common-cause study. Those bands come from Edition 2.0. Edition 3 went out as committee draft for vote IEC 65A/1171/CDV in 2025 and reached the FDIS stage in October 2026, according to the IEC project record as mirrored by Montenegro's standards institute. Note which edition your beta comes from, and check Annex D again once Edition 3 is published. Use the licensed scoring procedure for your subsystem; these bands are not a generic data-center beta or a measured fleet failure rate.

The consequence is severe and counterintuitive. An idealized parallel pair of units, each at 99.9% availability, computes to about six nines (1 − 0.001²). Add a 5% beta-factor, assuming 5% of failures are common-cause, and system unavailability becomes β·U + (1−β)²·U² ≈ 0.05 × 0.001 + 0.9 × 10⁻⁶ ≈ 5.1 × 10⁻⁵. That is roughly four-and-a-quarter nines, because the common-cause term, not the independent-failure product, now dominates. Where the shared event can be named, such as a controls PLC with its own failure rate λs and restoration rate μs, model it explicitly as a series element with availability μs/(λs+μs), specify its recovery distribution, and reserve β for the causes still unrepresented. β alone cannot convert component unavailability into a service result. Redundancy cannot buy you below the common-cause floor. Diversity is worth paying for: different firmware versions across redundant controllers, A/B coolant sourcing, independent power-transient paths, staggered maintenance windows and, above all, never pushing the same firmware to every redundant unit on the same day. A model that omits shared causes can claim nines that those causes erase. One that represents the shared events explicitly does not need to add beta again for the same failures.

Closed-form methods buy tractability with assumptions the cluster violates: independence, constant rates, exponential repair, a single target. Monte-Carlo trades a compact algebraic answer for repeated samples of the event and recovery process you define, so its honesty still depends on what those samples represent. You build a generative model. Sample each component's time-to-failure from its real lifetime distribution (Weibull for wear-out, empirical for the burn-in hump). Inject correlation explicitly: a common-cause event that fails a set of units together, a grid transient that hits the whole hall. Model the repair queue with its finite crews and depot delays. Then run the clock thousands of times and read off the distribution of outcomes, not just a point estimate. The output is a distribution of annual downtime, goodput and bad windows, with confidence intervals for finite-sample error kept separate from uncertainty in the input rates. Analytical methods can also produce distributions where their assumptions fit.

For an AI cluster, the same simulation produces both targets at once. Layer the workload model on top of the failure model (checkpoint interval, detection time, restart-and-recompute cost, lemon-node ejection policy) and the run yields goodput directly. Each sampled GPU failure costs a detection delay, a restart and a recompute of the work since the last checkpoint, and the share of GPU-time that survives is the goodput. Restart-all should not be hard-wired. Give the model separate recovery states: repair in place when device memory survives, swap in a spare that holds replicated in-memory state, keep running degraded, or restart the job from a checkpoint. Each state needs its own trigger, latency and fallback. A good recovery class can turn a restart plus lost work into a loss of seconds. Chapter 10.7 covers the mechanisms and their published results. In this chapter, restart-all is the conservative bound.

Simulation answers questions such as "if I halve my checkpoint interval, what happens to my goodput distribution at 100k GPUs?" The answer depends on a failure rate that scales with cluster size, a recompute cost that depends on the interval, and a correlation structure the simple algebra drops. Simulation is not the only honest route: Meta's closed-form expected-ETTR result gives a cheap baseline for the mean, and renewal or state-space treatments carry much of the rest. What simulation buys is the distribution and its tail, which an expectation cannot give you. The cost is calibration. A Monte-Carlo result is only as good as the rates and correlations you feed it, which is why this method takes the Chapter 14.3 AFRs and the Appendix F scenarios as raw inputs instead of inventing them.

For the CDU pool, λ/μ = 0.004 gives stationary weights [1, 0.012, 0.000096, 0.000000384]; the acceptable numerator is 1+0.012. For the power pool, the weights are [1, 0.008, 0.000032] and the acceptable numerator is 1+0.008. Normalize each by its full sum. Shared-control availability is 0.5/(0.5+0.0005). Multiplying the independent pool availabilities gives expected annual downtime of (1−A)×8,760, about 9.9 hours/year. The top event is S OR (P1 AND P2) OR any CDU pair, so the minimal cuts are {S}, {P1,P2}, {C1,C2}, {C1,C3}, {C2,C3}. Removing S leaves about 1.1 hours/year from the two equipment pools.

The script’s reduced node-only renewal check gives about 90.9% goodput, and it uses the compute, save and recovery timing from Chapter 9.4. NIST’s r-out-of-n method supplies the independent-state comparison; the shared-crew recurrence above is the guide’s explicit state extension.

The point of the method is the roll-up: take the per-component failure rates from Chapter 14.3 and the failure scenarios from Appendix F and compose them, level by level, into one number for the whole cluster. The path runs: component AFRs → per-node failure rate → RBD/k-of-n for the redundant subsystems (power, cooling, network core) → fault tree to catch the shared cut sets → renewal, state-reward or event simulation for synchronous-job goodput → cluster-level availability AND goodput. Each level has a method matched to its question, and the levels compose. The facility’s event rates, durations and affected load feed the goodput process; a power event that interrupts required ranks is another correlated failure that stalls the job. Mean availability alone cannot identify the recovery cost. Population-time equipment failure estimates derived from unique FRU events become one input to the goodput run, but Meta's job-interruption counts do not supply component AFR or per-node λ.

Check the roll-up against ground truth before you trust it. Validate a goodput model against the observation window and badput decomposition of the fleet or job it describes. Calibrate against that fleet’s own measured event log, show sensitivity to event definition, workload mix, checkpoint policy and correlated failures, and do not tune failure rates merely to reproduce a target goodput. Verify that the modeled state behavior satisfies the Tier outcome you selected, then calibrate numerical availability against the facility's incident and maintenance history or an explicitly chosen empirical dataset. The model is ready for field use when its number survives comparison with relevant published benchmarks and predicts an untouched incident-history window with matching event definitions, exposure and workload boundaries. The worked case in this chapter proves arithmetic, not field calibration. → AFR inputs and the failure taxonomy in Chapter 14.3; the FMEA scenario catalog in Appendix F; the checkpoint math behind the goodput term in Chapter 9.4.

The deliverable is a ranking, not a single availability number. Once the model is built and calibrated, compute the sensitivity (importance) of the result to each input: how much cluster availability or goodput moves per unit of investment in this component, this redundancy step, this MTTR reduction, this checkpoint interval. That ranking is the engine under the Chapter 12.2 tradeoff curve and the answer to the only question the CFO actually asked: where does the next dollar of redundancy buy the most?

The rankings routinely surprise. On the availability side, shrinking MTTR with on-site spares, a depot SLA or a larger crew attacks the second-failure-during-repair window directly and can beat another redundant unit. Removing a shared cause can beat both when that event dominates. Compare all three in the same state process and cost horizon. At the worked prices below, positive incremental value decides the choice, and changed shared-event exposure can reverse it. On the goodput side, Meta's revised RSC-1 analysis shows how hard the levers bite at scale. Take the largest feasible run, about 12,000 GPUs. To get ETTR near 0.9, Meta's model needs either checkpoint writes of about 10 seconds or a failure rate of about 1 per 1,000 node-days instead of 6.50. The lemon-node result is observed, not modeled: ejecting suspect nodes cut failures of 512+-GPU jobs from 14% to 4%. Use the model's sensitivity ranking to spend on the dominant interruption and recovery states of the service you run: checkpoint and node policy in Meta's case, but liquid-flow continuity, replica placement or facility path states in others. The sensitivity analysis is what proves that to a skeptic with a budget, and it is why this model, not intuition, should set the redundancy spend. → the tradeoff curve it feeds in Chapter 12.2; the SLA commitments it underwrites in Chapter 12.4.

Under these assumptions, choose fast recovery: it recovers about 500,000 GPU-hours/year and has the largest positive net value. The combined fast-plus-control run reaches about 96.4% goodput and adds about $9,000/year net over fast alone, so buy both at these prices. Recompute that marginal value after any input change. The other equipment options fail this discretionary economic screen at their assumed prices; they remain mandatory if a required state needs them. Flip: fast recovery stops paying for itself when its annual increment exceeds ΔG×1,024×8,760×$2.0, about $1.0 million/year at the base rates. Increasing shared-control event frequency tenfold lowers base goodput to about 89.7%; buying more independent units cannot remove that shared-event loss.

The base annual mean is about 90.8%, and the simulated annual fifth-to-95th-percentile range is about 90.4%–91.2%. The simulation reports its own sampling error on the mean; the runnable script above reproduces only the analytical checks. Neither that sampling error nor the window percentiles measure uncertainty in the assumed rates. Take the hypothetical 92% goodput floor and credit bands of Exhibit T in Chapter 12.4. Under them, the separate base 720-hour simulation breaches in 1,880 of 2,000 windows, with a mean credit of about 12.8% of the affected-service fee; fast recovery has zero breaches in this sample. Zero simulated breaches is not zero risk. Vary rates and recovery inputs before pricing a monthly promise, then validate on a held-out fleet window with matching exposure definitions.

The same modeling errors recur, each producing a number that is precise and wrong. Four stand out:

• Using the workload metric as a topology selector. A training goodput model that omits facility maintenance and fault states can under-provide continuity. A facility RBD that omits silicon, checkpoint and recovery losses can overstate delivered value. Model both, then compare N, N+1, distributed or 2N designs with compute-stack and fleet alternatives. → Chapter 12.2.
• Assuming independence. Reporting the parallel-product nines without a beta-factor, six nines that a 5% beta collapses to roughly four, while omitting the shared bus, CDU loop, controller or fleet-wide firmware push that can fail every ‘redundant’ path together. An essential shared series element caps the block at its own availability, so model its rate and recovery plus repair coupling, and use beta only for residual causes not already counted. This is the most common way an availability number is overstated.
• Stopping the fault tree at the convenient block. Drawing N+1 CDUs as independent parallel blocks without pushing to basic events, and missing the single shared valve or PLC that is a first-order cut set. The redundancy diagram drew right over the single point of failure. → Appendix F.
• Modeling once and reusing across scale. Taking the goodput number from a 16k-GPU model and assuming it holds at 100k. When per-node faults are independent and equal, a restart-all job’s MTTF falls roughly inversely with node count, and changed recovery algorithms, placement and shared events can alter that scaling. Re-run the model at each point on the ramp. → Chapter 14.3.

Collect the event counts, operating exposure and complete recovery traces from Chapter 14.3 before committing to the modeled service gain; guessed rates cannot support that promise. Appendix F owns physical failure-mode identifiers, and Chapter 11.1 owns adversarial initiators. This chapter owns their stochastic consequences.