Spares Strategy, RMA Logistics & Repair Operations

Once a cluster reaches steady state, the reliability question shifts from whether hardware fails to how fast you put it back. Chapter 14.3 quantified the failure rates, and Chapter 14.4 covered the training-resilience software that survives a failure. This chapter covers the physical economy that closes the loop: the spare on the shelf, the technician with the FRU, the box going back to the OEM, and the board on a repair bench. Scoping rarely models the three things that govern it: how deep the spares pool is, how close it sits to the rack, and how fast a failed unit can be dispositioned and replaced. A fleet with excellent component reliability can still bleed goodput once the bottleneck moves from the failure rate to the replacement rate.

Each choice has a downstream cost. Spare too thin and jobs stall waiting on parts that are weeks out on an RMA. Spare too deep and millions in depreciating silicon sit on a shelf, against a contested 2–3-year bear-case economic clock (→ Chapter 1.8). Route every failure through the OEM warranty channel and turnaround is measured in weeks. Self-spare and self-repair and you carry the inventory and the engineering, but you swap in minutes. Design for tray-level FRUs and floor MTTR is short but the spares bill of materials is expensive. Design for board- or module-level and you save inventory dollars, but technicians need more skill and more time at the rack. The treatment here is global and vendor-neutral, current to 2026.

Size a spares pool from a forecast. For each failure class, estimate unique failed FRUs over a population-time exposure matched by generation, age, duty and failure definition. Convolve that distribution with replenishment lead time, then add a service-level buffer for variance and stockout consequence. Do not substitute job-interruption counts for equipment AFR. Meta’s Llama 3 report gives GPU- and HBM-attributed interruptions for one run, not unique failed devices or fleet exposure, so it cannot support a universal accelerator AFR or spares percentage. Replacement lead times remain gated upstream by CoWoS packaging and HBM allocation (→ Chapter 2.3), and you must measure them for the contracted part and logistics path.

A flat “2% of the fleet as spares” fails because failure rates are wildly non-uniform across the bill of materials. Optics, cables and GPU boards need separate measured demand rates. HBM-attributed job interruptions do not by themselves identify failed GPU FRUs. Power-supply and fan FRUs fail on their own curves, and CDUs, manifolds and quick-disconnects fail on a mechanical and fluids curve that has nothing to do with silicon. A single flat percentage over-spares the reliable parts and under-spares the failure-prone ones. What works is sizing each failure class against its own AFR and lead time, which is why the disposition table below is organized by component, not by a global ratio.

Trace. Mean demand during delivery is μ = (12 FRU / 100 days) × L days. At L = 10 days, μ = 1.2 FRU. For Poisson demand D, P(D ≤ S) = exp(−μ) × Σ(k=0…S) μ^k/k!. Three usable units cover 96.6231% of cycles and four cover 99.2254%, so buy four, the smallest stock that meets the target. That is 99% cycle coverage, not a 99% item fill rate, and it says nothing about correlated failures.

Flip. At L = 20 days, μ = 2.4. Six units cover 98.8406% and seven cover 99.6661%, so buy seven. Solving P(D ≤ 4) = 0.99 gives L = 10.659217… days, about 10.66 days. Planning in whole days, treat 10 days as the limit, because 11 days gives only 98.8675%. Either expedite to that limit or deepen stock. NIST’s Poisson CDF supplies the method. Variable delivery time, burst demand and repair returns need their own distribution or a simulation.

The same method on compute trays. The harder sparing decision in 2026 is the GB300 NVL72 compute tray: low demand, a long RMA turnaround, and a degraded 72-GPU NVLink domain while you wait. Take 200 racks, or 3,600 trays at 18 per rack, and assume 0.1 tray failures a day across the fleet (about 1% of trays a year). That rate is a teaching value; substitute your measured one. At a 45-day RMA turnaround, μ = 4.5: nine spare trays cover 98.29% of cycles and ten cover 99.33%, so hold ten, or 40 GPUs. At 90 days, μ = 9: sixteen cover 98.89% and seventeen cover 99.47%, so hold seventeen, 68 GPUs of idle silicon and nearly a full rack's worth of compute trays. Five times the failure rate does not mean five times the pool: at 0.5 failures a day (about 5% of trays a year), a 99% target takes 34 spares at a 45-day turnaround and 61 at 90 days.

Without a spare on the shelf, the rack runs on 68 of its 72 GPUs until a tray arrives, and a job that needs all 72 cannot use it at all. The most common GB300 contract pattern is what SemiAnalysis calls an “NVL64+” SLA (see the dispositions below). One failed tray leaves the rack counted as up; a second failure in that rack before the replacement arrives drops it to the 64-GPU floor; a third failure takes the rack down. Halving the turnaround saves seven trays of capital. That is the value of an advance-replacement term in the OEM support contract, provided the term really does bring the wait down from 90 days to 45. At 90 days, holding a whole spare rack starts to compete with a tray pool. Chapter 14.3 owns equipment exposure, Chapter 14.1 accounts for lost useful output, and Chapter 1.3/1.8 value the stockout and carrying cost.

RMA (Return Merchandise Authorization) is the formal channel by which a failed unit goes back to the OEM for warranty repair or replacement. At fleet scale it is a pipeline with measurable stages, and the goodput you lose is the integral of how long a unit spends in each: detect → triage/qualify → swap → return → repair/replace → replenish. The expensive, often-skipped stages are the bookends. Triage/qualify decides whether the unit actually meets the RMA criteria. An OEM will reject a return that its field-diagnostic tool says is healthy, and a rejected RMA is wasted shipping plus a still-empty slot. Replenish is where a shallow pool fails. A local spare refills the slot in minutes for a discrete FRU, and in under an hour for an HGX node that has a hot spare. An NVL72-class tray fault instead means a degraded rack or a rack swap. The spare you consumed, though, is not back on the shelf until the RMA closes weeks later. So RMA latency does not gate floor MTTR; it gates how fast your pool recovers, and a slow RMA pipeline with a shallow pool is how you walk into a stockout three failures deep. The commodity-PSU case above shows the shape: at the same 99% cycle-coverage target, doubling the replenishment interval from 10 to 20 days raises the stock you must hold from four units to seven.

The qualify stage is more disciplined than most operators expect, and it is where RMAs get rejected. For GPU memory, the vendor publishes objective thresholds you can check in the field. NVIDIA sets the row-remap failure flag when a further uncorrectable-error remap is attempted on a bank that already holds eight such rows, on a remap of a row already remapped, or after 512 total remappings. The RMA criteria are met only when that flag is set and the NVIDIA Field Diagnostic validates it, so a counter of eight is not an automatic return. On Blackwell, a third remap attempt on a bank invokes an on-package HBM channel repair against a spare channel, which recovers the part and avoids the RMA when it succeeds (NVIDIA GPU Memory Error Management docs, 2025). Read the row-remap failure flag in-band (NVML/nvidia-smi) and out-of-band (SMBPBI), so triage is automated and your returns are accepted on the first pass instead of bounced back across a customs border.

Every failed unit gets one of four dispositions, and choosing wrong is either slow or wasteful. Anything field-replaceable defaults to a hot-swap from local spare. An optic, PSU or fan swaps in minutes, and the failed unit gets handled offline. For an HGX node with a hot spare, SemiAnalysis’s ClusterMAX 3.0 fault-injection tests (Sept 2026) set the bar at back in service in under an hour, end to end. The best automated providers managed 24–35 minutes. Providers that needed a manual approval or SRE steps took 43–57 minutes. On faults that were not handled automatically, one node took two hours to rejoin, and another provider replaced its node by hand “within hours.” An NVL72-class compute tray is different. A tray fault knocks four of the 72 GPUs in an NVLink domain offline. Providers usually do not hot-swap a tray into the live fabric; they run the rack degraded or replace the whole rack. Contracts price that in: the most common GB300 NVL72 SLA pattern SemiAnalysis sees is what it calls “NVL64+”, under which the rack counts as down only when three or more of its 18 compute trays have failed at once. SemiAnalysis’s fault-injection tests also timed a GB300 node: at one provider it needed 8 h 40 min to return to service (→ Chapter 10.5). RMA to OEM sends the unit back under warranty: zero repair cost to you, but weeks of pipeline latency and a spare consumed in the interim. Depot board-repair sends the unit to a specialist bench that reworks at component level (reflow a failed VRM, replace a fan module, re-seat or replace an optic). It suits boards out of warranty or failures in a cheap discrete part, but it needs a repair partner and a logistics leg. Harvest/cannibalize strips a write-off unit for its still-good FRUs. It is the rational end-state for a board with a dead GPU package but healthy NVLink, power and cooling subassemblies, especially once the generation is off the new-build supply and the parts are otherwise unobtainable.

Which of these you can reach depends on FRU granularity. A fault domain that is replaceable as a small FRU (a single optic, power supply, fan, or GPU board on an OAM/UBB-style baseboard) is fast and cheap to swap and cheap to spare. A fault domain welded into a large integrated assembly (a full compute tray, a fused NVLink backplane, a sealed liquid-cooled module) is slower to swap, costlier to spare, and more likely to force an RMA of a large, heavy, valuable unit when only a small part failed. NVIDIA says the Vera Rubin NVL72 (shipping since June 2026, → Chapter 5.4) replaces cabling inside the compute tray with a PCB midplane; GB300 NVL72 trays keep their cable-based connections. The fault domain is bigger, though: a tray fault degrades a whole 72-GPU NVLink domain, and every compute-tray swap means handling fluid. NVIDIA's MGX 1.2 specification, published by OCP in October 2026, moves tray blind-mate to larger UQD-08 couplers and liquid-cools a 5,000 A busbar; it is a specification, not a feature of racks shipping today. Settle FRU granularity at procurement, against the rack architecture you are buying.

Serviceability is a design property you inherit from the hardware vendor, partly negotiate at procurement, and pay for every day in operation. The good FRU is hot-swappable, blind-mate, keyed and serial-tracked. An optic slides out and a spare slides in without taking the switch down, with MTTR in minutes. That is the canonical reason Meta kept copper inside the rack: passive DAC’s far better per-link MTBF makes the rare swap trivial, and the link that does fail does not require shutting anything down. The hostile FRU forces you to drain a liquid loop, break a blind-mate manifold, or pull a 1.4 kA busbar connection to reach a failed part. That work is slower and higher-risk. It follows the OEM service procedure, and nobody starts until electrical, pressure and stored energy are verified isolated (→ Chapter 12.1).

Liquid cooling rewrites the serviceability math. Replacing a GPU tray in a direct-to-chip rack is no longer a dry electrical job. It is fluid handling, with dripless quick-disconnects (UQD/UQDB, → Chapter 5.4), leak-detection re-verification and re-priming. That raises both the technician skill floor and the swap MTTR, and it gives your spares strategy a cooling bill of materials (spare QDCs, hoses, gaskets, CDU pumps) that an air-cooled fleet never had. Score exactly this in design-for-serviceability reviews at procurement: how many minutes, how many tools and how much fluid handling each likely FRU swap needs, with that part isolated and the rest of the rack still running.

At a single GW-class campus the spares-and-RMA operation is a small logistics company in its own right, and consumer-electronics intuitions do not apply. The hardware is heavy: a populated GB200/GB300 NVL72 rack weighs about 1.4–1.5 t (3,000–3,300 lb), a Vera Rubin NVL72 about 1.8 t (~4,000 lb), and a compute tray is a two-person lift. It is fragile in transit, since shock and tilt damage to integrated racks is a documented failure mode and ESD discipline is mandatory on boards and optics. It is fluid-wetted on the reverse leg, because a returned liquid-cooled tray must be drained, capped and handled so residual coolant neither damages it nor violates shipping rules. And it is export-controlled: advanced accelerators stay US export-controlled items after their first sale (ECCN 3A090 for the chips, 4A090 for systems that contain them; → Chapter 14.9), so even a warranty RMA across a border needs the right licenses, import-of-record and customs classification, or the box sits in a bonded warehouse for weeks.

The disciplines that make this work are mundane and decisive. Serialized tracking of every FRU, by serial number rather than part number, means a failed unit’s warranty status, RMA history and lemon flag (→ lemon-node detection in Chapter 14.3) are known on scan, and a unit that has bounced through repair twice is retired instead of reinstalled. Tiered stocking puts hot spares on-rack or in-row for the highest-failure-rate FRUs, a site cage for the medium tier, and a regional depot for low-velocity, high-value, long-lead items, which minimizes both stockout risk and total carrying cost. A defined reverse-logistics path with the OEM and a repair partner, including who is import-of-record for cross-border returns, turns the replenishment gate into a known number you can plan a pool depth around, not a surprise.

One operating-model choice sits above all the tactics: run inside the OEM warranty/RMA channel, or around it with a self-spared, self-repaired pool. Inside the channel, the OEM owns the repair, eats the board-level cost and supplies replacements. In exchange you accept its turnaround (weeks, gated by its own allocation and your shipping legs) and you live with the warranty cliff. Self-sparing means pre-buying a deep local pool and standing up triage and possibly board-repair capability. Turnaround collapses to a hot-swap, but you carry the inventory (depreciating, supply-constrained, export-controlled) and the engineering payroll, and you self-insure the repair cost. The largest operators converge on a hybrid. They self-spare the high-failure-rate, fast-swap FRUs to protect MTTR and goodput, and still RMA the expensive board-level units to the OEM, to avoid eating their repair cost and to keep warranty intact where it still applies.

The recurring mistakes all come from sizing spares by inventory cost alone:

• Flat-percentage sparing. Holding “2% of the fleet” over-spares reliable boards and under-spares optics and HBM-heavy GPU classes. Size each failure class against its own AFR and lead time, or you will stock out on exactly the parts that fail most.
• Distant pool to save carrying cost. A regional depot looks efficient on the inventory line, but it quietly reintroduces hours-to-days of MTTR, drags availability down and pushes the synchronous-job overprovisioning tax back up. The carrying cost you saved is dwarfed by the goodput you lost.
• RMA pipeline with no replenishment model. Trusting that a consumed spare “comes back” without modeling the weeks-long RMA latency, then walking into a stockout three failures deep. Pool depth must cover the replenishment gate, not the swap time.
• Ignoring the warranty cliff and export friction. Budgeting spares as if warranty covers the full deployed life, then discovering mid-life that the disposition has shifted to paid repair or harvest, or that a cross-border RMA is stuck in customs. Both are predictable, so plan for them at procurement, not at the first failure.