Reference Designs & Worked Examples

This appendix is the reusable arithmetic behind Part 1's archetype framework (Chapter 1.1) and requirements matrix (Chapter 1.7). It does four things, in order. (1) A per-archetype design-basis sheet freezes the inputs every later number inherits. (2) A scalable-unit (SU) budget gives the power, cooling, water and network draw of one atomic deployment block for each accelerator generation. (3) A 50 MW campus is sized up from the SU, with the power chain, cooling plant and water loop derived. (4) A 100k-GPU cluster reference BOM counts GPUs, racks, CDUs, switches, optics and storage, and prices them from an August 2025 GB200 estimate, 2026 assumptions and unquoted networking.

The method is multiplier-first. A reference design is a chain of ratios: GPUs per rack, racks per SU, kW per rack, L/min per kW, NICs per node, optics per NIC, GB/s of storage per GPU. Once those are pinned, rack aggregates are multiplications you can audit, and shared plant and fabric tiers need the explicit counts below. Counts in the tables are exact arithmetic from the ratios given. Dated server estimates and assumed allowances need project quotes before budget release. Density, flow and current figures are generation-stamped because they step discontinuously across GB200 → GB300 → Vera Rubin → Rubin Ultra Kyber, so do not interpolate across a generation boundary.

The design-basis sheet is the one page everything downstream inherits. It turns the workload-profile and design-basis artifacts from Chapter 1.1 into concrete numbers. Pick the column that matches your dominant archetype and the rest of the appendix is parameterized for you. The two reference builds in §3–§4 use the frontier-training column unless noted, because it is the most constraining. Inference builds have their own basis: enterprise HGX relaxes density and fabric, while frontier serving runs the same NVL72-class racks and often buys more service redundancy, not less.

The scalable unit is the atomic deployment and costing block: order it, integrate it at the factory (L11/L12), ship it, energize it, repeat. Sizing the SU once and then multiplying is what makes campus and cluster arithmetic tractable. The SU is anchored to the NVIDIA DGX SuperPOD GB200 reference: 8 × NVL72 racks = 576 GPUs per SU, with the full SuperPOD at 16 SUs (128 racks, 9,216 GPUs). The budget below gives one SU’s draw across four generations; later sections count SUs, not racks.

The columns differ because the multipliers step. A GB200 SU is ~1.06 MW of IT. Rubin Ultra Kyber NVL144 has a ~600 kW per-rack roadmap planning point, and Feynman NVL1152 connects eight Kyber racks, with its power unpublished as of 2026-10-05. Check the Kyber rack’s dimensions, mass and service clearances before assuming it fits the same footprint. This is the density-ramp trap from Chapter 1.1 as a budget line: the floor, water and busbar you reserve today must survive it.

OCP published NVIDIA's MGX 1.2 Accelerated Computing Rack and Trays Design Specification as revision 1.2.0, effective 6 October 2026. It is a specification, not shipping hardware, and it allows racks up to 250 kW on a 5,000 A liquid-cooled busbar with blind-mate liquid cooling. It sits between Vera Rubin's 188/228 kW operating profiles and Kyber. Cite revision 1.2.0 in procurement schedules, not the summit announcement. MGX 1.2 sets no new Kyber figure (Chapter 4.7, Chapter 7.14).

Now multiply. The brief is a 50 MW-class IT frontier-training reference campus on HPE GB200 NVL72, built from whole SUs, with the power chain, cooling plant and water loop derived. IT is sized on rack TDP. Busway and the rack breakers are sized to HPE’s ~192 kW EDPp per rack, which comes to about 73.7 MW across 384 racks. That is a distribution rating, not demand at the POI. How much of it reaches the utility depends on how synchronized the load is and on what rack capacitance, BBUs and UPS/BESS absorb (Chapter 4.5). We also reserve floor, water and busbar headroom for a GB300 → Vera Rubin density step, the irreversible substrate from Chapter 1.1.

SU count. Choose 48 SUs as a clean arrangement: three 16-SU SuperPOD-scale halls, or six 8-SU halls. That is 48 × 8 = 384 NVL72 racks and 48 × 576 = 27,648 GPUs. Treating 132 kW per rack as an exact nominal cap, total IT is 50.688 MW of GPU-rack IT, which exceeds a hard 50.000 MW cap by 0.688 MW. If 50.000 MW is a hard cap, use at most 47 SUs: 376 racks, 27,072 GPUs and 49.632 MW of GPU-rack IT. Supporting network, storage and management IT still have to fit. Chapter 4.1 adds up the total load, and Chapter 5.1 carries every residual-air and liquid duty into the plant. Neither rack count proves that total IT fits.

Campus port map. In each rail and pod, leaf ports 1–32 face hosts and 33–64 face spines 1–32. On each spine, ports 1–27 face leaves, 28–32 stay dark, and 33–48 and 49–64 face its two group cores. Each core’s four consecutive 16-port groups face the matching spine in each of the four pods. Every cable end has one address. The count totals 27,648 host + 27,648 leaf–spine + 32,768 spine–core = 88,064 channels, 176,128 module ends and 704,512 active fiber paths. Surveyed lengths determine fiber-kilometers, and electrical lanes are counted separately.

At 108 leaves per rail the campus exceeds Chapter 8.5’s 64-leaf two-tier limit, so the core stays in. At ≤16,384 GPUs a two-tier instance becomes eligible. Release only after Chapter 13.7 and Chapter 13.9 pass the purchased stack and its failure states. A spare chassis repairs an outage; it cannot keep a single-attached rank connected.

The flagship build is a 100,000-GPU GB200 reference training cluster, costed as a bill of materials. Built from the SU, ceil(100,000 ÷ 576) = 174 SUs is the minimum. Choosing eleven 16-SU halls gives 176 SUs = 1,408 NVL72 racks = 101,376 GPUs (≈ 100k). At 132 kW/rack that is ~186 MW of GPU-rack IT and ~223 MW of facility load at an assumed PUE of 1.2. Add the supporting IT separately before a project capacity release. The capacity may sit on one campus or be split across sites; DCI does not mean this illustrative logical cluster runs as one cross-site synchronous job (Chapter 8.8).

The figures use SemiAnalysis’s August 2025 server-only estimate and assumed storage, CDU and plant costs. Use the counts as arithmetic: they balance ports, power and flow, but they do not draw a topology. Use the dollars as an order-of-magnitude frame. The GPU and rack line is a major cost, but its share of the total depends on priced networking and a reconciled facility scope, and a percentage allowance cannot supply the core.

Large-cluster port map. Leaves keep the campus assignment. In each rail, twelve pods use spine ports 1–32 for leaves, and the final pod uses 1–12 with 13–32 dark. Spine ports 33–64 face eight group cores, four ports per core. Each core’s thirteen four-port pod groups occupy ports 1–52, and 53–64 stay dark. This maps 101,376 endpoints to 8,544 switches; all channel and module counts are in the BOM. The map is realizable, but it is not a minimum-switch proof.

The map above is the GB200 case. On an Ethernet fabric, a GB300 NVL72 GPU’s 800G ConnectX-8 runs as two 400GbE planes, and with the same 64-port switches each plane repeats this map: 17,088 switches and 1,236,992 module ends for the same 101,376 GPUs. On InfiniBand XDR the same NIC is one 800G port on a switch of different radix, and higher-radix Ethernet switches also change the tier count, so recount under Chapter 8.5 before pricing a GB300 fabric.

A campus core loss removes half of its group’s upper capacity; a cluster core loss removes one eighth. A leaf loss disconnects 32 ranks. Test these cuts under Chapter 8.5 with actual placement and payload service, because a healthy port balance does not show that a failure is survivable. Do not release the budget until route, recovery, power and quote data are in. At ≤16,384 GPUs, a two-tier map that passes the same tests can remove the core purchase.

Choose the reference population whose counted fabric and surviving workload fit the project, and buy the complete scope. 8.5 changes the topology and 1.8 values it; an incomplete fabric allowance leaves hardware, power and commissioning outside the purchase. Finish the supporting-IT inventory with 7.13 and the counted fabric in 8.5. Then 4.1 sets electrical demand and the source capacity that survives a failure, and 5.11 sets cooling flow and heat rejection with one train out. Release the integrated budget only with the real storage, management and network load, a tested failure state and quotes that do not overlap. Rack nominal MW alone settles none of these.