Vocabulary, Mental Models & the Metric Stack

Two parties using one word for different quantities is the recurring failure mode in AI-data-center projects, and it usually surfaces only after a contract is signed or a slab is poured. A developer quotes "100 MW" and means grid draw; the tenant hears critical IT power and sizes a cluster against capacity that the building overhead also needs. A neocloud advertises "99.99% uptime" and means one node's hardware availability; the customer assumed it described the training job's goodput. A vendor cites "1.8 TB/s" without saying it is the scale-up domain, and a network architect budgets the scale-out fabric against it. None of these are lies. They are vocabulary collisions, and each one costs real money downstream.

This chapter fixes the shared language: the power vocabulary (three numbers that get conflated), the capacity units (rack to gigawatt campus), the metric stack (efficiency, useful-work and unit-economics families, and how they trade off), the availability algebra (the nines, their downtime budgets and how components compose) and the three-tier network hierarchy (scale-up, scale-out, scale-across). It is an orientation chapter, not the canonical home for any of them. Each metric's rigorous definition, measurement plan and gotchas live in its discipline chapter, and we point you there. The goal is that when Part 3 says "facility power", Part 8 says "oversubscription" or Part 12 says "goodput", you read the word the way the author meant it.

Start with power, the limiting resource of the 2026 era and the most often mis-stated number in the building. Three distinct power figures run along the chain, and a project that does not keep them apart will mis-size something expensive.

Critical IT power is the power delivered to the compute: the racks, after the last PDU, the number the workload actually consumes. Total facility power adds everything needed to keep that IT alive: cooling plant, pumps, fans, lighting, and losses in the UPS and transformers. The ratio between them is PUE. ISO/IEC 30134-2 defines PUE as total facility energy over IT energy across the same continuous 12-month period; the power-ratio form everyone quotes is a design or interim derivative, not the annual metric. Grid draw (or utility demand) is net import at the meter. Contracted capacity is the MW limit the interconnection agreement sets, a limit rather than a reading. On-site generation or storage separates both from facility load.

Take a peak-load case with 100 MW of critical IT plus 20 MW of coincident non-IT overhead. It needs 120 MW of facility power. That 1.2 is an instantaneous ratio; an annual PUE of 1.2 does not tell you the peak. Size the service from the peak operating cases, and study inrush and transients separately instead of hiding them in an unexplained headroom allowance. Quote the wrong one of these in a lease or a power-purchase term sheet and the error propagates into switchgear ratings, transformer orders and the size of the queue slot you reserve.

Two further distinctions trip people up. MW vs MVA vs kVA: real power (MW) does the work and is what you pay for in energy. Apparent power (MVA/kVA) is the magnitude of complex power, with S² = P² + Q² in sinusoidal steady state. Transformers and generators commonly carry kVA/MVA ratings, while switchgear is normally specified by voltage, continuous current, interrupting/withstand ratings and the applicable standard. At a power factor of 0.9, a 100 MW load is ~111 MVA, so equipment sized in MW to match the load is undersized by the inverse of the power factor. That 0.9 is a textbook figure; at the rack the gap is much smaller, because server power supplies correct their own power factor. 80 PLUS Platinum and Titanium server units must hold at least 0.95, at half load and at 20% load respectively, so a modern AI hall draws close to unity. What still matters upstream is the rating basis: transformers and generators are sized in kVA, the utility sets the band of reactive power the site may draw or supply at its connection point, and rectifier harmonics carry current that does no useful work (Chapter 4.4). The sharp swings of synchronous training are a real-power problem, covered below. → Chapter 4.1.

TDP vs electrical design point peak (EDPp): a GPU's thermal design power (TDP) is the per-chip steady-state limit (~700 W for H100, ~1.0-1.2 kW for B200/GB200 and up to ~1.4 kW for GB300 Blackwell Ultra). EDPp is the higher peak the power chain must carry when the chips surge: HPE rates its GB300 NVL72 rack at about 155 kW EDPp against 132 kW nominal. Size your power chain from equipment ratings and the design-load profile, including transients that can swing tens of percent on synchronous all-reduce boundaries; use measured per-rack and per-cluster draw to calibrate the model. Sum nameplate TDP naively and you will over-provision average capacity and under-provision for the transient. → Chapter 4.5.

The second piece of shared vocabulary is the unit of capacity you scope, budget, order and reason about failures in. AI facilities are built and discussed at five nested scales. Choosing the wrong granularity for a decision is itself a mistake: you procure in scalable units and reason about blast radius in fault domains, and the two do not always align.

The rack is the atomic power-and-cooling unit. An HPE GB200 NVL72 rack has a 132 kW nominal rack TDP and weighs ~1.47 t wet (3,245 lb fully loaded), and the slab, the busway and the cooling manifold are sized against it. Size against the generation you will actually deploy. GB300 NVL72 shipped in 2025 and is deploying through 2026 alongside GB200; Lenovo rates it at 135 kW rack TDP and up to 155 kW peak, while NVIDIA's reference architecture puts a full GB300 NVL72 rack at up to 142 kW and HPE recommends provisioning busways for 192 kW. The VR200 NVL72 has been shipping since June 2026. NVIDIA's DSX reference design plans Vera Rubin NVL72 (VR200) cabinets at 330 kW TDP, its facility basis for power and cooling. OEM datasheets rate operating profiles lower: Pegatron lists 188 kW Max Q / 228 kW Max P. Size power and cooling reserve to 330 kW when you follow the DSX reference, and use 188/228 kW for expected operating load.

The NVLink / scale-up domain is the set of accelerators with load/store addressability over NVLink: 8 in an HGX node, 72 in an NVL72, and 144 per rack in the announced Kyber generation. The announced Rubin Ultra NVL576 system targets a domain of 576 across eight optically stitched 72-GPU MGX racks, a different enlargement from the single ~600 kW Kyber NVL144 rack, whose eight-rack system is NVL1152. The GB200 NVL72 has 13.4 TB of GPU memory across its 72-GPU domain; the GB300 NVL72 raises that to 20 TB. This is the most important unit for the software, because tensor- and expert-parallelism must fit inside it to run at full bandwidth.

The scalable unit (SU) is the repeatable building block of the cluster: a fixed bundle of racks, leaf switches, cooling and power that the design replicates N times to scale. It is the unit you procure and commission, and the unit a reference architecture (DGX SuperPOD and its peers) is specified in. The pod is a deployment block of several SUs sharing a spine layer and often a fault domain. The gigawatt campus is the 2026 frontier unit of strategic conversation: multi-building sites in the 1 GW-plus class that headline hyperscaler announcements and where power, water and grid impact become regional questions. → reference architectures in Chapter 0.4; the SU as a budgeting artifact in Chapter 0.2; fault domains in Chapter 12.1.

There is no single "efficiency" number for an AI factory, and pretending there is leads to vanity metrics. Three families each answer a different question, and a well-run facility reports from all three because optimizing one in isolation degrades the others. The canonical homes differ by family: facility metrics are defined and measured in Chapter 15.1, useful-work metrics (MFU, MBU, goodput, ETTR) in Chapter 14.1 with the reliability reframe in Chapter 12.2, and unit economics in Chapter 1.8. Here we only orient you to what each family measures and where they conflict.

Facility-efficiency metrics ask how much of the power and water going in is overhead rather than compute. PUE (total facility energy / IT energy over the same year) is the incumbent. Uptime Institute’s 2026 survey puts the respondent mean at 1.52 and the capacity-weighted average at 1.36 (survey report July 2026; Uptime’s analysis 6 August 2026). The gap reflects larger facilities running more efficiently, so keep the two weightings apart; Chapter 15.1 defines both survey benchmarks. Best-in-class liquid-cooled designs reach 1.05-1.15. WUE-site measures annual on-site water consumption per kWh of IT energy. LBNL's December 2024 report modeled the US fleet's 2023 average at just over 0.36 L/kWh. Dry, non-evaporative heat rejection can bring cooling-water consumption near zero, while evaporative designs vary with climate and operating conditions. ERF/ERE credit reused heat, REF credits renewable energy, and CUE measures carbon per unit of IT energy. PUE and WUE trade against each other: evaporative cooling buys a better PUE by spending water (worse WUE), so a facility chasing a headline PUE can quietly become a water glutton. → Chapter 15.4.

Useful-work metrics ask the question PUE cannot: of the energy that reached the chips, how much produced actual progress? MFU (model FLOPs utilization) and MBU (model bandwidth utilization) measure how close a workload runs to the hardware's theoretical compute and memory-bandwidth ceilings. Goodput is useful work delivered within the quality and service limits set for the job; as a time fraction, it is the share of wall-clock time that advances the job. It is not MFU, and it is the metric that governs training economics. ETTR (effective training time ratio) and its complement, the time lost to failures and recovery, sit underneath it. Meta's Llama 3 405B run on up to 16,384 H100s was interrupted roughly every three hours and still reached more than 90% effective training time (Meta, July 2024). Before comparing a provider's figure with that, match the metric, job size, window and excluded downtime. A facility can post a beautiful 1.08 PUE and still waste a quarter of its energy on a fabric that starves the all-reduce or a checkpoint cadence that loses hours per failure, and PUE would never show it. Facility efficiency is necessary but never sufficient: that is the GOODPUT thread of the guide. → Chapter 12.2.

Unit-economics metrics ask what the work costs. Tokens-per-joule (and its inverse, joules-per-token) is the rising efficiency metric for inference, because one number tied to the product captures the whole stack: chip, fabric, cooling and software. $/GPU-hr is the supply-side cost: an owned cost built from complete cash inputs, or a rental price set by a dated service contract. $/M-tokens expresses cost or price per unit of work. The illustrative H100 case in Chapter 1.8 gives a node-rental cost of $1.90/M output tokens. On the price side, Ramp's effective token-price index fell 41%, from its March 2026 peak of $1.15 to $0.68 per million tokens on 9 September 2026 (Chapter 1.3). Ramp does not say how the index is weighted or whose prices it samples, so treat it as one firm's tracker rather than a market-wide tariff. These are the numbers the business lives on, and they integrate everything above: a facility with great PUE and poor goodput, or great goodput and a bad power contract, shows up here as a bad $/M-tokens. → Chapter 1.8.

Availability numbers are abused in infrastructure marketing because the algebra behind them is rarely shown. Get the vocabulary straight and you can read any "number of nines" claim for what it promises.

Availability is A = MTBF / (MTBF + MTTR): mean operating time between failures over that time plus mean restoration downtime, in a repairable two-state model. The headline form is "the nines": 99.9% (three nines) is ~8.8 hours of downtime per year, 99.99% (four nines) is ~52 minutes, and 99.999% (five nines) is ~5.3 minutes. You raise availability either by failing less often (higher MTBF) or by recovering faster (lower MTTR). In a cluster where restore and replay dominate lost progress, fast checkpoint-restart and lemon-node detection can buy more goodput per dollar than a component-reliability upgrade. Keep the nines separate from Uptime's Tiers, which certify topology: Tier III requires concurrent maintainability, and Tier IV adds fault tolerance under the standard's defined conditions.

The part that matters for design is composition. Components in series (each a single point of failure for the path) multiply: a path of ten 99.9% components is 0.999^10 = ~99.0%, not 99.9%, because series composition always makes the whole worse than any part. Components in parallel (redundant, where one surviving path suffices) multiply their unavailabilities: two parallel 99% paths give 1 - (0.01 x 0.01) = 99.99%. Both formulas are ideal models that hold only when the elements fail independently, and an N+1 pool is not one-of-two but k-of-n, needing N of the N+1 units, which composes less generously. That is the mathematical case for N+1 and 2N redundancy, and its precondition. A controller or cooling loop that both paths need is a shared failure point: model it in series with the remaining redundant branches. Other common-mode events need joint-failure probabilities. Either can erase the apparent redundancy gain, by an amount that depends on the failure model. → the redundancy vocabulary (N, N+1, 2N, block- and distributed-redundant, catcher topologies) is defined in Chapter 0.5; Chapter 12.5 builds the quantitative RBD, Markov and Monte-Carlo models, dependent failures included, and Chapter 9.4 covers checkpoint recovery.

The last piece of shared vocabulary is the network. Bandwidth, latency and oversubscription mean different physics at each of three tiers, and conflating them is how a fabric gets mis-budgeted by an order of magnitude.

Scale-up is the fabric inside a supported load/store and collective-communication domain (NVLink and its peers), connecting the GPUs that act as one large accelerator. It is the fastest, shortest-reach, highest-cost-per-bit tier, mostly copper because the reach is sub-metre. Tensor- and expert-parallelism live here. NVLink 5 gives each GPU in a GB200 or GB300 NVL72 1.8 TB/s bidirectional, or 900 GB/s each way, and the rack about 130 TB/s in aggregate. NVIDIA's Vera Rubin NVL72 product page (checked October 2026) lists NVLink 6 at 3.0 TB/s bidirectional per GPU, or 1.5 TB/s each way and 216 TB/s per rack.

Scale-out is the back-end fabric between domains, InfiniBand or Ethernet/RoCE, connecting racks and pods into a cluster. Compare it per GPU: 400 Gb/s on GB200's ConnectX-7, 800 Gb/s on GB300's ConnectX-8, and an aggregate 1.6 Tb/s of ConnectX-9 on Vera Rubin NVL72, whose first racks reached CoreWeave by June 2026. Those are 50, 100 and 200 GB/s per direction, so scale-up runs 18x, 9x and 7.5x faster than scale-out: about an order of magnitude, 8-18x by generation, before protocol and workload effects. That gap is why software keeps tight parallelism inside the scale-up domain and pushes data- and pipeline-parallel traffic onto the scale-out fabric. It is an optimization preference, not a taxonomy: expert-parallel routing crosses nodes whenever the experts span domains, and some implementations run tensor-parallel traffic there too, so budget the fabric against the job's actual parallelization and traffic matrix. Oversubscription is a real choice at this tier. Published fabrics span 1:1, 2:1-3:1 and reported 7:1 examples, but the project ratio follows the measured traffic matrix, topology, placement, failure headroom and step-time or tail-latency SLO, not the workload label.

Scale-across is optional DCI among buildings or campuses for portfolio connectivity, replication, burst or overflow capacity, service failover and cross-site computation that has passed testing. It does not require one job to span sites; any cross-site workload must fit the measured latency, bandwidth, algorithm, placement and correlated-failure budget. A claim about "oversubscription" or "latency" is meaningless until you say which tier it describes. → scale-up in Chapter 8.2; scale-out topology and oversubscription in Chapter 8.5; scale-across multi-campus fabric in Chapter 8.8.

Nothing here is meant to be the last word. Each term has a discipline chapter where it is defined rigorously, measured honestly and pushed to its edge cases; this chapter is the dictionary you keep open while reading the rest. When Part 3 reorders the siting hierarchy around power, it assumes you read "grid draw" and "MVA" correctly. When Part 5 chooses a cooling approach, measured kW/rack is one input among several (the liquid/air heat split, water temperature, climate, redundancy and the refresh cycle), not the verdict. When Part 8 chooses an oversubscription ratio, it assumes you know which network tier is on the table. When Part 12 argues that AI factories optimize goodput rather than availability, it assumes you can do the availability algebra well enough to see why that is a real distinction.

Keep availability and accepted-work attainment as separate acceptance conditions: higher throughput cannot make up for a missed recovery or latency limit you contracted for. The three threads of the guide, POWER-BOUND, GOODPUT and DENSITY-RAMP, are each at bottom a discipline about reading one of these metrics honestly across a four-year build against a moving target.