Software, Orchestration & Efficiency at the Frontier

Every other part of this guide buys capacity with concrete, copper and silicon, assets that depreciate from the day they are energized. This chapter covers the one layer that moves the other way. Software can improve an installed AI factory between hardware generations, and the rate is not marginal. The compute needed to reach a fixed level of model quality has been halving roughly every eight months in pre-training, about three times faster than Moore's Law (Epoch AI, Mar 2024). Combined with cheaper hardware, that cut the cost of serving a GPT-3-quality token by something on the order of 10x a year from 2021 through 2024 (a16z, November 2024). Buyers see far less relief, because they keep moving to newer models. Across the businesses Ramp observes, its effective token-price index fell 41% from March to September 2026 (Chapter 1.3). An operator who scopes a facility against today's cost-per-token, and assumes it holds, is modeling a world that will not exist by the time the slab cures.

The rest of the chapter follows the layer that turns watts into tokens. It covers the Jevons-rebound scenario, in which sufficiently elastic demand turns efficiency into more consumption; the reasoning / test-time-compute shift that moves the fleet from prefill-heavy to decode-heavy; utilization as the hidden ROI lever, and the orchestration plane that moves it; power-aware orchestration, once the substation rather than the chip is the binding constraint; and the portability vs CUDA-moat fork that decides whether your software stack is an asset or a leash.

The biggest uncertainty in AI infrastructure planning is whether cheaper compute reduces aggregate spend or unlocks enough demand to raise it. The Jevons rebound is the upside-consumption scenario: when demand is sufficiently elastic, efficiency gains increase total use even as unit cost falls. It runs on three compounding curves at once. Algorithmic efficiency: the compute to hit a fixed capability halves about every eight months (95% CI 5-14 months; Epoch AI, Mar 2024), which is the same as effective compute doubling on that clock. Inference cost at fixed quality has fallen roughly an order of magnitude per year (a16z, 2021-2024 series, November 2024; Epoch's threshold-by-threshold price series runs ~9x to ~900x/yr). Hardware efficiency: each accelerator generation roughly doubles useful FLOPS/watt, and each precision step-down (BF16 -> FP8 -> NVFP4) roughly doubles tensor-core throughput again. Price: the curves combine into the ~1,000x three-year drop in the cost of a GPT-3-quality token, from ~$60/M to ~$0.06/M (a16z, November 2024).

If you scope the facility against falling cost-per-token, you must scope demand against the elasticity that the falling cost unlocks. When a capability gets 10x cheaper, the set of economically viable applications expands. Aggregate token spend rises only if usage grows by more than 10x, stays flat if usage grows by exactly 10x, and falls if usage grows by less. DeepSeek’s early-2025 release cut inference cost sharply, yet the largest US cloud and AI companies did not cut their 2025 capex plans, and the ones announced in the following weeks ran above 2024 spending. That fits a rebound story but does not prove one, since the budgets were already in motion. Next year's algorithmic win lets you serve more tokens per megawatt. Whether that reduces required capacity or expands consumption is the scenario fork developed in Chapter 16.5. → demand framing in Chapter 1.3; macro curve in Chapter 16.1.

If algorithmic efficiency is the deflationary force, test-time compute is the workload-dependent inflationary counterforce: hard queries can consume far more inference compute even as efficiency lowers the cost of each pass. The 2024-2025 shift was the discovery that you can buy capability at inference time by letting a model think longer: extended chain-of-thought, sampling and self-consistency, tree search, tool-use loops. The infrastructure consequence is a change in the shape of the workload. For exceptionally complex tasks, NVIDIA’s February 12, 2025 example says long thinking can require over 100× the compute of a single inference pass. Agentic workflows compound it, chaining many such calls per user action. Deloitte's November 2025 outlook forecasts inference at about two-thirds of AI compute in 2026, up from roughly one-half in 2025 and one-third in 2023. It expects almost all 2026 AI computing to run in giant AI data centers or on high-end enterprise servers. That is a compute-share forecast, not a measured power draw (Deloitte TMT Predictions 2026).

Long-output reasoning can leave the fleet that serves it decode-bound. Long inputs can instead make prefill the gate (an agentic loop sends its whole, growing context again with each call), and so can batching or the choice of model. Prefill (processing the prompt) is compute-bound and parallel. Decode (generating tokens one at a time) is memory-bandwidth-bound and serial, and it is the phase a reasoning model spends almost all of its time in. A decode-dominant fleet is sized differently. It values HBM bandwidth and capacity over raw FLOPS, lives or dies on KV-cache management, and benefits from prefill/decode disaggregation, where the two phases run on separately optimized pools connected by a fast KV-cache transport rather than sharing one engine. Scope a fleet for short, prefill-heavy completions and the reasoning era will find you HBM-starved and KV-thrashed. → inference economics in Chapter 1.3; serving engineering in Chapter 10.11.

Cost-per-token has a numerator (the cost stack) and a denominator (tokens actually produced), and most decks never show the number that dominates the denominator: utilization. MegaScale (ByteDance and Peking University) measured 55.2% model-FLOPs utilization for a named 175B-parameter training run on 12,288 GPUs. Utilization claims must name the metric, workload, window and denominator. Lower useful output spreads the same fixed acquisition and staffing cash across fewer paid tokens; Chapter 1.8 prices that effect and Chapter 2.5 tests the resulting debt coverage. Moving utilization ten points can spread fixed cost across substantially more useful output, but the return must still be compared with the actual cooling, fabric and siting alternatives. → the breakeven math in Chapter 1.8.

Utilization is won or lost in the orchestration plane, the scheduler that decides which job runs on which GPUs, when, and at what fraction. The industry is converging here from two lineages. Slurm and its descendants come from HPC: batch-oriented, gang-scheduling (all-or-nothing allocation of a tightly coupled job) and topology-aware down to the NVLink domain. Slurm is still the dominant scheduler in large training fleets. Kubernetes comes from cloud-native serving: declarative, elastic, ideal for bursty inference and multi-tenancy, but historically weak on gang semantics and topology. AI-native schedulers (KAI, Run:ai, Volcano) are closing those gaps by adding gang scheduling, fair-share quota, bin-packing, fractional-GPU sharing and topology awareness on top. The 2026 reality is convergence: Slurm-on-Kubernetes bridges and operators let one cluster run batch training and elastic inference under a single control plane. → the scheduling plane in Chapter 10.1; topology-aware allocation in Chapter 10.2.

The scheduler is necessary but not sufficient. Real utilization is throttled by badput, the fraction of GPU-hours that produce no useful work because a job failed, restarted, drained or stalled. Its counterpart is goodput: useful work delivered within the job's quality and service limits or, as a time fraction, the share of wall-clock time that actually advances the job. Goodput is not MFU, which measures how much of peak FLOPS the work uses. In Google's framing, effective ML productivity is goodput, and an interruption that forces a synchronous job to restart loses the work since its last checkpoint; local recovery costs less. ClusterMAX provider tiers and marketed examples can motivate a 90%-versus-96% sensitivity, but they establish neither an industry average nor a portable best-in-class floor. Put your own fleet's measured productive time and badput categories into TCO. Orchestration and reliability are therefore one problem: lifting utilization means raising goodput, which means fast checkpointing, hot spares, autonomous fault recovery and topology-aware re-scheduling around failed NVSwitch trays. → goodput vs availability in Chapter 12.2; checkpointing in Chapter 9.4; fleet recovery in Chapter 10.7.

Once the binding constraint moves from chips to megawatts, the defining condition of the 2026 era, the orchestration plane inherits a second job it never had in HPC: scheduling against the electrical limit, not just the silicon. You can run the cluster capacity-bound, provisioning power to the rated peak of every accelerator and leaving headroom unused. Or you can run it power-bound, deliberately over-subscribing the electrical capacity and using software to keep aggregate draw inside the substation's limit. The second path is where tokens-per-megawatt, the metric that governs once power is the scarce input, is won.

The workload determines how much headroom there is to harvest. Synchronous training produces enormous correlated power swings: thousands of GPUs hit a collective at the same instant, so the fleet's instantaneous peak is close to the sum of its parts. POLCA measured ~3% unused power headroom in its studied training fleet, and that statistic is not the margin before your protection trips. Inference is the opposite. Request arrivals are uncorrelated and per-server peaks rarely coincide, and POLCA measured ~21% headroom in its studied inference fleet. Microsoft’s POLCA separately simulated roughly 30% more provisioned servers under its power controls and workloads (Patel et al., POLCA, ASPLOS 2024; arXiv 2308.12908). So an inference fleet is a strong candidate for power over-subscription and a training fleet is not. Mis-applying one regime to the other either strands megawatts (capacity-binding inference) or trips the cluster (over-subscribing training).

These swings matter for reliability too, but the two big grid load-loss events came from a different mechanism than workload ramps. In July 2024 a six-fault, 82-second sequence on a Virginia 230 kV line took about 1.5 GW of data-center demand off the grid as sites transferred to backup. Ashburn on 2026-07-22 was larger: one normally cleared fault, and about 3.8 GW left the grid the same way. The 2024 event was one of the drivers of NERC’s May 2026 Level 3 alert. Ride-through is now a binding rule in parts of the US. In ERCOT, non-exempt new Large Computational Loads of 75 MW or more must, since 2026-08-01, stay connected through specified voltage sags and then restore demand inside set time limits (NOGRR 282). SPP’s rules cover new high-impact large loads requested from 2026-01-15. PJM has only proposed rules (2026-09-08). NERC's computational-load standards passed final ballot in October 2026 but still await Board adoption and FERC approval. Neither those standards nor FERC's RD26-7-000 order specifies a ride-through curve (regulatory status: Chapter 4.10). Workload-induced ramps are their own mechanism: test voltage ride-through and backup transfer as separate behaviors. Power-aware orchestration also means smoothing: stagger collective phases, ramp job starts and hold floor load with synthetic work, so the facility looks grid-friendly. The orchestration plane thus becomes a grid-services participant: it can shed, cap or shift flexible (batch, training-checkpoint) load to monetize demand response and unlock interconnection headroom. → grid impact and the loss-event problem in Chapter 15.8; the power-bound thesis in Chapter 16.1; transient absorption and ride-through engineering in Chapter 4.5 and Chapter 4.10.

The efficiency story has an uncomfortable corollary: most of it is realized through one vendor's software stack. CUDA is not a compatibility layer, it is a moat: nearly twenty years of kernels, libraries (cuDNN, CUTLASS, NCCL) and a runtime that the whole training-and-inference toolchain (PyTorch, vLLM, TensorRT-LLM) is tuned against first. The moat shows up as a realized-performance gap, not a paper-FLOPS gap, and the gap moves with the software date. By mid-2026, AMD's MI355X on ROCm matched or undercut B200's cost per token on some models and trailed badly on others. On DeepSeek V4, MI355X's per-user token rate rose more than 100x in the model's first month (SemiAnalysis, May–October 2026; Chapter 7.9). Compare at matched model, precision, batch and concurrency, latency target, software revision and cost basis before pricing the gap. Where the shortfall appears, it is the lock-in tax made measurable, not a portable discount across inference. ROCm is closing it, and Google's XLA/TPU and AWS Neuron/Trainium stacks are mature within their own ecosystems. But each non-NVIDIA path trades any quoted hardware-cost saving for porting work on its own stack (ROCm, XLA or Neuron) and a portability ceiling, and a matched complete-system quote and useful-output test must earn back that burden.

Portability is an option you buy at the cost of velocity. Standardize on a vendor-neutral runtime (an abstraction layer, open serving stacks like vLLM that target multiple backends, or framework-level portability) and you preserve the ability to dual-source accelerators, arbitrage price across vendors and survive a supply-allocation shock. But you must measure any performance loss against vendor-native kernels at the same model quality, context, concurrency and SLO, and you carry the maintenance cost of a second toolchain. Lock to the moat and you get maximum velocity and the deepest library support, at the price of pricing power surrendered to a single supplier and a fleet that cannot migrate. Vendor concentration now reaches the scheduler too: NVIDIA has owned SchedMD, Slurm's developer, since December 2025 and has committed to keeping Slurm open source and vendor-neutral (Chapter 10.1). That is NVIDIA's own pledge, with no independent body to enforce it, so a fleet that plans to dual-source accelerators should confirm with each Slurm release that the features it relies on still work on its non-NVIDIA hardware. For most operators the rational posture is portability at the serving layer (where the workload is commoditizing fastest) and native at the frontier-training layer (where a measured MFU gain may earn back the lock-in cost). → software ecosystems and lock-in in Chapter 7.9; the node software stack in Chapter 10.4.

The final software layer closes the loop between tokens and watts: energy and carbon FinOps, the practice of attributing energy, cost and emissions down to the job, the model and the tenant, then scheduling against those signals. The instrumentation matters because the obvious facility metric is going stale. PUE is increasingly inadequate for liquid-cooled AI. As cooling overhead shrinks (some DLC design claims sit near 1.05-1.15 under the boundary each claim defines), nearly all the energy is IT load, so a flat PUE hides the question that now matters: how much useful work per joule of IT energy. The field is shifting toward work-based and total-energy metrics (TUE, tokens-per-kWh, carbon-per-token) because the efficiency frontier has moved inside the IT load, where PUE cannot see. → the post-PUE metric stack in Chapter 15.1.

FinOps-grade metering turns three otherwise invisible levers. Carbon-aware scheduling shifts flexible batch and training-checkpoint load to hours and regions with cleaner grid mix. That may improve the 24/7 carbon-free-energy score if eligible attributes cover the destination hours, though recovery energy and the actual inventory boundary still count. Cost-aware admission prices GPU-hours by real-time energy cost and lets the scheduler defer low-priority work off-peak, the same flexibility that monetizes demand response. Tenant attribution charges tokens at their true energy-and-carbon cost, so the application layer sees the signal and optimizes its own prompts and model choices. Flexibility is only monetizable if it is metered. An operator may shift or cap a job with coarse telemetry, but cannot substantiate its attributed energy bill or carbon benefit without a reconciled allocation. Grid-service settlement follows the tariff’s meter boundary. → carbon and 24/7 CFE in Chapter 15.3; grid services in Chapter 15.8.