Accelerator Selection, TCO & Procurement Strategy

The preceding chapters covered the silicon: NVIDIA's annual cadence (Chapter 7.2), AMD's open challenge (Chapter 7.3), the hyperscaler XPUs and custom ASICs (Chapter 7.4, Chapter 7.5), the HBM and packaging constraints that gate all of it (Chapter 7.6, Chapter 7.7), and the software ecosystems that lock in or free the choice (Chapter 7.9). Selection pulls those threads into one decision: which complete accelerator configurations pass, how many you need, who can deliver them, and on what schedule? Here a spec sheet becomes a testable power, memory and delivery obligation.

Each step gates the next. Name the governing metric first, because the wrong denominator silently picks the wrong chip. The eligibility calculation then turns measured per-node useful output into a fleet count and a power requirement. Whether you optimize tokens per dollar or tokens per megawatt depends on a prior question, power-limited or capex-limited, which re-ranks the same shortlist. The heterogeneous fleet is an engineering choice with a carrying cost, and the chapter ends on supply allocation and the RFP that makes cross-vendor bids honest. Chapter 1.8 owns the pricing and economic-life calculation; this chapter supplies the eligible equipment and service inputs it runs on. The per-generation perf/watt and cost-per-token trajectory is in Chapter 16.2; here we use today's snapshot to decide.

Most selection mistakes come from comparing chips on the wrong axis. A GPU that wins on raw FLOPS can lose on delivered tokens. A chip that wins on $/GPU-hour can lose on $/M-tokens, and a part that wins on $/M-tokens at the bench can lose on tokens/MW once the grid limits you. The service you must deliver, plus your simultaneous power, capital and schedule limits, determines the useful denominator, so name it before you shortlist.

Raw FLOPS and HBM capacity are inputs, not metrics. They show what the silicon could do, not what it will do on your workload at your batch size and precision. The realized-MFU gap between paper FLOPS and delivered throughput is specific to the workload and software release, so read any dated comparison (MI300X against H100, say) at its own workload, precision and release. Shortlist on what a chip delivers, not on what its data sheet promises.

$/GPU-hour fits when you buy or sell raw compute supply, such as a neocloud renter buying raw allocation. A training decision instead needs cost and elapsed time to the same validated model quality. $/M-tokens fits when you sell a product to an end user, because tokens are what the customer buys. It is now the governing metric for most operators. Deloitte’s November 2025 forecast puts inference at ~2/3 of AI compute in 2026, which is not the mix at every operator. tokens/MW (tokens per second per megawatt, which is what perf/watt means here) takes over the moment power is what limits you, and in 2026 that is true for a growing share of operators. Utilization and tokens per GPU-second link the three, and a part that wins on one can lose on another. A credible pro-forma states which denominator it optimizes and why. → metric definitions in Chapter 0.3; the financial denominators in Chapter 1.8.

Chapter 1.8 converts confirmed cost and service inputs into the governing economic metric. Selection supplies both sides: the complete installed configuration and the useful output it can deliver within the acceptance limits.

The numerator is more than the chip. Start with the installed configuration. SemiAnalysis’s October 3, 2024 eight-GPU H100 chassis example is $270k before, $256.4k after host/DPU optimization, before rack/cluster fabric, storage and software. The GPU is only part of it: host CPUs, HBM (capacity and bandwidth obligations, see Chapter 7.6), NICs, the NVSwitch/scale-up fabric, the chassis and integration. That node is a historical anchor, not what you buy in 2026. Buyers now take new frontier capacity a 72-GPU rack at a time. GB300 NVL72 is the volume part in 2026. Vera Rubin NVL72 has been shipping since June 2026. AMD’s Helios (72 × MI455X) is in production; AMD (2026-10-12) says first systems are installed at OpenAI, Meta, Microsoft and Oracle. Each rack is one SKU, and the scale-up switch trays, busbar and liquid loop come inside it (Chapter 7.2). For scale, Morgan Stanley’s May 2026 bill-of-materials estimate counts about $4.0M of components in a GB300 NVL72 rack and about $7.8M in a Vera Rubin NVL72 rack, which works out to roughly $55k and $108k of rack cost for each of the 72 GPUs. Those are component costs before integration margin, networking beyond the rack, storage and facility, and NVIDIA publishes no list price, so use them to sanity-check a quote, not to replace one. Integrators quote above the rack too: Supermicro’s September 2026 blueprint for a single Vera Rubin NVL72 scalable unit spans 16 compute racks and 1,152 GPUs, and bundles the networking, storage, power and cooling. Price the numerator at whatever boundary your quote covers, rack or scalable unit, then divide down to $/GPU before comparing offers.

A second, air-cooled tier persists alongside the racks: 8-GPU HGX B300 nodes and RTX PRO-class enterprise servers. It prices like the node example, not like the rack. Then add recurring power and cooling, networking, facility/lease, staff, and software/support cash, and keep server and network acquisition prices with the upfront inputs. Chapter 1.8 separates that cash timing from the book-depreciation schedule. Carry the actual node or rack acquisition price, shared-system costs, paid capacity and net disposal assumption into Chapter 1.8; do not derive another hourly ownership model from the sticker price here.

The denominator is useful output under Chapter 14.1’s accounting boundary, and “useful” carries the weight. For training it is delivered model-FLOPS-utilization on the job, not nameplate FLOPS: a fabric that starves the all-reduce (see Chapter 8.5) collapses the denominator however fast the chip is. For inference it is tokens per GPU-second at your SLO, precision and batch efficiency, which is where FP4/NVFP4 quantization (Chapter 7.10) and serving discipline (Chapter 10.11) move the number several-fold. Carry the earning-life scenarios, the contested 2–3-year bear case versus published 4–6-year useful-life estimates, separately from the 5–6-year book policies into Chapter 1.8. A shorter earning period can reorder the shortlist on matched cash flow; a longer book schedule alone cannot create earning time or disposal receipts.

Chapter 1.6 frames the procurement archetypes, and Chapter 1.8 runs the matched-service NPV that decides among them. Rent when the work ends before the hardware’s earning life does, or when you need capacity before owned racks can arrive. Own when demand is high, sustained and long enough to spread the upfront price. Three contract terms flip the answer most often. The reservation length matters because a take-or-pay term keeps billing every calendar hour after the work stops. The cancellation or exit right decides whether you can stop paying. The provider’s ready-service date tells you whether capacity set up for your workload is there when the job needs it. A posted rate shows none of the three, so price from a dated quotation that names them. Chapter 1.8’s historical H100 rates keep Spheron spot, Lambda configurations and AWS/Azure on-demand observations apart for that reason.

Price both sides on the same service, including host, fabric, storage, facility allocation and operating support. Carry the installed price, utilization, economic earning life and net resale proceeds into the cash flows, and keep book depreciation as a separate expense schedule. A missed delivery date means bridge capacity and fewer owned earning periods. Resale proceeds are not automatic: H100-class and later accelerators stay export-controlled after their first sale, so reselling one means screening the buyer and destination, and sometimes obtaining a license (Chapter 14.9). Cash-breakeven utilization answers a different question, whether the commitment can service its debt, which Chapter 2.5 tests; it is not the own-versus-rent crossover.

Composition is the next choice: a single-vendor fleet, or a mix of GPUs, hyperscaler XPUs and custom ASICs. The trade is software velocity against cost-per-token leverage, and the second stack has a cost you can quantify.

Single-vendor (in practice, NVIDIA + CUDA) buys software velocity, the broadest library and kernel coverage, and operational simplicity: one stack to staff, one toolchain to validate, one supplier relationship to manage. The cost is allocation pain and price exposure, since you inherit the supply queue and pay the merchant premium. Heterogeneous fleets chase cost-per-token leverage and supply resilience. The case for a second stack is real: a matched AMD quote and a measured inference run on your model can beat an NVIDIA offer once acquisition and support terms are compared on the same boundary. Custom ASICs are gaining share where inference is stable and high-volume. TrendForce’s April 2026 revision puts ASIC-based AI servers at about 27% of 2026 AI-server shipments. That is a server-shipment forecast, not a fleet workload allocation.

A mixed fleet no longer has to mean several vendors. NVIDIA’s Groq 3 LPX rack holds 256 SRAM-based LPUs built on licensed Groq technology, with full production announced in August 2026. It plugs into Vera Rubin NVL72 as a low-latency decode tier, on the same vendor’s platform. It gives you decode-specialized silicon without a second supplier, though still with a second toolchain. LPUs do not run CUDA natively, so you test and tune the decode path separately. LPX relies on on-chip SRAM and DDR5 instead of HBM, so it does not compete for HBM supply. NVIDIA still controls allocation, price and support, so LPX is not a second-source hedge.

Match silicon to sub-workload, and price the carrying cost of the second stack honestly. Custom ASICs and TPUs can win on proven, high-volume workloads where the architecture is frozen and cost-per-token dominates, but you give up portability and inherit a provider-controlled toolchain (XLA/JAX or Neuron, see Chapter 7.9). TPUs are mostly rented through Google Cloud; since Q2 2026 Google also delivers TPU systems to select customers’ own data centers. Most of that contracted revenue lands in 2027, Google says (Chapter 7.4 lists the access routes). A customer that takes delivery can price the TPU as an owned asset in the buy/rent/build table, with the delivery schedule and acceptance bar of a GPU order. Leaving the platform later still means a full software re-port. Merchant GPUs can reward changing workloads: frontier training, research, anything that needs CUDA’s velocity and kernel coverage, because reprogrammable flexibility is worth the premium while the model architecture is still moving. The carrying cost of heterogeneity is a second software stack: engineers, validation, CI, kernel ports, and the realized-MFU gap on the less-supported part. Carry it only if the cost-per-token saving on the workload you actually run exceeds that engineering tax. → the GPU-vs-ASIC framing in Chapter 7.5; lock-in quantification in Chapter 7.9.

Selection assumes you can get the silicon. In 2026 you frequently cannot, at least not on your timeline, and a procurement strategy that ignores allocation is guesswork. The binding constraint upstream of every accelerator is HBM and advanced packaging. Suppliers that disclosed their volumes had committed their 2026 HBM output before the year began. By September 2026 Micron reported agreements covering the vast majority of its 2027 HBM bit supply, at significantly higher prices year over year. HBM4 reached volume in H2 2026, and CoWoS-class packaging is a binding constraint on AI compute (see Chapter 7.6, Chapter 7.7). Your purchase order does not set the arrival date; the supplier's HBM and CoWoS allocation does. For a 2027 delivery, ask how the vendor’s contracted memory price flows into your unit price, and whether that pass-through is capped. Optics follow the same rule. On its September 2026 earnings call Broadcom said supply of EML and CW lasers, the light sources in single-mode optics, was well short of demand. Get a dated module or laser allocation into the purchase order, and confirm that any alternate laser is on the switch supplier’s approved list before you count it as a second source (Chapter 8.10).

Three consequences follow for procurement. First, lead time is a selection criterion, not a footnote: a part that is 15% cheaper per token but arrives two generations into your ramp may strand a power slot you fought years to energize (see Chapter 3.2). Second, allocation favors anchor commitments. Take-or-pay volume, multi-generation agreements and, increasingly, vendor-equity entanglements move you up the queue, which is why allocation strategy cannot be separated from the financing structure (Chapter 2.5) and the end-to-end equipment supply chain (Chapter 2.3). Third, a second-source qualification is a hedge with option value: carrying a qualified AMD or ASIC path is not only a cost-per-token play, it is insurance against a single-supplier allocation squeeze. The heterogeneous fleet and the allocation hedge are the same decision seen from two angles.

The RFP makes or breaks accelerator selection, because it is the only instrument that makes heterogeneous bids comparable. The common, expensive failure: vendors bid on the spec sheet, you compare paper FLOPS, and the realized-MFU gap turns the cheapest bid into the most expensive cluster. A defensible RFP fixes the comparison on delivered goodput, not nameplate performance.

• Specify the metric, not the part. State the workload, the precision and the SLO (TTFT/TPOT for inference; job-completion-time for training), and demand bids in your governing denominator, $/M-tokens at the SLO or $/effective-PFLOP-hour at delivered MFU, so a cheap, slow part cannot win on sticker price.
• Mandate a benchmark on your workload. Require a measured run: a representative model, your batch size, your sequence length. A vendor’s number on a favorable case does not count. Use public results to frame the request. MLPerf Inference v6.1, published in September 2026, introduced an end-to-end RAG test, and its biggest entry, at 512 accelerators, was the largest system the benchmark has ever received. Vera Rubin NVL72 appeared in the same round under preview, the category for systems not yet available. A preview result does not compare with an available one, so rank only results within one availability category. MLPerf and SemiAnalysis InferenceX together show which scenario and quality threshold the bid should match. Neither one runs your own graph, so the bid has to require that measurement. Post-training bids should also cover the CPU, memory and storage that rollout sandboxes consume. MLCommons’ Training v6.1 adds an RL post-training benchmark, with submissions from October 2026, that places the rollout sandboxes on the same GPU nodes as the model, so the two contend for CPU, memory and storage. It has no published results yet.
• Score on tokens/MW when power-limited. If your interconnection is capped, a bid that wins on $/M-tokens but loses on perf/watt cannot be energized at scale. Require a tokens/MW line and weight it to the constraint that binds you.
• Bind the delivery schedule. Demand a contractually committed, HBM/CoWoS-allocated delivery curve with remedies for slip. Without that commitment, your ramp date rests on the supplier’s HBM and CoWoS allocation, not on your contract.
• Set a goodput acceptance bar. Tie acceptance and payment milestones to a sustained, measured goodput threshold (see Chapter 7.14), not to power-on, so a cluster that benches well but cannot hold that goodput through burn-in is the vendor’s problem, not yours.

Selection resolves into an ordered procedure, and each step gates the next. (1) Name the denominator: $/GPU-hr, $/M-tokens or tokens/MW, from what you sell and what limits you. (2) Determine the regime, power-limited or capex-limited, because it picks between tokens per dollar and tokens per megawatt. (3) Screen for eligibility: memory fit, measured useful output, power and a confirmed delivery date, as the worked example below does. (4) Price buy, rent or build in Chapter 1.8 with confirmed prices and the economic earning life, not the book life: a shorter earning life can re-rank the shortlist, a longer book schedule cannot. The answer is a crossover that shifts as price, utilization and earning life move, not a fixed policy. (5) Compose the fleet: single-vendor for velocity, heterogeneous for cost-per-token and supply resilience, pricing the second stack’s carrying cost against the workload’s stability. (6) Execute the RFP on realized goodput, with a bound, allocated delivery schedule and a goodput acceptance bar. The output is not a chip. It is a purchase commitment, a power budget and a depreciation schedule that the rest of Part 7 and Part 1 have to live with.

Divide the required output by per-node output, round up to whole nodes, multiply by node power, then check memory, power and arrival. A needs 48,000 / 1,000 = 48 nodes and 960 kW, but 160 GB cannot hold the 180 GB resident set. C needs 48,000 / 700 = 68.6, so 69 nodes, and 69 × 15 = 1,035 kW, which exceeds the 1,000 kW budget. B needs 48,000 / 800 = 60 nodes and 60 × 16 = 960 kW; it holds the resident set and leaves 40 kW of input margin. B arrives at its latest predecessor plus transit, max(12, 14, 18) + 2 = week 20, exactly on the deadline. A one-week slip in B’s system release moves arrival to week 21 and removes the only eligible offer before any financial comparison starts. A budget of 1,035 kW would let C pass on power, but without a confirmed delivery date it still cannot be awarded.

Send B’s node count, installed and running power, operating window, release and delivery milestones and support obligations to Chapter 1.8, with quotations for acquisition, facility enabling works, staff, energy, migration and residual value. The screen decides eligibility, not the award: buy only after acceptance test data and signed delivery commitments replace the hypothetical performance and dates.