Accelerator Selection, TCO & Procurement Strategy

By the time you reach this chapter you have met the silicon: NVIDIA's annual cadence (Chapter 7.2), AMD's open challenge (Chapter 7.3), the hyperscaler XPUs and custom ASICs (Chapter 7.4, Chapter 7.5), the HBM and packaging constraints that gate all of it (Chapter 7.6, Chapter 7.7), and the software ecosystems that lock or liberate the choice (Chapter 7.9). This chapter is where those threads are pulled into a single decision: which accelerators do you actually buy, how many, from whom, on what financial basis, and on what schedule? It is the procurement-execution chapter — the place where a spec sheet becomes a purchase order, a power budget, and a depreciation schedule.

This is the procurement-execution chapter, and it works in sequence. It starts by naming the governing metric, because the wrong denominator silently picks the wrong chip. It builds the TCO model that turns a per-GPU price into a cost-per-useful-token. It confronts the master fork — power-limited vs capex-limited — which decides whether you optimize tokens-per-dollar or tokens-per-megawatt. It lays out buy-vs-rent-vs-build as a crossover that moves with utilization and depreciation. It treats the heterogeneous fleet as an engineering choice with a carrying cost. And it closes on supply/allocation reality and the RFP that makes cross-vendor bids honest. The depreciation argument that underwrites all of it lives in Chapter 1.8; here we execute against it. The per-generation perf/watt and cost-per-token trajectory is consolidated in Chapter 16.2; here we use today's snapshot to decide.

The single most common error in accelerator selection is comparing chips on the wrong axis. A GPU that wins on raw FLOPS can lose on delivered tokens; a chip that wins on $/GPU-hour can lose on $/M-tokens; a part that wins on $/M-tokens at the bench can lose on tokens/MW once the grid binds. The denominator you choose is the decision — so name it before you shortlist.

Raw FLOPS and HBM capacity are inputs, not metrics. They tell you what the silicon could do, not what it will do against your workload at your batch size and precision. The realized-MFU gap between paper FLOPS and delivered throughput is large and vendor-specific — a part can carry ~1.5x the paper FLOPS of a rival and still deliver only 37–66% of its throughput on a real workload because the software stack is immature (SemiAnalysis MI300X benchmarks, 2025). Never select on the spec sheet; select on what the spec sheet delivers.

$/GPU-hour is the right metric when you are buying or selling raw compute supply — a neocloud renter, a training tenant. $/M-tokens is the right metric when you are selling a product to an end user, because tokens are what the customer buys; this is now the governing metric for most operators because inference is ~2/3 of AI compute in 2026 (Deloitte TMT Predictions 2026). tokens/MW (equivalently perf/watt) is the right metric the moment power is the binding constraint — and in 2026, for a growing share of operators, it is. The three are linked by utilization and tokens-per-GPU-second, and a part that wins on one can lose on another. The honest pro-forma states which denominator it optimizes and why. → metric definitions in Chapter 0.3; the financial denominators in Chapter 1.8.

The TCO model is the machine that converts a per-GPU sticker price into the governing metric. It has a numerator (all-in cost) and a denominator (useful output), and most arguments about accelerator selection are really arguments about one term in this fraction.

The numerator is more than the chip. Start with the all-in server: an 8-GPU H100 node runs ~$283k–$318k excluding storage (SemiAnalysis AI Neocloud Playbook, 2025), and the GPU is only part of it — host CPUs, HBM (a top-3 BOM line, see Chapter 7.6), NICs, the NVSwitch/scale-up fabric, the chassis, and integration. Then layer the recurring stack: depreciation (the largest single line for an owner), power and cooling, networking amortization, facility/lease, staff, and a software/support envelope. The canonical self-operated build-up lands near ~$0.74/GPU-hr at 2048-GPU scale and 90% utilization, rising to ~$1.03/hr for small clusters (SemiAnalysis, 2025 — a single-source, contested figure) — note the scale penalty, which is itself a selection input.

The denominator is useful output, and 'useful' carries the weight. For training it is delivered model-FLOPS-utilization against the job, not nameplate FLOPS — a fabric that starves the all-reduce (see Chapter 8.5) collapses the denominator no matter how fast the chip. For inference it is tokens-per-GPU-second at your SLO, your precision, and your batch efficiency, which is where FP4/NVFP4 quantization (Chapter 7.10) and serving discipline (Chapter 10.11) move the number several-fold. The depreciation life you assume — the contested 2–3 year economic life vs the 5–6 year book life — is the dominant lever on the numerator and is argued in full in Chapter 1.8; here we simply note that the same hardware shortlist re-ranks if you change that one assumption, so model both.

Chapter 1.6 framed procurement archetypes qualitatively and Chapter 1.8 scored them as an NPV; this is the accelerator-level execution of that fork. The question 'should I own GPUs or rent them?' has no fixed answer — it is a crossover that moves with three variables: utilization, depreciation life, and resale liquidity.

Renting from a neocloud is right when utilization is spiky, the workload is short or experimental, or you need time-to-first-job in days. The 2026 rental ladder spans nearly an order of magnitude for the same H100 — a ~$1.03/hr spot floor, neocloud median ~$2.29–3.50/hr, AWS ~$6.88/hr, Azure ~$12.29/hr (SemiAnalysis H100 Index / AM Compute, 2026) — and that ladder is volatile, with the 1-year contract index up ~40% from October 2025 to March 2026 as supply tightened. Owning wins when you can hold utilization high against your own self-operated cost (~$0.74/GPU-hr at scale), because the spread between that cost and the rental rate is your saved margin — but only if the asset stays full. The crossover sits where your achievable utilization, against your assumed depreciation life, makes owned cost beat rented price. Below your breakeven utilization (~70% for a debt-financed cluster), renting wins even at a headline premium, because the owned asset bleeds against its debt service. And the crossover moves with depreciation: shift from a 5-year to a 3-year economic life and the owned-cost line rises sharply, pushing the crossover toward renting for all but the most durable, highest-utilization workloads.

The next fork is composition: a single-vendor fleet, or a heterogeneous mix of GPUs, hyperscaler XPUs, and custom ASICs. This is not a religious question — it is a trade between software velocity and cost-per-token leverage, and it carries a real, quantifiable carrying cost.

Single-vendor (in practice, NVIDIA + CUDA) buys software velocity, the broadest library and kernel coverage, and operational simplicity — one stack to staff, one toolchain to qualify, one supplier relationship to manage. The cost is allocation pain and price exposure: you inherit the supply queue and pay the merchant premium. Heterogeneous fleets chase cost-per-token leverage and supply resilience. The economics that justify the second stack are real: AMD parts price ~15–30% below comparable NVIDIA SXM and on inference can close the effective-TCO gap to near zero or favorable (SemiAnalysis, 2025–2026); custom ASICs are eating the inference workload precisely on a tokens/$/W basis. The structural pull is strong — ASIC-based AI servers are projected at ~27.8% of shipments in 2026, with ASIC share of the inference segment rising toward ~40% (TrendForce / industry synthesis, 2026), because stable-architecture inference is exactly where fixed-function silicon's efficiency beats general-purpose flexibility.

The discipline is to match silicon to sub-workload and to be honest about the carrying cost of the second stack. Custom ASICs and TPUs win on stable, high-volume inference where the architecture is frozen and cost-per-token dominates — but you surrender portability and inherit a less mature toolchain (XLA/JAX or Neuron, see Chapter 7.9). Merchant GPUs win where the workload is changing — frontier training, research, anything that needs CUDA's velocity and kernel coverage — because reprogrammable flexibility is worth the premium when the model architecture is still moving. The carrying cost of heterogeneity is a second software stack: engineers, qualification, CI, kernel ports, and the realized-MFU gap on the less-supported part. Only carry it if the cost-per-token saving on the workload you actually run exceeds that engineering tax. → the GPU-vs-ASIC framing in Chapter 7.5; lock-in quantification in Chapter 7.9.

Selection assumes you can get the silicon. In 2026 you frequently cannot — not on your timeline — and a procurement strategy that ignores allocation is a wish list. The binding constraint upstream of every accelerator is HBM and advanced packaging: 2026 HBM production sold out, HBM4 reaching volume only in H2 2026, and CoWoS-class packaging is the most-cited binding constraint on AI compute through 2030 (see Chapter 7.6, Chapter 7.7). Your purchase order does not set the arrival date; the supplier's HBM and CoWoS allocation does.

The consequences for procurement execution are concrete. First, lead time is a selection criterion, not a footnote: a part that is 15% cheaper per token but arrives two generations into your ramp may strand a power slot you fought years to energize (see Chapter 3.2). Second, allocation favors anchor commitments — take-or-pay volume, multi-generation agreements, and (increasingly) vendor-equity entanglements move you up the queue, which is why allocation strategy is inseparable from the financing structure (Chapter 2.5) and the end-to-end equipment supply chain (Chapter 2.3). Third, a second-source qualification is a hedge with option value: carrying a qualified AMD or ASIC path is not only a cost-per-token play, it is insurance against a single-supplier allocation squeeze. The heterogeneous fleet and the allocation hedge are the same decision viewed from two angles.

The RFP is where accelerator selection succeeds or fails, because it is the only instrument that makes heterogeneous bids comparable. The failure mode is universal and expensive: vendors bid on the spec sheet, you compare paper FLOPS, and the realized-MFU gap turns the cheapest bid into the most expensive cluster. A defensible RFP forecloses that by fixing the comparison on delivered goodput, not nameplate performance.

• Specify the metric, not the part. State the workload, the precision, the SLO (TTFT/TPOT for inference; job-completion-time for training), and demand bids in your governing denominator — $/M-tokens at the SLO, or $/effective-PFLOP-hour at delivered MFU — so a cheap, slow part cannot win on sticker price.
• Mandate a benchmark on your workload. Require a measured run on a representative model at your batch and sequence length, not a vendor-supplied number on a favorable case. The MI300X-vs-H100 benchmarking literature exists precisely because measured diverges from spec; bake the measurement into the bid.
• Score on tokens/MW when power-limited. If your interconnection is capped, the bid that wins on $/M-tokens but loses on perf/watt cannot be energized at scale — require a tokens/MW line and weight it to the binding constraint.
• Bind the delivery schedule. Demand a contractually-committed, HBM/CoWoS-allocated delivery curve with remedies for slip — because an unbacked lead time is the single most common reason a 'won' procurement misses its ramp.
• Set a goodput acceptance bar. Tie acceptance and payment milestones to a sustained, measured goodput threshold (see Chapter 7.14), not to power-on — so a cluster that benches well but cannot hold MFU through burn-in is the vendor's problem, not yours.

The chapter resolves into an ordered procedure, because each step gates the next. (1) Name the denominator — $/GPU-hr, $/M-tokens, or tokens/MW — from what you sell and what binds you. (2) Determine the regime — power-limited or capex-limited — because it picks between tokens-per-dollar and tokens-per-megawatt. (3) Build the TCO model with an explicit, dual depreciation assumption (economic and book life), since that one term re-ranks the shortlist. (4) Decide buy vs rent vs build at your real utilization and resale liquidity, treating the answer as a crossover that moves, not a fixed policy. (5) Compose the fleet — single-vendor for velocity, heterogeneous for cost-per-token and supply resilience — pricing the carrying cost of the second stack against the workload's stability. (6) Execute the RFP on realized goodput with a bound, allocated delivery schedule. The output is not a chip; it is a purchase commitment, a power budget, and a depreciation schedule that the rest of Part 7 and Part 1 have to live with.