Accelerator Landscape & Taxonomy

Part 7 starts with one question: what kind of accelerator is this? The answer is the architectural family, not the part number. The four familiar family names are shorthand for points on four independent axes, set out in the next section. The labels summarize different bets on how flexible the silicon should be, who writes the compiler, who owns the scale-up fabric and how long you are locked in. The choice propagates the way the workload fork in Chapter 1.1 does. It sets the software stack your team learns, the interconnect you plumb your racks for, the memory supply chain you depend on and the depreciation argument your CFO will fight over.

Compare products on the properties that drive consequences: programmability, who controls the toolchain, and the scale-up domain. Then add the two business models on top. With merchant silicon you buy a chip and pay the designer's margin for a portable stack. With captive silicon you co-design a chip and take on NRE, software, yield, packaging, capacity and lifecycle obligations in place of a merchant purchase price. Captive programs lean on in-house teams or external design partners (Broadcom, Marvell and, per Counterpoint, MediaTek and Alchip), and one fab, TSMC, is the dominant foundry for the leading GPUs and hyperscaler accelerators. The chapter ends on a costly literacy gap in accelerator procurement: reading a datasheet without being lied to by it. Dense versus sparse, peak versus sustained and the choice of precision inflate a headline FLOPS number by 4x before you have run a single token.

The labels below are reference groupings across four independent axes, not four exclusive families: ownership and access, execution architecture, workload specialization and memory semantics / scale-up domain. At a given TSMC node and HBM generation, peak FLOPS alone does not settle any of them.

• Execution architecture and workload specialization. SIMT, systolic and reconfigurable-dataflow designs are execution choices. Training-general, inference-specialized and operator-specific targets are a separate workload axis. A compiler-programmable TPU or XPU can be workload-specialized without being fixed-function.
• Ownership, access, and toolchain control. Merchant purchase, cloud rental and captive internal deployment decide how you obtain the silicon. CUDA, XLA and Neuron decide the software lock-in. A merchant GPU ships with a mature stack you can hire for, while a captive XPU's compiler exists to serve one owner's models and cloud instances.
• Memory semantics and scale-up domain. Load/store addressability, compiler-managed collectives and message passing are different semantics. Domain size is how many accelerators keep those semantics before traffic drops onto a slower scale-out network. NVLink, ICI, NeuronLink and UALink therefore compare on both semantics and reach. Part 8 covers the fabric engineering; here they are taxonomy axes.

These axes order the table below. Read each one separately before accepting the bundle of consequences in the rightmost column.

Each row of the table is a set of inheritances. Choose the systolic TPU and you also choose XLA/JAX. Google rents TPUs mostly through Google Cloud. It announced selective on-premises sales in April 2026, and since Q2 2026 it has delivered TPU systems into select customers' own data centers and recognized the revenue. Anthropic's expanded TPU capacity (announced April 2026, arriving from 2027) and a Blackstone venture (announced May 2026, 500 MW expected in 2027) are separate arrangements, not open purchase channels. None of these routes establishes a resale market.

Choose a hyperscaler XPU and you choose anchor-tenant economics. The part exists because one operator's internal demand justified the tape-out, and your access depends on their spare capacity and their SDK's maturity, not a merchant price list. Choose an inference ASIC and you may win dramatically on cost per token in a narrow serving regime, then lose the moment the dominant model architecture shifts under a hardwired path. The families trade flexibility for efficiency for control, and your scale and software org set the answer, not a FLOPS chart. → the per-family deep dives: NVIDIA in Chapter 7.2, AMD in Chapter 7.3, hyperscaler XPUs and TPU access in Chapter 7.4, custom ASICs in Chapter 7.5.

One major stack sits outside the table because most readers cannot buy it. Huawei's Ascend NPUs run on its own CANN toolchain and scale up over its UnifiedBus. The Atlas 900 A3 SuperPoD, sold on Huawei Cloud as CloudMatrix 384, links up to 384 Ascend 910C chips, and Huawei planned the Atlas 950 SuperPoD for up to 8,192 chips (Huawei Connect, September 2025). At Huawei Connect 2026 in September, Huawei said the Atlas 950 is in commercial use, pulled the Ascend 960DT forward to Q1 2027 and announced the Atlas 960E SuperPoD for Q3 2027: up to 4,096 NPUs joined by near-packaged optics. These are vendor roadmap claims. The design point is the opposite of NVL72: many weaker chips in a large optical scale-up domain, which costs more racks, more optics and more power for the same compute. US Commerce guidance (May 2025) warns that using Ascend chips anywhere can breach US export controls, so outside China it is a reference point for that trade, not a procurement option.

The deepest split in the taxonomy is architectural. It explains the perf/watt gap and the portability gap at once. A GPU is a SIMT (single-instruction, multiple-thread) machine: thousands of cores run the same instruction across different data, with a large register file, programmable caches and integrated Tensor Cores for matrix math. Its virtue is generality, since any kernel its instruction set and software support can run. Its cost is silicon area and power spent on control logic, scheduling and data movement that a more specialized datapath can spend on math.

A TPU is built around a systolic array: a 2D grid of multiply-accumulate units through which data flows rhythmically. Each cell passes partial sums to its neighbor, so operands are reused across the array without re-fetching from memory. For dense matrix multiplication, the dominant op in transformer training and serving, this is extraordinarily efficient because it minimizes the data movement that dominates the energy budget. That is why Google's TPU v7 Ironwood reaches roughly 4,614 FP8 TFLOPS per chip, at a perf/watt that Google markets aggressively against Blackwell (Google Cloud / TrendForce, Nov 2025). The cost is rigidity. Ops without an efficient implementation for the array's dataflow, such as irregular sparsity, dynamic control flow and exotic attention variants, can run poorly or fall back to slower paths. XLA compilation boundaries and Pallas custom kernels decide which paths exist. GPU control and scheduling consume area and energy, while TPU dataflow specialization can make an irregular workload harder to map. Neither tax is a vendor-wide percentage: compile the actual graph, inspect fallbacks and time its dynamic shapes. → numerics and precision in Chapter 7.10.

The business-model layer sits on top of the architectural one and matters as much. Merchant silicon is sold to anyone. NVIDIA and AMD design a chip, TSMC fabs it, and the buyer pays a gross margin (NVIDIA's consolidated gross margin has run above 70%) in exchange for a portable software stack, a hireable skills market, a deep secondary market and someone else's roadmap risk. Captive silicon is designed by the operator who will run it. Google, Amazon, Microsoft, Meta and now OpenAI build accelerators tuned to their own models and clouds, judged on total program economics rather than a merchant purchase price. The prize is enormous at hyperscale. For a million-accelerator fleet the comparison that matters is total delivered cost per useful unit of work, including NRE, software, yield, packaging, capacity, operations and lifecycle risk. A vendor's gross-margin percentage is not buyer savings.

Almost no operator designs the whole chip alone. Physical design, SerDes, packaging and tape-out expertise sits with a few merchant-silicon houses and with in-house teams such as AWS's Annapurna Labs, which designs Trainium and Inferentia. Broadcom is the partner behind Google's TPU line, Meta's MTIA (co-development announced April 2026) and OpenAI's first in-house accelerator. Marvell signed a multi-generation custom-AI supply agreement with AWS in December 2024, though the announcement does not say it covers every Trainium generation. Marvell also signed a Google custom-silicon agreement on 29 July 2026, disclosed 19 August 2026, covering inference accelerators and products that attach to the TPU ecosystem. Counterpoint's January 2026 research places MediaTek in Google's design supply chain and Alchip in AWS's; that is analyst reporting, not vendor confirmation. Chapter 7.5 has the dated design-partner share estimates. The strategic reading: 'building your own silicon' rarely means vertical independence. It swaps a merchant purchase for a program of design partner, foundry, packaging and software, and underneath both, usually the same fab.

The costly literacy gap in accelerator procurement is taking the headline FLOPS number at face value. Vendors quote the largest defensible figure, and the gap between that figure and what your workload sustains can be a factor of four or more before you have run a single token. Three traps recur, and each inflates the number in a different way.

Trap 1: dense vs sparse. The biggest headline numbers usually assume structured sparsity (commonly 2:4, two of every four weights zeroed), which raises the quoted matmul throughput by up to 2x. NVIDIA's prior-generation GB200 NVL72 rack is marketed at ~1.44 ExaFLOPS FP4 with sparsity, and its dense figure is half that. Do not assume the ratio carries over. The GB300 NVL72 datasheet gives 1,440 PFLOPS sparse and 1,080 PFLOPS dense FP4, and Vera Rubin NVL72 gives 3,600 PFLOPS NVFP4 for inference (sparse) against 2,520 PFLOPS for training (dense). Those are vendor peak specifications, not measured speed-ups, and most production workloads do not realize the full sparsity gain. AMD quotes MI355X FP4 at 10.1 PFLOPS without sparsity, so a naive 'their number is smaller' comparison sets a dense figure against a sparse one. Never halve a headline to get dense. Take the dense figure the datasheet prints and read the footnote it carries, at the same precision, before comparing two vendors.

Trap 2: precision inflation. A chip's biggest FLOPS number is at its lowest-precision format. MI355X shows the pattern cleanly: ~2.5 PFLOPS FP16, ~5 PFLOPS FP8 and ~10.1 PFLOPS FP4/FP6, a 4x spread from the precision the marketing slide chose. The step size is product-specific, though. NVIDIA's Vera Rubin page lists, per GPU, 4 PFLOPS dense FP16/BF16, 17.5 PFLOPS FP8/FP6 training and 35 PFLOPS NVFP4 training, so the first step up is 4.375x (a guide calculation from vendor peaks). Read each product's dense spec instead of deriving it from bit width. If your training run needs BF16 or FP8 for stability, the FP4 headline is a number you will never see. Match the quoted precision to the precision your workload actually runs at. → the precision ladder in Chapter 7.10.

Trap 3: peak vs sustained. Peak FLOPS assumes every multiply-accumulate unit is fed every cycle. Real workloads are throttled by memory bandwidth, collective-communication stalls, kernel launch overhead and thermal limits. The honest metric is Model FLOPS Utilization (MFU), sustained useful FLOPS over peak. Meta’s July 2024 Llama 3 Table 4 measures it at 41% BF16 for 405B pre-training on 16,384 H100 GPUs at 8,192-token sequence length. Failures and restarts then take their own cut. Goodput, the share of wall-clock time that advances the job, is a separate fraction from MFU and multiplies into delivered throughput (Chapter 14.1). A part with a higher peak but a worse compiler and a thinner memory pipe can lose on sustained throughput to a part with a lower headline. That is why measured benchmarks diverge from spec sheets, and why a procurement RFP must demand sustained-MFU numbers on your models, not peak FLOPS on the vendor's.

MLPerf is the public cross-check. Compare only results that share a workload, scenario, accuracy target and division, and note the software and hardware versions, the result date and the availability category. MLCommons's Inference v6.1 announcement (16 September 2026) classified the Rubin and Vera Rubin NVL72 submissions as Preview. Available means purchasable or rentable hardware with available software; Preview targets a later Available submission. → realized-MFU gap and switching cost in Chapter 7.9; the qualified configuration from Chapter 7.11 enters the lifecycle cost model in Chapter 1.8.

The last decision the taxonomy forces is fleet composition, and it is a goodput-versus-resilience trade. A single-family fleet, an all-NVIDIA GPU deployment for example, buys operational simplicity: one software stack, one scheduler integration, one set of failure modes, one driver matrix, a shared pool of engineers and one resale channel. The price is total exposure to one vendor's roadmap, allocation queue and pricing power. A heterogeneous fleet uses GPUs for flexible training, a TPU or Trainium pool for steady high-volume work and an inference ASIC for a stable serving tier. It buys price leverage (a credible second source disciplines the incumbent's quote), supply resilience (independently qualified allocations can soften a family-specific shock) and workload-fit efficiency. The price is multiplied operational complexity: every additional family is another compiler to maintain, another set of kernels to port, another realized-MFU gap to measure and another on-call runbook.

One vendor can also supply heterogeneous silicon. NVIDIA's Groq 3 LPX rack (as of 16 October 2026) holds 256 LPUs, 128 GB of SRAM and 12 TB of DDR5, and sits beside Vera Rubin GPU racks. LPU execution keeps its own compiler and memory behavior; shared vendor integration does not make it CUDA-native or remove the second toolchain. That buys workload-fit diversity but none of the price leverage of a second vendor, and you stay exposed to one vendor's pricing and roadmap. Decide which kind of diversity you are buying. Chapter 7.2 covers inference partitioning and Chapter 7.9 software support.

The rule of thumb that survives contact with operators: match fleet diversity to scale and to the size of your software org. Below a few thousand accelerators, a second stack still has to justify its recurring engineering and on-call cost; accelerator count alone does not set the threshold. At hyperscale, heterogeneity can pay, because a single-vendor fleet at million-chip scale carries an existential allocation and pricing exposure, and a funded compiler team can absorb the integration cost. The middle is hard, and it is where most bad decisions get made: a mid-size operator adds a second family for price leverage it is too small to realize, and drowns in operational complexity it was too small to absorb. → switching-cost quantification in Chapter 7.9; technical eligibility and RFP construction in Chapter 7.11; buy/rent economics in Chapter 1.8.