AMD Instinct & the Open Challenger

For most of the modern accelerator era the AMD question answered itself: the hardware was competitive on paper, the software was not, and a frontier operator could not afford to find the difference on a production training run. As of mid-2026 that has shifted. AMD ships the MI355X, which matches NVIDIA's Blackwell Ultra on HBM capacity (288 GB each) and sits above HGX B200 on dense FP4, though it still trails Blackwell Ultra on that one number. Announced commitments back the roadmap: OpenAI (6 GW), Meta (up to 6 GW), Microsoft on Azure, Oracle's public cloud (50,000 MI450 GPUs) and Anthropic (up to 2 GW). AMD is also ramping a 72-GPU open rack, Helios, built on Meta's OCP Open Rack Wide, with initial systems installed at OpenAI, Meta, Microsoft and Oracle (AMD, 2026-10-12). That takes AMD into the one place NVIDIA was uncontested, the rack-scale coherent domain. The silicon argument for a second source is largely won. What remains open is realized performance, software maturity and the timing of an open fabric standard, and that is where the money is made or lost.

The product line runs MI300X, MI325X and MI355X. The MI400 generation splits into the rack-scale MI455X inside Helios, the 8-GPU MI440X server and the HPC-oriented MI430X, and MI500 is roadmapped for 2027. The scale-up bet rides UALink-over-Ethernet. Two questions decide whether any of it is worth buying: where AMD wins today, and what the ROCm-maturity tax costs you where it does not. Paper FLOPS and HBM datasheets are necessary but not sufficient. Goodput, meaning useful output per unit of wall time that meets your acceptance target after restarts and recovery losses, decides whether the hardware discount survives the porting bill (Chapter 7.9 has the definitions).

AMD's strategy reads off its product cadence: lead with the axis NVIDIA was weakest on, memory capacity per GPU. Capacity decides how many accelerators a large model must be sharded across, so it drives inference cost per token more directly than any other spec. The MI300X (CDNA 3, launched December 2023) shipped with 192 GB of HBM3 at 5.3 TB/s and a 750 W TDP, against the H100's 80 GB. With 2.4x the capacity, a 70B-class model fit on one GPU at some precisions and batch sizes. Where it fits, the win is in topology, since a replica on fewer GPUs can carry less collective traffic and span fewer failure domains. The extra HBM goes to weights, KV cache and overhead; it does not shrink any one GPU's share of the cache.

The MI325X (still CDNA 3, Q4 2024) widened the lead to 256 GB of HBM3E at 6 TB/s and 1,000 W, a mid-cycle capacity bump aimed at NVIDIA's H200 (141 GB). The MI355X (CDNA 4, TSMC N3P, mid-2025) closed most of the compute gap: 288 GB of HBM3E at 8 TB/s, native FP6 and FP4 datatypes, ~10.1 PFLOPS of dense MXFP4 and ~5 PFLOPS of dense FP8, at a 1,400 W liquid-cooled board power. Against Blackwell Ultra, the platform 2026 volume ships on, capacity is level: B300 and GB300 also carry 288 GB HBM3E. The published FP4 peaks are not level. NVIDIA rates GB300 NVL72 at 15 PFLOPS NVFP4 and 5 PFLOPS FP8 per GPU, dense. HGX B300 is rated at 13.5 and 4.5, and HGX B200 at 9 on dense FP4. So GB300's dense FP4 peak is about 1.5x the MI355X's and HGX B300's about 1.34x, while the MI355X is level with GB300 on dense FP8. These are vendor-published peaks in different formats (MXFP4 against NVFP4). The memory lead reopens at the current generation, MI455X against Vera Rubin: 432 GB of HBM4 at 23.3 TB/s against 288 GB at 19.2 TB/s, NVIDIA's shipping figure. AMD's own comparison page sets MI455X against the 22 TB/s NVIDIA announced at Rubin's launch. Native FP4/FP6 is what makes that memory pay off on modern inference, and the quantization tradeoffs are in Chapter 7.10. Chapter 7.6 has the capacity arithmetic.

The fourth row changes the strategic picture. Single-GPU and single-node parts let AMD compete box-for-box, but from the GB200 generation onward NVIDIA's moat was the rack: a 72-GPU NVLink domain across which software distributes work, which no competitor could match without its own scale-up fabric. The rack-scale MI455X inside Helios is AMD's answer; the anchor agreements below call these MI400-generation GPUs for Helios the "MI450 Series". Each MI455X carries 432 GB of HBM4 in 12 stacks at 23.3 TB/s, the figure on AMD's MI455X page and in its Hot Chips talk (2026-08-24). AMD's 2025 Helios preview, and one tray description still on the Helios page, say 19.6 TB/s, so take the contracted value from your OEM's dated schedule and acceptance results. At rack scale, the published Helios design joins 72 of them into a single scale-up domain with ~260 TB/s of aggregate scale-up bandwidth, ~31 TB of HBM4, ~1.4 EFLOPS FP8 and ~2.9 EFLOPS FP4. On the headline rack numbers that is NVL72-class, the Rubin rack NVIDIA briefly marketed as NVL144. Contract the implemented fabric, software and delivery date with the integrator, then make it demonstrate rack-scale performance and recovery.

NVIDIA's NVLink is a proprietary, production-proven scale-up switch fabric built for the job. AMD cannot ship that, so it is betting on the open alternative, UALink: a consortium standard (AMD, Intel, Google, Meta, Microsoft, Broadcom and others) finalized as UALink Common 2.0 and 200G Data Link and Physical Layer 2.0, with 200 Gb/s per lane, load/store memory semantics, sub-1-microsecond round trips and up to 1,024 accelerators in a domain. On paper UALink is a genuine peer to NVLink; the constraint is timing. UALink's published catalogue lists Common 2.0 and 200G Data Link and Physical Layer 2.0 separately. AMD specifies UALoE for MI455X, so require the actual switch SKU, enabled lanes, firmware and supported rack topology in the Helios offer.

Tunneling over Ethernet buys AMD a 72-GPU scale-up domain on schedule, riding the mature, multi-vendor Ethernet PHY and switch ecosystem instead of waiting on a brand-new switch ASIC. The cost is a memory-semantic scale-up fabric running over a substrate never designed for it, with a software and reliability stack younger than NVLink's by several production generations. What NVLink sells is not the bandwidth number. It sells failure modes, firmware, collective libraries and diagnostics beaten into shape across two or three large deployments, and an open fabric inherits none of that for free. Chapter 8.2 has the full scale-up comparison: NVLink vs UALink vs UALoE, domain sizing, and the copper and optics reach implications. The narrower point here: committing a training campus's scale-up fabric to UALoE requires a rack-scale collective, congestion and fault-recovery test on the delivered implementation.

The answer is workload-specific, and it splits along the training/inference line that governs everything else in Chapter 7.11. AMD's structural advantage is memory-capacity-bound inference. When the constraint is fitting a large model and its KV cache into HBM at a given concurrency, capacity per GPU is the lever, and the lead depends on the matched generation: MI355X and B300 both specify 288 GB, while MI455X carries 432 GB against Rubin's 288 GB. More HBM per GPU can cut the accelerator or node count, but only a test of the complete serving profile shows whether it does for yours. Where the model does fit, larger HBM lets you run bigger batches before you spill, which raises throughput per GPU. SemiAnalysis's May 2025 inference benchmarks showed that texture for MI300X. On Llama 3 405B, MI300X beat H100 on both absolute performance and performance per dollar at the tested operating points, and at ultra-low latency on Llama 3 70B it led on performance per dollar. SemiAnalysis did not test H100 on DeepSeek V3 (671B). In that dated test the memory advantage and the system prices both favored MI300X.

The mirror image is where those dated AMD results did not win: latency-sensitive, throughput-tuned inference against the H200, and tightly coupled training. The same benchmarks show the tested H200 configuration delivering lower latency than MI300X across many configurations. That is a combined hardware-and-serving-stack result, in which NVIDIA's TensorRT-LLM and CUDA collective stack extracted more of the paper FLOPS than AMD's vLLM-on-ROCm path did at test time. It is a measured gap for that comparison, and it is why the datasheet does not settle the question.

The current cross-vendor basis is SemiAnalysis's InferenceX v2 (2026-02-16), which ran MI355X against B200, B300 and GB300 NVL72 serving stacks. Its result splits along the same line as AMD's two bets. On single-node FP8 serving, AMD's SGLang path gave better performance per TCO dollar than NVIDIA's SGLang path, and on FP8 disaggregated prefill MI355X was roughly level with B200. With disaggregated serving, wide expert parallelism and FP4 combined, AMD was not competitive: those optimizations did not yet compose in its software stack, and rack-scale GB300 NVL72 led on throughput per GPU. The software moves fast. In a vLLM report of 2026-09-10, AMD and Embedded LLM ran MiniMax M3 in MXFP8 at concurrency 32 on four GPUs with TP4/EP1 held fixed, and output rose from 109.1 to 342.4 tokens/s/GPU between the day-zero and tuned checkpoints. That is a software gain on MI355X, not a cross-vendor TCO result. Any current comparison you rely on should fix the model revision, input and output lengths, precision, engine and release, topology, total hardware denominator, TTFT and TPOT targets, and pricing basis.

A matched AMD quote has to buy down this cost: paper FLOPS are an upper bound, and goodput is what you pay for. A dated MI300X-versus-H100/H200 run can leave paper FLOPS and HBM bandwidth unused, so read the result at that run's model, engine, precision and release. Look at the software path: less-optimized GEMM and attention kernels, a younger RCCL collective library, framework features that land on CUDA first, and a long tail of operators that work but are not yet tuned. Recent ROCm releases have improved vLLM and SGLang support, attention kernels and framework coverage, and AMD's day-zero model enablement has improved. ROCm 10.1 was released on 2026-10-05. The gap that remains depends on the model, engine and release, so measure it on yours. CUDA has a ~15-year head start, a deeper third-party library ecosystem, and the pull that leads many model authors to optimize its path first. Chapter 7.9 has the full software lock-in treatment, including how to quantify switching cost.

The tax is paid in three currencies, and a TCO model that ignores any of them flatters the backend being introduced. First, engineering payroll: the FTEs who port, tune and maintain the ROCm path, real money that only amortizes across a large enough fleet. Price release churn into that line. ROCm 10.1 upgrades amdclang++ from LLVM 23 to 24, and AMD says rocm-smi is now fully deprecated in favor of amd-smi. Each release therefore carries compiler, build-script and monitoring migration work. Second, realized utilization: lower SLO-meeting inference throughput, or less retained training progress, at the same hardware cost inflates your effective cost per token and eats into the discount. Third, schedule and risk: new-model enablement lag and thinner operational tooling mean slower time to production and more debugging at scale. Convert the discount and the tax into one unit, cost per million tokens at your measured useful output and service targets, before you compare. Chapter 7.11 covers technical qualification and Chapter 1.8 the appraisal.

AMD bought its FLOPS the way NVIDIA did, with power. The MI300X's 750 W grew to the MI325X's 1,000 W and the MI355X's 1,400 W, and MI455X/Helios is a liquid-cooled rack-scale part from the outset. The facility consequence tracks the NVIDIA density ramp in Part 5. At 1,400 W the MI355X is past the air-cooling cliff and needs direct-to-chip liquid. AMD's air-cooled option is the MI350X, with the same 288 GB and 8 TB/s at 1,000 W and ~9.2 PFLOPS MXFP4, roughly 9% less peak compute, in standard air-cooled 8-GPU servers. Helios is liquid-only. It uses Meta's Open Rack Wide double-wide form factor, with in-rack power shelves feeding a 50 V busbar, and it arrives as a liquid-cooled, rack-scale integration unit, not a box you rack by hand. So evaluate a liquid-cooled AMD part on cost per token on the same facility basis as NVIDIA: cooling plant, high-density power chain, reinforced floors.

The numbers still differ in detail. The AMD and Schneider Electric Helios reference design (2026-07-23) supports racks up to 246 kW and modular clusters up to 10.4 MW of IT load, with liquid capturing up to 84% of the heat. At 84%, about 40 kW a rack (a guide calculation) still goes to air. AMD's separate architecture description (2026-08-04) puts liquid capture at roughly 89%, so size the plant from your OEM configuration's heat split. The mass differs too. Lisa Su put the double-wide Helios rack at nearly 7,000 lb (about 3.2 t) at CES on 2026-01-05, against roughly 4,000 lb for a Vera Rubin NVL72 (NVIDIA, 2026-03-16). Treat them as dated platform figures, not rack masses measured on one basis. Ask for the delivered configuration's operating and transport masses, dimensions and support reactions, and compare footprint pressure and point loads separately (Chapter 6.2, Chapter 6.7).

What does differ in Helios's favor is the open OCP rack lineage. It rides Meta's Open Rack Wide contribution, a hedge against single-vendor rack lock-in that leaves the density and cooling physics untouched. Chapter 7.11 builds the cross-vendor cost-per-token model that decides all of this, and Chapter 16.2 has the consolidated per-generation roadmap.

The purchase decision needs the current MI455X specification and a dated Helios offer. AMD is a real second source where the workload and delivery check out, and for memory-bound inference it is frequently the right economic choice on its own merits, before you count the strategic value of supply leverage. For tightly coupled frontier training, establish the risk by testing the specific RCCL collective maturity and the open scale-up fabric (the implemented UALoE or native UALink bill of materials) at the required scale and recovery boundary. AMD's trajectory turns on whether ROCm closes the realized-MFU gap and whether UALink hardware matures fast enough to make multi-vendor rack integration a genuine procurement hedge. The anchor-tenant deals are the market's bet that it does. So capture a quoted saving where measured migration and support costs leave one, pursue the second source for leverage wherever it breaks even, and keep irreversible commitments (a training campus's scale-up fabric) on a shorter leash than reversible ones (a portable inference stack) until the open fabric has shipped at scale.