HBM: The Binding Constraint on AI Compute

Computing performance used to track the logic die: more transistors, more clocks, more FLOPS. Accelerators changed that. A low-batch decode GPU can spend much of its time waiting on memory, its arithmetic units starved far more often than saturated. The DRAM stacked beside the die is also one of the costliest parts of the package, and it can gate a delivery date. High-Bandwidth Memory (HBM) can bind AI capacity or bandwidth in 2026. Operators who understand that buy chips by the terabyte per second and the gigabyte, not by the petaFLOP.

HBM here is a decision surface, not a spec sheet. Three generations (HBM3E, HBM4 and HBM4E) sit at different volume and sample stages across a three-supplier oligopoly whose qualifications and allocations differ by product. The memory is the largest line in a flagship accelerator's modeled manufacturing cost, and every accelerator architect fights the same capacity-vs-bandwidth tradeoff over it. The 2026 supply crisis also spills into the broader DRAM market. The constraint connects upward to the procurement game (Chapter 2.3) and sideways to the packaging that determines how many stacks fit beside a die (Chapter 7.7).

The reason HBM exists is arithmetic intensity: the ratio of floating-point operations to bytes moved from memory. Dense matrix multiply, the core of training, has high arithmetic intensity and can keep the cores busy. Autoregressive decode at low batch is different. It generates one token at a time and reads the active weights and the attention state that step needs, though MoE routing, cache reuse, attention layout and batch size all change the bytes transferred. When reuse is low, arithmetic intensity can be low: the accelerator is bandwidth-bound, and the math engine idles waiting for bytes. Deloitte’s November 2025 forecast puts 2026 inference at roughly two-thirds of AI compute (Chapter 1.3), but that mix does not classify every phase. Measure compute, HBM traffic and collective time independently.

HBM is the answer to that wall. Instead of a few DDR channels reaching across a motherboard, HBM stacks 8, 12 or 16 DRAM dies vertically, connects them through the silicon with thousands of through-silicon vias (TSVs), and places the stack within millimeters of the compute die on a shared interposer. In HBM4 the interface is 2,048 bits wide per stack, against 64 data bits for a DDR5 DIMM (two 32-bit subchannels), delivering terabytes per second at a fraction of the energy per bit of off-package DRAM. That proximity is also the trap: HBM only works as part of an advanced package, which creates a package-assembly gate distinct from the HBM supplier’s qualified-memory gate (Chapter 7.7).

HBM advances on a JEDEC-anchored cadence, and 2026 sits on the seam between two generations. HBM3E carried most 2026 volume: 8-Hi and 12-Hi stacks at roughly 1.0 TB/s per stack, giving the ~8 TB/s per GPU you see on a Blackwell B200/B300, an AMD MI355X or a Google Ironwood. HBM4 entered volume with Vera Rubin, shipping since June 2026. The interface doubles to a 2,048-bit PHY per stack, and per-stack bandwidth rises from the JEDEC baseline of about 2 TB/s to roughly 2.4 TB/s in NVIDIA's Vera Rubin configuration, though Samsung says its bins reach 3.3 TB/s. With eight stacks that is 19.2 TB/s per Vera Rubin GPU, about 2.4x a B300; NVIDIA's launch announcement said ~22 TB/s. HBM4E follows as the speed-bin and capacity refresh, and TrendForce places its volume ramp in the second half of 2027. Samsung announced its 12-layer HBM4E sample shipments on May 29, 2026, and SK hynix followed on June 18. SK hynix's samples hold 48 GB and run at 16 Gbps per pin. Samples are not qualified volume.

The 2026 inflection is not bandwidth alone but who controls the base die. HBM4 moves the logic base die at the bottom of the stack from a commodity DRAM process to an advanced logic node (TSMC for SK hynix, Samsung's own foundry for Samsung), turning it into a semi-custom interface co-designed with the accelerator. That deepens the coupling between memory vendor, foundry and GPU designer, and it is the structural reason HBM4 qualification is slower and more entangled than a normal node shrink. One engineering subtlety sits beneath the marketing. Hybrid bonding got years of hype, but 2026 volume HBM4 (12-Hi) still uses microbump joining (MR-MUF or TC-NCF). TrendForce reported in January 2026 that SK hynix intends to stay on MR-MUF for its 16-Hi parts. Taller stacks are where hybrid bonding is expected to arrive next.

Every row of the table hides a fork. HBM4 widens the stack interface from 1,024 to 2,048 bits. Per-GPU bandwidth still depends on transfer rate and enabled stacks, and it arrives in a supply environment where commitments differ by supplier and window. So for a 2027 delivery the question is not only whether HBM4 improves your constrained workload. It is whether you can get HBM4, or later HBM4E, in volume on your schedule, or should lock allocation on a mature generation and accept its bandwidth ceiling.

Only three companies make HBM at scale: SK hynix, Samsung and Micron. By Counterpoint's count SK hynix still leads but is slipping, from 58% of HBM revenue in Q1 2026 to 50% in Q2. Samsung moved fastest, from 21% to 33%. By its own account it shipped the first commercial HBM4 in February 2026 and was first to sample 12-layer HBM4E, announced at the start of June 2026. Micron holds 18%, is ramping HBM4, and is co-developing a custom HBM4E with NVIDIA. Press reports disagree on who supplies what for Vera Rubin. Yonhap, via TrendForce in January 2026, had SK hynix at about two-thirds of NVIDIA's HBM4, while Korean press cited by TrendForce in March had Samsung leading. Treat any split as unconfirmed. These are revenue shares; TrendForce's March 2026 forecast for 2026 bit output was ~50% for SK hynix and ~28% for Samsung. Allocation is contractual, so none of these shares is your place in the queue. All three suppliers have said their 2026 HBM is committed: SK hynix and Micron in late 2025, Samsung in January 2026. The market is allocated, and that says nothing about where your order sits.

Concentration this tight has two consequences for your build. First, pricing power sits with the supplier: HBM is sold on long-term contracts, allocated quarters or years ahead, and the spot margin for late entrants is punishing. Second, qualification is a moat: getting a new supplier or generation into a shipping accelerator takes supplier-specific co-engineering and reliability data, so who can supply whom moves slowly and is largely locked by the time you place orders. In practice your integrator must contract the memory milestone and substitution rules, and your place in the queue was set by the accelerator vendor's allocation, not yours (Chapter 2.3).

HBM is the largest single line in a flagship accelerator's modeled manufacturing cost, ahead of the compute dies. Epoch AI's December 2025 cost model attributes about $2,900 of a B200's ~$6,400 production cost to its 192 GB of installed HBM3E (eight 24 GB stacks; HGX B200 exposes 180 GB), against ~$900 for the two compute dies. HBM and CoWoS packaging together make up about two-thirds. This is a modeled cost, not NVIDIA's disclosed bill of materials or your purchase price.

A modern accelerator carries 8 to 12 HBM stacks. At the 1H 2026 reference prices for 12-Hi stacks, about $300 for HBM3E and $500 rising past $600 for HBM4 (Chosun, August 2025; TrendForce, January 2026), eight HBM3E stacks cost about $2,400 and twelve HBM4 stacks at $600 about $7,200. That is memory alone, before the GPU die, interposer or substrate, and before the memory qualification, package-test responsibility and yield allowance that a delivered quote must carry. Those prices do not carry into 2027: TrendForce (September 29, 2026) forecasts 2027 blended HBM ASP up 121% year on year, and Micron reports significant price increases on its 2027 contracts. Budget 2027 deliveries from your integrator's quote, not from 2026 reference prices.

This reframes accelerator selection (Chapter 7.11). Two parts with similar FLOPS can carry different quoted HBM costs, and that difference matters only through delivered service and total system cost. It also reframes the vendor's roofline math: every extra TB/s a designer buys costs real money, so architects trade between spending the memory budget on capacity (more, taller stacks for bigger models and longer context) and bandwidth (faster stacks to feed the cores). That fork sets the part's personality.

Given a fixed package area, a fixed interposer reach and a fixed memory budget, an accelerator designer cannot maximize both HBM capacity and HBM bandwidth without bound. They trade against each other through stack count, stack height and the speed bin chosen. The right answer is set by the dominant workload, and getting it wrong produces a part that is technically impressive and commercially mismatched.

Optimize for capacity when the workload is large-model inference, long context or KV-cache-heavy agentic serving. Here the binding limit is whether the weights plus the KV cache for a useful batch size fit in HBM at all. Insufficient capacity forces you to shard the model across more GPUs, raising cost per token through communication overhead, or to evict KV cache, raising latency. That is why per-GPU HBM capacity has climbed so aggressively: H100 80 GB, H200 141 GB, B200 180 GB, B300 288 GB. Then it paused. The shipping Vera Rubin GPU keeps 288 GB while bandwidth rises 2.4x to 19.2 TB/s, a generation that spent its memory budget on speed. Rubin Ultra was announced at ~1 TB of HBM4E per four-die package. Since then, supply-chain reports in August 2026 had NVIDIA testing 192–256 GB configurations. SemiAnalysis has reported two compute dies per package (30 June 2026) and, on 13 September 2026, 192 GB per GPU. NVIDIA has confirmed none of these reports (Chapter 7.2). The same capacity-bound workloads explain why 16-Hi stacks matter, since each added die in the stack is pure capacity. Optimize for bandwidth when the workload is decode-bound throughput at moderate model size, where the cores starve for bytes faster than they starve for capacity. There the faster speed bin and the wider HBM4 interface earn their BOM.

Mismatching this fork is expensive and quiet, because the part still runs; it just runs uneconomically. A capacity-optimized part used for bandwidth-bound decode leaves bandwidth on the table and serves fewer tokens per second than its FLOPS suggest. A bandwidth-optimized part used for a model that does not fit forces sharding and pays a communication tax on every step. The precision and KV-cache choices in Chapter 7.10 are the software-side lever on exactly this tradeoff: quantizing weights and KV cache to FP8/FP4 buys back HBM capacity and bandwidth you could not get in silicon.

Weights = 70×10⁹ × 2 B = 140 GB. Define K = 2 × 80 × 8 × 128 × 2 B × 8,192, the exact KV bytes per sequence. Eight sequences need about 21 GB of KV; with the 20 GB reserve, the resident set is about 180 GB. Capacity screen: floor((186−140−20)×10⁹/K) = 9 sequences. At nine, total residence is below 185 GB. At ten it is about 187 GB, above 186 GB, so the integer decision uses exact tensor counts rather than rounded totals. Reading weights and KV at 4.0 TB/s takes about 40 ms before compute and collectives. The 50 ms bandwidth boundary is (140×10⁹+8K)/0.050, about 3.2 TB/s; the unrounded inequality governs. At 3.0 TB/s memory alone takes about 54 ms and fails.

Keep one replica at batch eight and take it to a full serving test, because a memory lower bound is not a latency pass. If the full run exceeds the budget, change precision, batching or placement and retest. Record the qualified-HBM and package-release milestones on separate lines: a model that fits still does not secure a shippable accelerator.

The usable capacity comes from NVIDIA's GB200 specifications, and the memory-time bound follows the CUDA performance memory model. Chapter 7.11 turns a fitted replica into cost per token.

The HBM crisis does not stay in the data center. Micron puts the trade between HBM3E and DDR5 at roughly 3:1 for equal bits. That is Micron's capacity-conversion figure, not a physical wafer-area ratio. Each wafer start moved to HBM costs about three wafers' worth of commodity bits. Through 2025, HBM3E sold at four to five times server DDR5 per bit, and that premium drew wafers toward HBM. In 2026 the pull weakened. DDR5 prices rose until conventional DRAM earned more than HBM, as TrendForce forecast in October 2025 and Samsung said on its April 2026 call, and Micron's September 2026 deck says its 2027 HBM price increases narrow the margin gap. Suppliers kept adding HBM while pushing the 2027 contract round sharply higher. TrendForce expects the HBM3E premium to shrink to one to two times by end-2026, but only because DDR5 is rising. Samsung puts HBM at about 20% of the industry's DRAM wafer capacity in 2026 and expects close to 30% in 2027. Host DRAM and HBM now bid for the same wafers on price, so both stay tight. TrendForce's May 2024 forecast for 2025 put HBM above 30% of DRAM market value and above 10% of DRAM bit capacity; that is a dated forecast with separate value and bit-capacity denominators, not a 2026 actual. The result in 2026 was a broad memory shortage: DDR5 spot prices surged (commodity modules multiplying through 2025-2026), server DRAM budgets blew out, and even non-AI buyers found themselves competing for capacity redirected to feed accelerators.

For an AI operator, this spillover is a second-order cost that is easy to miss at scoping time. The host memory for your GPU servers, the LPDDR5X or DDR5/MRDIMM that pairs with the accelerators (Chapter 7.8), draws on the same crowded commodity pool, so HBM scarcity inflates the host BOM beside the accelerator BOM. A build that budgeted host memory at 2024 prices and assumed easy availability is exposed on a line item nobody flagged. HBM is not a contained component decision. It reprices the entire memory stack of the machine, and the ripple reaches your CPU sockets and your storage cache.

The HBM shortage and the packaging shortage are two gates with separate owners and contracts, but they clear together or not at all. HBM delivers its bandwidth only because it sits on a 2.5D advanced package, TSMC's CoWoS family or an equivalent, that places the stacks within interposer reach of the compute die. So an accelerator cannot ship unless both the HBM stacks exist and there is CoWoS interposer area and assembly capacity to mount them. Through 2026 both gates were spoken for in advance. All three HBM suppliers had said their 2026 output was contracted, CoWoS lines were fully booked, and accelerator vendors had locked up large shares of packaging capacity years ahead; NVIDIA was reported in May 2026 to hold about 60% of 2026 CoWoS allocation.

For procurement, this means the number of accelerators that reach your floor is decided upstream of the GPU vendor's order book, at the memory supplier and the OSAT or foundry packaging line, and it is decided well before your rack ships. It is the engineering reason the procurement chapter (Chapter 2.3) treats HBM/CoWoS allocation as the real lead-time driver above assembly, and why design-for-substitution (a qualified second supplier within one HBM generation and package design) is worth more than raw integration speed. The interposer area available to a package also sets how many HBM stacks can physically sit beside the die, the direct link from packaging geometry to the capacity-vs-bandwidth fork above. Chapter 7.7 covers that stack-count-per-package engineering.

HBM is the clearest case in this guide of a component decision that is really a strategy decision. A late buyer cannot assume that a spot purchase, a faster fab or a shorter qualification will recover the required date. The only levers you hold are which part you select for its memory personality, how early you secure allocation, and how much software flexibility you build to substitute precision for memory you could not get in silicon. Operators who track their capacity, bandwidth and qualified-delivery gates make three moves the others miss. They read accelerators memory-first, they treat allocation as a years-ahead commitment rather than an order, and they design fleets that can absorb a qualified substitute, with capacity, bandwidth and software rechecked before release.