HBM · High-Bandwidth Memory
Stacked DRAM mounted on-package with the accelerator; the bandwidth and capacity ceiling and the key supply bottleneck.
Vertically stacked DRAM bonded next to a processor for huge memory bandwidth; AI's binding supply constraint.
Also written as: High-Bandwidth Memory
Current numbers
144 chipsper Trn3 UltraServer = 362 FP8 PFLOPS, 20.7 TB HBM, 4x perf/watt vs Trn2
80 → 288 GB → ~1 TBHBM capacity per accelerator package: H100 80 GB; B200 180 GB; B300 288 GB (HBM3E); Rubin Ultra ~1 TB HBM4E on a four-chiplet package (roadmap)
H100 80GB → Rubin Ultra 1TBper-GPU usable HBM capacity trajectory (H100 80GB, B200 SXM 180GB, B300 SXM 288GB, Rubin 288GB HBM4, Rubin Ultra announced at 1TB)