MoE · Mixture of Experts
A sparse model that activates only a subset of expert sub-networks per token, cutting compute per token at scale.
An architecture routing each token to a few specialized sub-networks, widening parallelism and reshaping fabric needs.
Also written as: Mixture of Experts
Current numbers
~1/4 the number (~75% fewer, ~4×)Vera Rubin GPUs to train an MoE model vs Blackwell (training metric)
25T tokenslongest documented 4-bit pre-training run (Nemotron 3 Super, 120B-total / ~12B-active hybrid Mamba-Transformer MoE, NVFP4, 25T total seen tokens)
~10xtokens/watt advantage of Blackwell-class over Hopper on MoE inference — the power-limited lever