Precision, Quantization & the Compute-Memory Tradeoff

The largest number on every accelerator datasheet is almost always an FP4 number. The GB200 NVL72, GB300 NVL72 and Vera Rubin NVL72 headlines are FP4 rack peaks, and AMD specifies 10.1 PFLOPS as the MI355X's MXFP4 matrix peak. These numbers exist because a natively supported lower precision raises throughput on the same matrix hardware, although the power it draws and the speedup your model sees still have to be measured. On B200 and MI355X, a tensor core does roughly two FP8 and four FP4 multiply-accumulates for each BF16 one, in the same silicon at the same clock. Later parts widen the gap: dense BF16:FP8:FP4 is about 1:2:6 on GB300 and about 1:4:9 on Vera Rubin. Halving the bit-width also halves the bytes moved across HBM, NVLink and the network, and in memory-bandwidth-bound inference that second effect often matters more than the first.

Read the headline with its footnote. NVIDIA's FP4 rack peaks are sparse specifications, and the dense figure is not a fixed fraction of them. GB200 NVL72 lists 1,440 PFLOPS sparse and 720 dense; GB300 NVL72 lists 1,440 sparse and 1,080 dense. Vera Rubin NVL72 lists 3,600 PFLOPS of NVFP4 inference (sparse specification) against 2,520 PFLOPS of NVFP4 training (dense specification). Quote the dense figure and say which footnote it carries. Never halve a headline to get it.

Precision is not free performance. Every bit you remove narrows a number's dynamic range or its mantissa resolution, and somewhere down the ladder the outputs start to drift: a perplexity bump, a few points off MMLU, a long-context retrieval that quietly fails. This chapter climbs the precision ladder rung by rung (FP32 → BF16 → FP8 → FP6 → FP4). It explains how microscaling, and the NVFP4-versus-MXFP4 split inside it, made FP4 viable at all, weighs FP4 training against holding at FP8, and treats the KV-cache as the precision decision that most directly buys or burns inference goodput.

A floating-point format splits a bit budget between exponent bits (dynamic range: how large and small a number can get) and mantissa bits (precision: how finely it resolves values within that range). The history of AI numerics is the discovery that neural networks need a lot of range but tolerate surprisingly little precision, followed by a ruthless cut to the mantissa to cash that in.

FP32 (1-8-23) was the training default through about 2017 and is now a niche: a place to accumulate, and a fallback for a few numerically fragile operations (some normalizations, some loss scaling). TF32 (NVIDIA's 1-8-10 internal tensor-core format) was a transitional hack, FP32's range with a truncated mantissa, that bought a free speedup with no code change. BF16 (1-8-7) defined the modern era. It keeps FP32's full 8-bit exponent, so loss scaling and gradient dynamics behave like FP32, while throwing away two-thirds of the mantissa. BF16 is still the safe default for training stability and the reference everything else is measured against. FP16 (1-5-10) trades range for precision, which is why the industry needed loss scaling; outside legacy inference paths it has largely lost to BF16.

The interesting rungs sit below 16 bits, where a raw format no longer has the range to survive alone and needs a scaling scheme. FP8 comes in two flavors: E4M3 (more precision) for forward-pass activations and weights, and E5M2 (more range) for gradients. By 2026 FP8 is the production default for inference and, increasingly, for the bulk of training matmuls. FP6 (typically E3M2/E2M3) is the awkward middle rung. AMD's CDNA4 and some inference stacks support it, and it sometimes wins as a quality-preserving step for weights that FP4 degrades too far. What it buys depends on the platform. On NVIDIA it runs at the FP8 rate, so it saves memory but not compute. On CDNA4 it runs at the FP4 rate, so it is a quality step over FP4 that costs no throughput, only memory (6 bits plus scale against 4). Either way it stays a specialist tool rather than a tier. FP4 (E2M1: two exponent bits, one mantissa bit, sixteen representable values) is the current frontier and the reason this chapter exists: it is where the throughput is, and where the accuracy cliff lives.

A rung pays off when the hardware runs it natively. Halving the payload bit width halves raw operand bytes, but scales, padding and missing kernels change the bytes actually stored and the throughput actually delivered. NVIDIA B200's published dense rates run 1:2:4 across 16-bit, FP8 and FP4, a ratio that not every format or device shares. At FP8 and below, the raw format's dynamic range is too narrow for the spread of values inside a real tensor: activations have outliers and weights have long tails. The fix is to factor each tensor into a low-precision payload plus a higher-precision scale. The granularity of that scale (per-tensor, per-channel or per-small-block) is the most important quality lever below 8 bits, and it is where NVFP4 and MXFP4 part ways.

FP8 training shows how far the granularity has moved. Hopper-era recipes used one scale per tensor, often set from the maximums of recent steps, which works only while each tensor's value distribution stays stable from step to step (NVIDIA, July 2025). DeepSeek-V3 trained in FP8 with a scale for every 1×128 activation tile and every 128×128 weight block, so one outlier coarsens only its own tile (DeepSeek-V3 technical report, December 2024). Blackwell runs MXFP8 in hardware: 32-value blocks, each with a power-of-two scale. Per-tensor FP8 remains available there, but a Hopper-tuned recipe moved onto the MXFP8 path gets a different scaling scheme, so check its convergence again instead of assuming it transfers.

A single scale factor for a whole tensor has a flaw: one outlier forces the scale large, which crushes every small value into the same few FP4 codes. You spend all sixteen representable values describing the outlier and have nothing left for the signal. Microscaling (MX) fixes this by giving every small block of elements its own scale. A block of small values gets a small scale and fine resolution, and a block with an outlier gets a large scale locally without contaminating the rest of the tensor. The Open Compute Project's MX formats standardized the idea, and it turned FP4 from a research curiosity into a shippable inference format.

The choice inside microscaling is the one practitioners actually have to make, because it is wired into the hardware and the checkpoint format. MXFP4 (the OCP open standard) uses 32-element blocks with an E8M0 scale, a pure power-of-two exponent, so every scale snaps to the nearest 2ⁿ. NVFP4 (NVIDIA's Blackwell-native format) uses 16-element blocks with an E4M3 (FP8) scale. The smaller block adapts more locally, and the real mantissa in the scale lets it land between powers of two. A second-level FP32 per-tensor scale backs it for global range. The payoff is measurable. NVIDIA's scale-encoding illustration shows mean-squared error of 0.72 (E8M0) versus 0.08 (E4M3): (0.72 − 0.08)/0.72 is about 89%. That compares encoded scale values, not full tensor quantization or model accuracy (NVIDIA, Introducing NVFP4, June 24, 2025).

For years the consensus was that FP4 was inference-only: fine for a frozen model, fatal for a gradient-descent loop where small errors compound over trillions of steps. That consensus broke in 2025. Peer research showed that native FP4 training can be Pareto-optimal at a fixed compute budget: for a given number of FLOPS, an FP4 run can reach better loss than a BF16 or FP8 run, because the throughput gain buys more tokens or more parameters per dollar (Castro et al., Quartet, arXiv 2025). NVIDIA then demonstrated it at scale with Nemotron 3 Super, a 120B-total / ~12B-active hybrid Mamba-Transformer MoE pre-trained on 25 trillion total seen tokens in NVFP4, the longest publicly documented 4-bit training run as of March 2026 (NVIDIA, Introducing Nemotron 3 Super, 2026). In June 2026 NVIDIA documented Nemotron 3 Ultra, with 550B total and 55B active parameters, pre-trained in NVFP4 on 20 trillion text tokens. By model size it was the largest publicly documented NVFP4 pre-training run when published, though it saw fewer tokens than Super.

The word doing the work in 'Pareto-optimal' is recipe. FP4 training does not work by flipping a precision flag. It works because a stack of techniques keeps the gradient signal alive: master weights held in higher precision (BF16/FP32) while only the matmul operands are FP4; stochastic rounding, so quantization error is unbiased instead of systematically truncating; selective high-precision layers (embeddings, the final projection and the most sensitive blocks stay at FP8/BF16); and careful handling of the backward pass, where gradients have the widest dynamic range and are the first thing to break. Skip any of these and the run either diverges or silently underperforms a cheaper FP8 run. That is the worst outcome, because you paid the FP4 risk premium and got nothing.

Inference is where quantization pays off most and where failures are most visible to users. The post-training quantization (PTQ) toolkit is mature. Weight-only schemes (GPTQ, AWQ) quantize the weights and keep activations at higher precision; they suit memory-bound, small-batch decode, where the bottleneck is reading weights from HBM. Weight-and-activation schemes (the FP8/FP4 path) quantize both and unlock full tensor-core throughput, which you need when you are compute-bound at large batch or in prefill. SmoothQuant-style transforms migrate activation outliers into the weights so both can be quantized cleanly. The choice follows the workload. A latency-bound chatbot serving one request at a time is memory-bound and wants weight-only INT4/FP4. A high-throughput batch pipeline is compute-bound and wants full FP8/FP4 matmuls.

In practice publishers now often ship the quantized format themselves. gpt-oss stores its MoE weights in MXFP4 at 4.25 bits per parameter, which lets the 120B model fit on one 80 GB GPU. Kimi K2 Thinking uses INT4 MoE weights produced by quantization-aware training, and Moonshot describes the result as lossless. NVIDIA publishes Nemotron 3 Ultra as an NVFP4 checkpoint next to a BF16 one. Before you adopt a checkpoint, confirm your hardware runs its format natively. Without a native path the weights get dequantized and take several times the memory. On Trainium2, for example, gpt-oss runs in BF16, and only Trainium3 runs it in MXFP4. Quantizing again from a publisher-quantized checkpoint into a different format means a full re-evaluation.

A checkpoint labeled NVFP4 is often mixed precision. NVIDIA's NVFP4 release of Nemotron 3 Ultra (June 2026) puts the routed experts in NVFP4. The shared experts, the Mamba projections and the KV cache are FP8, while roughly 17 billion parameters remain BF16. The 550B-parameter checkpoint takes 352 GB, about 5.1 bits per parameter rather than the format's 4.5. On NVIDIA's model card, most of the percentage-scored benchmarks land within about a point of BF16, and roughly half of them improve. Four lose 2.5 points or more: Terminal Bench 2.1 drops from 56.4 to 53.9, BrowseComp from 44.4 to 41.4, TauBench banking from 22.6 to 19.2 and OmniScience non-hallucination from 78.7 to 75.5. Budget memory from the checkpoint's actual size, and gate on the tasks your service runs.

The hard part is the accuracy gate, because it is easy to measure the wrong thing. A perplexity delta of a few hundredths looks reassuring and can hide a real regression on a downstream task that matters: code generation, multi-step math, instruction-following in a specific domain. The 2026 discipline is to gate quantization on a task-eval suite per model family, not one perplexity number. The same NVFP4 recipe that costs one point on MMLU-Pro for one model can cost several points of pass@1 on a code benchmark for another. NVFP4's headline result, a drop of one point or less against FP8 on DeepSeek-R1-0528 with AIME 2024 actually improving by two points in NVIDIA's measurement, is real, but it is a per-model result, not a guarantee. Reasoning models that emit long chains are especially exposed: a small per-token error compounds over a 10K-token decode, so a format that is fine for short answers can degrade a long reasoning trace.

For long-context and reasoning inference, the binding resource on a decode node can shift from weights to the KV-cache, the per-token key/value tensors that every attention step must re-read. Its size scales with batch size times sequence length, and for a frontier model at 100K+ context it can dwarf the model weights in a decode node's HBM budget. That makes KV-cache precision the lever that most directly governs how many concurrent users a node can hold. BF16 → FP8 halves the raw cache payload and FP8 → INT4 halves it again, but fixed weights, scales, reserve and bandwidth stop that from automatically doubling concurrent sequences. In an inference business, extra fitted sequences improve goodput only while quality and latency still pass (Chapter 7.11).

The catch is that the KV-cache is more sensitive to quantization than the weights, because errors in keys and values feed straight into the attention softmax and compound across every later token. FP8 KV-cache is supported on some engines and models; test long-context recall, reasoning and tail latency on your actual recipe. INT4 KV-cache is where long-context recall starts to degrade, because the model loses the fine distinctions that let it retrieve a fact buried 80K tokens back. So it is typically applied selectively, with recent tokens at higher precision and older tokens compressed, rather than uniformly. On Blackwell, NVIDIA's NVFP4 KV cache (December 2025) cuts KV memory by about 50% against FP8 and dequantizes to FP8 before the attention math, so the gain is memory and bandwidth, not FP4 attention arithmetic. NVIDIA reports under 1% accuracy loss against BF16 and FP8 on code, knowledge and RULER 64K long-context benchmarks for Qwen3-480B-A35B. That is NVIDIA's own measurement, so gate it on your own long-context evals before trusting it.

The mature pattern is a tiered KV hierarchy: hot KV in HBM at FP8, warm KV spilled to host DRAM or CXL, and cold KV offloaded to NVMe-attached flash. Storage tier and precision are chosen independently, because colder data need not tolerate more error. It is the inference-side analog of the storage memory hierarchy, and it is where KV quantization, disaggregated prefill/decode and the new Ethernet-attached context-memory tier converge (Chapter 9.7).

Baseline accuracy = 9,000/10,000 = 90.0%. FP8 = 89.4%, a 0.6-percentage-point drop; NVFP4 = 88.8%, a 1.2-point drop. Both runs clear throughput and latency, but only FP8 clears 89.0%. NVFP4 must reach at least 8,900 correct items on this fixed test, an increase of 20, while still passing the other gates. The threshold is in percentage points, not relative percent accuracy.

Ship FP8, even though NVFP4 moves a third more tokens. To reopen NVFP4, recalibrate or change the quantization recipe and rerun the complete evaluation. Extra speed cannot buy back wrong answers, and the cost per useful token that Chapter 7.11 computes starts from this choice.

Every one of these mistakes treats precision as a single global flag, when it is a per-phase, per-workload decision gated on the right metric.

• Gating on perplexity instead of task evals. A reassuring perplexity delta hides a real regression on the brittle tasks (code, math, long-context retrieval) that users feel. Gate quantization per model family on a downstream eval suite, not on one intrinsic-metric number.
• Flipping the precision flag and calling it done. FP4 training without master weights, stochastic rounding and selective high-precision layers either diverges or silently underperforms a cheaper FP8 run, so you pay the FP4 risk premium for negative return.
• Uniform KV-cache quantization for long context. Pushing the whole KV-cache to INT4 to maximize concurrency can cost you long-context recall, because the oldest tokens, the ones a retrieval depends on, lose their precision. Tier the cache and keep recent, hot KV at higher precision.
• Picking a format for the headline FLOPS number. If you buy silicon on its FP4-with-sparsity peak and your quality gate then forces FP8 for most traffic, the FP4 peak is unreachable. Spec to the precision you can actually ship, not the datasheet's largest number (Chapter 7.1). Also check which precision a new generation actually improves. On sparse tensor rates, GB200 and GB300 NVL72 tie at 720 PFLOPS for FP8 and 360 PFLOPS for BF16. Dense FP4 is the one tensor rate that rises, from 720 to 1,080 PFLOPS, while INT8 drops from 720 to 24 POPS. A model that your quality gate holds at FP8 gains HBM capacity and faster attention softmax from the upgrade, but no tensor-core throughput. Port INT8 W8A8 recipes to FP8 before they meet Blackwell Ultra. An August 2026 arXiv audit, a single-author preprint, found that mainstream kernels and engines ship no INT8 tensor-core path there.