BF16 · Brain Float 16
A 16-bit floating-point format with a wide exponent range, a common default for stable AI training.
Current numbers
16 B/parameter for the stated layoutGuide derivation: persistent training state for BF16 mixed-precision Adam (BF16 weights and gradients, FP32 master copy and moments)
~14 bytes/paramassumed BF16 weights + FP32 master + two FP32 Adam moments; unique serialized state, before implementation metadata or replication
34% → 54%BF16 MFU on a 128-H100 cluster (GPT-3 175B) rose in 2024 from software and kernel maturation: same silicon, more throughput