FP32 · 32-bit floating point
Single-precision format, the top of the precision ladder; mixed-precision training keeps master weights and optimizer moments here even when the math runs lower.
Current numbers
16 B/parameter for the stated layoutGuide derivation: persistent training state for BF16 mixed-precision Adam (BF16 weights and gradients, FP32 master copy and moments)
~14 bytes/paramassumed BF16 weights + FP32 master + two FP32 Adam moments; unique serialized state, before implementation metadata or replication