Checkpoint
A periodic save of model weights and optimizer state so a long training run can resume after a failure.
Current numbers
2.45–14 TB (exact assumed layout)Exact constructed checkpoint layout: 175 billion × 14 B = 2.45 TB; 1 trillion × 14 B = 14 TB
age of newest surviving captured statetraining lost-work boundary: captured progress in the newest valid checkpoint whose durable copy and recovery dependencies survive the disaster
~14 bytes/paramCheckpoint state per parameter under a BF16 / FP32-master / Adam mixed-precision recipe (assumed, not measured)