All-reduce
A collective operation that sums gradients across all GPUs and shares the result, the dominant traffic in training.
Current numbers
~806–928 GB/s (2–32 GiB; sweep average ~872 GB/s)GB200 NVL72 in-domain all-reduce busbw across 2–32 GiB buffers (one provider's 72-rank run); scale-out gate set separately