TPOT · Time Per Output Token
The steady-state delay between successive generated tokens; the inter-token latency SLO governed by decode.
Current numbers
up to ~77%goodput gain from hybrid aggregation/disaggregation over SOTA when both TTFT and TPOT bind