vLLM
A popular open-source inference engine known for PagedAttention and high-throughput continuous batching.
Current numbers
~$1.90/M tokCost per million tokens, Spheron: Llama-3.1-70B FP16 on 8×H100 at $2.40/GPU-hr, ~2,800 tok/s (vLLM, batch 256, 512-in/512-out)