self-hosted Llama-3.1-70B FP16 on one 8×H100 node at $2.40/GPU-hr, ~2,800 tok/s per node (vLLM, batch 256, 512-in/512-out)
~$1.90/M tokderived
| Value kind | derived — Derived values are guide calculations; their result is only as fixed as the stated inputs, scope, and method. |
|---|---|
| Scope | Spheron's own marketplace rate. At GetDeploying's September 2026 median on-demand H100 price ($3.50/GPU-hr, 58 providers) the same throughput costs ~$2.80/M. Spheron reports FP8 roughly doubles throughput at the same hourly cost. For scale: NVIDIA's eight-H100 Llama 2 70B submission to MLPerf Inference v4.1 (August 2024, NVIDIA's own tuned software) implies ~21,500–24,400 tok/s from its stated per-GPU B200 ratios (4x server, 3.7x offline), ~$0.22–0.25/M at the same $19.20/hr. That is a benchmark best case, not production serving. |
| As of | 2026-04 |
| Source | Spheron (vendor blog, 2026-04-04) — assumptions stated; H100 median from GetDeploying (2026-09-29) |
| Derivation | $19.20/hr (8×H100 at $2.40/GPU-hr) ÷ (2,800 tok/s × 3,600 s / 1e6) = $1.90 per M tokens |
| Review | checking…review by 2026-12-31 · fast cadence |
| Recorded changes | last 2026-10-12 · 5 revisions tracked |
| Claim id | self-hosted-inference-8x-h100-19-20-hr-llama |
Where the guide uses it
← Full numbers register — every date-stamped figure in the guide, with revision history.