Host CPUs, GPU:CPU Ratios & System Composition

An accelerator does not run a workload by itself. It runs as the muscle of a system — a host CPU that loads the model, drives the data pipeline, schedules kernels, services interrupts, runs the OS and the container runtime, terminates the storage and management network, and increasingly does real application work: tokenization, retrieval, sandboxed tool execution, agent orchestration, and the reward-evaluation loop of reinforcement learning. The question this chapter answers is not "which GPU" — that is Chapter 7.2 through Chapter 7.5. It is the question that sits one level up: what machine do you wrap around the accelerator, and in what ratio. Get it wrong in one direction and the host starves the GPUs — they idle waiting on data that the CPU pipeline cannot deliver, and you have paid for the most expensive silicon in the building to wait. Get it wrong in the other direction and you have bought host cores and host memory that no kernel ever touches — dead capex riding on a power budget you are otherwise fighting to conserve.

This was a quiet, settled question for the training era. The answer was "as little CPU as you can get away with": eight GPUs per host, the cheapest x86 that could keep the pipeline fed, and as much of the silicon budget as possible spent on accelerators. Two forces broke that settlement. First, coherent host-attach — NVIDIA's NVLink-C2C welding a Grace (and now Vera) Arm CPU to the GPU as a single memory-coherent superchip — turned the host from a discrete component into part of the accelerator's memory hierarchy. Second, and more disruptively, agentic inference and RL shifted real, sustained compute back onto the host. The orchestration layer that schedules sub-agents, routes tool calls, runs retrieval, and decides whether a task is done is CPU work, and it does not fit in the slivers of host left over from an 8:1 training box. The industry is re-deciding system composition in real time, and the fork has consequences that propagate into power, density, supply, and serviceability for the life of the fleet.

There are two physically and architecturally distinct ways to attach a host CPU to accelerators, and the choice is the first fork in system composition. They differ in the interconnect, the memory model the programmer sees, the failure blast radius, and — critically in 2026 — in how tightly they couple you to a single vendor's supply chain.

Coherent host-attach (the superchip model). NVIDIA's Grace-Blackwell and Vera-Rubin packages connect an Arm host CPU to the GPU(s) over NVLink-C2C, a chip-to-chip link delivering ~900 GB/s of coherent bandwidth on Grace-class parts (rising on Vera). "Coherent" is the load-bearing word: the CPU's LPDDR5X and the GPU's HBM live in one cache-coherent address space, so the GPU can reach host memory and the CPU can reach HBM with hardware coherence, no explicit copy, no staging through a bounce buffer. The host memory becomes a fast, large NUMA tier behind the GPU — the natural home for KV-cache spillover, large embedding tables, MoE expert weights that do not fit in HBM, and the working set of long-context and agentic inference. The cost: the CPU is welded to the GPU. You take the ratio the package ships (2 GPUs : 1 Grace in GB200; the Vera-Rubin pairing in 2026), the LPDDR5X capacity is soldered and fixed, and you are buying one vendor's CPU, GPU, and interconnect as an indivisible unit.

Discrete host-attach (the PCIe model). The classic HGX/8-GPU baseboard hangs accelerators off a pair of x86 sockets (AMD EPYC or Intel Xeon) over PCIe — Gen5 at ~64 GB/s per x16, Gen6 at ~128 GB/s. The CPU and GPU keep separate memory spaces; moving data between them is an explicit DMA over the PCIe link or, for GPU-to-GPU, a sideband over NVLink that bypasses the host entirely. You give up coherent host memory — host RAM is not a transparent extension of HBM, and PCIe bandwidth is an order of magnitude below NVLink-C2C — but you gain composability. You choose the CPU vendor, the core count, the socketed DDR5 capacity (terabytes if you want them), and the GPU:CPU ratio independently, and you can service or upgrade the host without touching the accelerator. For workloads that do not need coherent host memory — most discrete-GPU inference, and any fleet that values dual-source supply over the last increment of host-GPU bandwidth — this is the pragmatic default.

The GPU:CPU ratio is the single number that most concisely captures system composition. For the training era it was effectively a constant: 8:1. An HGX baseboard carried eight accelerators and a dual-socket host whose job was to keep the data pipeline full and stay out of the way. The host did almost no application compute — pre-training is GPU-bound on collectives and matmul, and the CPU's contribution is loading shards, augmenting data, and launching kernels. The rational move was to minimize host spend: the dollars and the watts belonged to the GPUs. Even host memory got cut — neocloud reference builds trimmed training-server RAM from 2 TB to 1 TB to save roughly $13.6k per server precisely because the trainer never used it.

Agentic inference and RL break the constant. The work that defines an AI agent — decomposing a request into sub-tasks, scheduling them, routing tool calls, passing state between sub-agents, running retrieval, executing code in sandboxes, and evaluating whether the original goal is met — is overwhelmingly host-side, control-heavy compute. It is branchy, latency-sensitive, and does not vectorize onto a GPU. RL training compounds this: every action an agent takes during a rollout has to be executed and scored, and that evaluation loop runs on the CPU. As agents proliferate, the host stops being a feeder and becomes a co-processor doing sustained work, and the ratio that kept GPUs the scarce resource per node no longer holds. TrendForce projects the ratio shifting from the training-era 1:4–1:8 toward 1:1–1:2 in agentic deployments; Intel frames the same move as 1:8 heading toward 1:1. Arm goes further on the aggregate: roughly 30 million CPU cores per gigawatt in a traditional AI data center rising to ~120 million per gigawatt in the agent era — a fourfold increase in host cores per unit of power, and a >$100B data-center-CPU opportunity by 2030.

The consequence for system composition is concrete. A fleet scoped at 8:1 for training and then repurposed for agentic serving will be CPU-starved: the orchestration and tool-execution layer saturates the thin host, queues form ahead of the GPUs, tail latency blows past the SLO, and accelerators sit underutilized behind a host bottleneck. The fix is not more GPUs — it is more CPU per GPU, which means a different node, often a different rack SKU, and a host-side power and memory budget the training composition never planned for. This is why NVIDIA designed Vera explicitly "for agents": 88 custom Olympus cores, 176 threads via spatial multithreading, and up to 1.5 TB of LPDDR5X at ~1.2 TB/s — a host built to do real work, not just feed.

The memory the host carries is the third axis of composition, and it tracks the host-attach fork tightly. The coherent superchips use LPDDR5X — the low-power DRAM from the mobile world — soldered directly onto the package. The reasons are bandwidth-per-watt and density: Grace pairs 480 GB of LPDDR5X at ~546 GB/s with a ~250 W TDP, and NVIDIA's framing is roughly 2x the bandwidth at half the power of conventional socketed server memory. Vera pushes this to up to 1.5 TB at ~1.2 TB/s. In a power-bound rack where every watt spent on host memory is a watt not spent on accelerators, LPDDR5X's efficiency is the whole point. The price you pay is rigidity: the capacity is fixed at manufacture, you cannot add or upgrade it, and a memory fault is a package fault — there is no DIMM to pull and replace, and LPDDR's RAS posture is weaker than registered server DRAM with full ECC and post-package repair.

The discrete x86 hosts use socketed DDR5, and increasingly MRDIMM (multiplexed-rank DIMMs) on the Intel side to claw back the bandwidth gap — Granite Rapids reaches MRDIMM-8800 against AMD's DDR5-6000/6400 on EPYC Turin. Socketed memory gives you what the superchip cannot: you size capacity to the workload (multiple terabytes for embedding tables, large KV-cache, or in-memory retrieval indices), you get full server-grade RAS, and you field-service or upgrade memory without touching the accelerators. The cost is power and density — registered DDR5 burns more per gigabyte than LPDDR5X, and the DIMM slots consume board area and a slice of the rack's thermal and electrical budget. The decision mirrors the host-attach fork: take the superchip and you take its soldered LPDDR5X efficiency-and-rigidity bargain; take the discrete host and you can spend power to buy capacity, bandwidth, RAS, and serviceability on your own terms.

A common mistake is treating composition as a single answer. A real AI operation runs more than one workload shape, and the host-attach model, the GPU:CPU ratio, and the host memory that suit a pre-training trainer are the wrong answers for an agentic-inference fleet — and vice versa. The discipline is to recognize that system composition is plural, and to compose distinct node and rack SKUs for the workloads that actually dominate your draw.

A training SKU wants the coherent superchip ratio as shipped (2:1 in GB200), minimal host application compute, host memory sized to the data pipeline not to the model, and every spare watt routed to accelerators. An agentic-inference SKU wants the opposite: a fat host — many cores for orchestration and tool execution, large host memory for KV-cache and retrieval state, and a GPU:CPU ratio near 1:1–2:1 so the host never becomes the queue ahead of the GPUs. A batch-inference SKU sits between them, throughput-bound and tolerant of a thinner host. Forcing one composition across all three is how you end up simultaneously paying for unused host memory on the trainers and starving the agentic fleet of the CPU it needs. This is the same plurality the archetype framework demands at facility scale (Chapter 1.1) — here it lands inside the node.

One last decision sits underneath all of this: how much of system composition do you let the vendor make for you. The rack-scale coherent platforms (GB200 NVL72, Vera-Rubin) ship as integrated units with the CPU:GPU ratio, the host CPU, the host memory, and the interconnect all fixed at the factory. You buy a rack, not a node, and the composition is a given. That is genuinely valuable — the integration risk of welding 72 GPUs, 36 CPUs, NVSwitch trays, and a liquid-cooling loop into one coherent domain is enormous, and the vendor has absorbed it. But it means the GPU:CPU ratio, the host ISA, and the host memory are not yours to tune. If your agentic workload wants a fatter host than the package ships, your only lever is a different SKU, not a different host.

The discrete path keeps composition in your hands. You build the node — choosing AMD or Intel, the core count, the socketed memory, the GPU count per host, the ratio — and you own both the flexibility and the integration risk. For a fleet with heterogeneous workloads and a procurement strategy that refuses to single-source, this is the path that lets composition track the workload. The 2026 reality is that most large operators run both: bundled coherent racks where the workload genuinely needs the coherent domain (training, memory-hungry inference), and composed discrete nodes where composability and dual-source supply matter more. The decision is not which path is correct in the abstract — it is which workloads belong on which path, and that lands you right back at the fleet-plurality discipline above. → procurement and fleet composition in Chapter 7.11.