Host CPUs, GPU:CPU Ratios & System Composition

An accelerator does not run a workload by itself. It is the muscle of a system: a host CPU that loads the model, drives the data pipeline, schedules kernels, services interrupts, runs the OS and container runtime, and terminates the storage and management network. Increasingly the host also does real application work: tokenization, retrieval, sandboxed tool execution, agent orchestration and the reward-evaluation loop of reinforcement learning. This chapter is not about which GPU to buy (that is Chapter 7.2 through Chapter 7.5). It is about what machine you wrap around the accelerator, and in what ratio. Err one way and the host starves the GPUs: they idle on data the CPU pipeline cannot deliver, and you pay for the most expensive silicon in the building to wait. Err the other way and you buy host cores and memory that no kernel touches, dead capex on a power budget you are already fighting to conserve.

For the training era this was a settled question. The answer was as little CPU as you can get away with: eight GPUs per host, the cheapest x86 that keeps the pipeline fed, and the rest of the silicon budget spent on accelerators. Two forces broke that settlement. The first is coherent host-attach. NVIDIA's NVLink-C2C welds a Grace (and now Vera) Arm CPU to the GPU as one memory-coherent superchip, which turns the host from a discrete component into part of the accelerator's memory hierarchy. The second, with wider consequences, is agentic inference and RL, which push real, sustained compute back onto the host. The layer that schedules sub-agents, routes tool calls, runs retrieval and decides whether a task is done is CPU work.

A DGX/HGX B300 training node has a 4:1 GPU:CPU ratio (2 CPU packages : 8 GPUs); NVL72 racks are 2:1. Agentic serving and RL rollouts put sustained, branchy work back on the host, so a fleet built at the training ratio can run out of CPU while its GPUs wait. The industry is re-deciding system composition in real time, and the choice carries into power, density, supply and serviceability for the life of the fleet.

There are two architecturally distinct ways to attach a host CPU to accelerators, and the choice is the first fork in system composition. They differ in the interconnect, the memory model the programmer sees, the failure blast radius and, in 2026, how tightly they tie you to one vendor's supply chain.

Coherent host-attach (the superchip model). NVIDIA's Grace-Blackwell and Vera-Rubin packages connect an Arm host CPU to the GPUs over NVLink-C2C, a chip-to-chip link with ~900 GB/s of coherent bandwidth on Grace and 1.8 TB/s on Vera (NVIDIA, March 2026). Coherence does real work here. The CPU's LPDDR5X and the GPU's HBM sit in one cache-coherent address space, so the GPU can reach host memory and the CPU can reach HBM with hardware coherence, with no explicit copy and no bounce buffer. Host memory becomes a large, fast NUMA tier behind the GPU, the natural home for KV-cache spillover, large embedding tables, MoE expert weights that do not fit in HBM, and the working set of long-context and agentic inference.

The cost is that the CPU is welded to the GPU. You take the ratio the package ships (2 GPUs per CPU on both GB200 and Vera Rubin: 72 GPUs : 36 CPUs per NVL72 rack). You take the LPDDR5X capacity it ships, soldered on Grace and in field-replaceable SOCAMM modules on Vera. And you buy one vendor's CPU, GPU and interconnect as one indivisible unit.

Discrete host-attach (the PCIe model). The classic HGX 8-GPU baseboard hangs accelerators off a pair of CPU sockets, usually AMD EPYC or Intel Xeon, over PCIe: Gen5 at ~64 GB/s per direction per x16 (~128 GB/s bidirectional), Gen6 at ~128 GB/s per direction. NVIDIA's HGX Rubin NVL8 reference designs also offer Vera as a PCIe host. The CPU and GPU keep separate memory spaces. Data moves between them by DMA or by a supported unified-memory or page-migration path over PCIe, while GPU-to-GPU traffic uses NVLink and bypasses the host entirely.

You give up coherent host memory. Host RAM is not a transparent extension of HBM, and PCIe delivers several times less bandwidth than NVLink-C2C (about 7x less than Grace's link at Gen5 x16). In exchange you get composability. You choose the CPU vendor, core count, socketed DDR5 capacity (terabytes if you want them) and GPU:CPU ratio independently, and you can service or upgrade the host without touching the accelerator. For workloads that do not need coherent host memory, which covers most discrete-GPU inference and any fleet that values dual-source supply over the last increment of host-GPU bandwidth, this is the pragmatic default.

The GPU:CPU ratio is the shortest description of system composition, and every ratio must name its denominator. A common training host carried eight accelerators and two CPU sockets: 8 per host, or 4:1 counted per CPU package. Its CPU's job was to keep the data pipeline full and stay out of the way. The host did almost no application compute: pre-training is GPU-bound on collectives and matmul, and the CPU's part is loading shards, augmenting data and launching kernels. The rational move was to minimize host spend and give the dollars and watts to the GPUs.

Even host memory got cut. Neocloud reference builds trimmed training-server RAM from 2 TB to 1 TB because the trainer never used it. That was one host-BOM cut, alongside a smaller CPU and the removal of two BlueField-3 DPUs. SemiAnalysis's October 3, 2024 example priced those changes at $270k → $256.4k per H100 server, a $13.6k saving in its 1,024-GPU scenario. A new quote must still preserve any DPU isolation or networking function you need.

Agentic inference and RL break the constant. The work that defines an AI agent is decomposing a request into sub-tasks, scheduling them, routing tool calls, passing state between sub-agents, running retrieval, executing code in sandboxes and judging whether the goal is met. That work is overwhelmingly host-side and control-heavy: branchy, latency-sensitive, and it does not vectorize onto a GPU. RL training compounds it, because every action an agent takes in a rollout must be executed and scored, and that loop runs on the CPU. As agents proliferate, the host stops being a feeder and becomes a co-processor doing sustained work, so the ratio that kept GPUs the scarce resource per node no longer holds.

Count cores instead of packages and the gap between platforms widens. In a DGX B300, two 64-core Xeon 6776P chips serve 8 GPUs, which leaves 16 host cores per GPU. In GB200/GB300 NVL72, one 72-core Grace serves 2 GPUs, so each gets 36. Vera Rubin NVL72 keeps one CPU per 2 GPUs but moves to the 88-core Vera: 44 cores per GPU, and up to 750 GB of host LPDDR5X behind each. AMD's Helios puts 4 GPUs on each EPYC package; built with the 256-core Venice part that would be 64 cores per GPU, but AMD has not named the SKU. The discrete path can buy cores too. Put two 192-core EPYC 9965 sockets in an 8-GPU host and each GPU gets 48 cores, more than Vera Rubin's 44, at 500 W of CPU TDP per socket. Those are compact Zen 5c cores, though, so a higher count is not the same as more host throughput per GPU. Line up platforms by host cores, core type and host memory per accelerator, and ignore package count.

Forecasts point the same way. TrendForce's 14 April 2026 agentic-deployment forecast shifts CPU:GPU ratios from 1:4–1:8 toward 1:1–1:2; it does not say how it counts, so it sets no CPU-package count inside a node. TrendForce News reported that Intel's CEO said on the Q1 2026 earnings call (April 2026) that the mix had moved from about 1:8 to about 1:4 and could reach 1:1 or tilt further toward CPUs. Neither figure names a denominator, the Intel remark gives no workload profile, and a training fleet and an inference fleet may land on different ratios. Arm makes the same argument for the market as a whole: as agents proliferate, orchestration, tool execution and state handling consume host cores while the GPU waits on them. That is a market projection, not a sizing rule for your rack.

The consequence is concrete. A fleet scoped at 4:1 for DGX/HGX B300 training nodes (NVL72 racks are 2:1) and then repurposed for agentic serving will be CPU-starved once orchestration and tool execution need more host than it shipped with. Queues form ahead of the GPUs, tail latency blows past the SLO, and accelerators sit underutilized behind a host bottleneck. The fix is more CPU throughput and memory for that work, from larger hosts or a separate CPU pool, with its power budget in the fleet design. NVIDIA designed Vera explicitly "for agents" for this reason: 88 custom Olympus cores, 176 threads via spatial multithreading, and up to 1.5 TB of LPDDR5X at ~1.2 TB/s, a host built to do real work, not just feed.

Host memory is the third axis of composition, and it tracks the host-attach fork closely. The coherent superchips use LPDDR5X, the low-power DRAM from the mobile world: soldered on Grace, in detachable SOCAMM modules on Vera. The reasons are bandwidth-per-watt and density. NVIDIA's GB200 profile pairs Grace with up to 480 GB of LPDDR5X at up to 512 GB/s, and the host power setting belongs to the OEM profile you select. On NVIDIA's own numbers, Grace's memory moves up to about 500 GB/s on roughly 16 W, a fifth of what 12 channels of ordinary DDR5 would burn (tuning guide; vendor figure). Vera lifts capacity to up to 1.5 TB and bandwidth to ~1.2 TB/s. NVIDIA says that is double the bandwidth at half the power of the memory in general-purpose CPUs (vendor claim, March 2026). In a power-bound rack, where every watt on host memory is a watt not spent on accelerators, that efficiency is the point.

The price depends on the generation, and the two are not the same bargain. Grace's LPDDR5X is soldered: capacity is fixed at manufacture and a memory fault is a package fault, with no module to pull. Vera changes that. NVIDIA packages its LPDDR5X in SOCAMM modules specified as detachable, field-replaceable and capacity-upgradable. JEDEC has since standardized the module as SOCAMM2 (JESD328 version 1.00, June 2026), and SK hynix began mass-producing a 192 GB SOCAMM2 for Vera Rubin in April 2026. Ask your OEM which suppliers' SOCAMM2 modules it accepts as spares. Serviceability and error protection are separate axes anyway. Grace documents ECC and channel sparing, so LPDDR is no reason to assume a host lacks server-grade protection. Check which RAS features and which repair unit each SKU actually supports.

Discrete hosts run socketed DDR5 and, more and more, MRDIMM (multiplexed-rank DIMMs). Intel's Granite Rapids takes MRDIMM-8800, where AMD's EPYC Turin takes DDR5-6400. AMD announced its 6th Gen EPYC (Venice) in July 2026, with flagship parts expected from Q4 2026: up to 256 cores and up to 16 MRDIMM channels at 12.8 GT/s. Intel's next P-core Xeon, Diamond Rapids, is its equal on paper: up to 256 cores, 16 memory channels running at the same 12.8 GT/s, and 128 lanes of PCIe Gen6 and CXL 3.0 on Intel 18A-P. Intel disclosed it at Hot Chips on 24 August 2026 and gave no ship date. Socketed memory buys composability. You size capacity to the workload (multiple terabytes for embedding tables, large KV-cache or in-memory retrieval indices), you get full server-grade RAS, and you can service or upgrade memory without touching the accelerators. Vera's SOCAMM narrows the gap on serviceability, since its modules are field-replaceable and upgradable, but not on capacity: up to 1.5 TB per CPU against multiple terabytes per socket.

The cost is power and density. Registered DDR5 burns more per gigabyte than LPDDR5X, and the DIMM slots take board area and a slice of the rack's thermal and electrical budget. The decision mirrors the host-attach fork. Take a Grace superchip and you take its soldered LPDDR5X bargain. Vera's SOCAMM changes the repair unit. Take the discrete host and you can spend power to buy capacity, bandwidth, RAS and serviceability on your own terms.

Composition is not a single answer. A real AI operation runs more than one workload shape, and the host-attach model, GPU:CPU ratio and host memory that suit a pre-training trainer may differ from those of an agentic-inference fleet. So system composition is plural: compose distinct node and rack SKUs for the workloads that dominate your draw.

A training SKU may accept the coherent superchip ratio as shipped (2:1 on the GB200/GB300 NVL72 generation), light host application compute (profile it to be sure), host memory sized to the data pipeline instead of the model, and every spare watt routed to accelerators. An agentic-inference SKU may need a larger host pool: many cores for orchestration and tool execution, large host memory for KV-cache and retrieval state, and a GPU:CPU ratio or separate CPU pool sized from CPU-seconds, memory and queueing data. A batch-inference SKU sits between them, throughput-bound and tolerant of a thinner host. Forcing one composition across mismatched profiles leaves you paying for unused host memory on the trainers while starving the agentic fleet of CPU. This is the same plurality the archetype framework demands at facility scale (Chapter 1.1); here it lands inside the node.

CPU demand = 100 requests/s × 0.20 CPU-s/request = 20 busy cores. Provision ceil(20/0.70) = 29 cores, so the 32-core pool passes and 16 cores fail. Memory = 30 × 2.0 GB + 40 GB = 100 GB, within 128 GB. The 16-core service ceiling is 16 × 0.70/0.20 = 56 requests/s; the 32-core ceiling is 32 × 0.70/0.20 = 112 requests/s. At 120 requests/s, ceil(120 × 0.20/0.70) = 35 cores: the pool must grow or the CPU work must fall.

In this base case, add the CPU pool, then trace queues, scheduler locality and request latency before adding GPUs; NVIDIA's CUDA best-practices guide covers the profiling. At or below 56 requests/s the thin host clears this CPU screen, and the added pool may buy no service benefit.

One more decision sits underneath the rest: how much of system composition do you let the vendor make for you. The rack-scale coherent platforms (GB200/GB300 NVL72, and Vera Rubin NVL72, shipping since June 2026) ship as integrated units. The CPU:GPU ratio, host CPU, host memory and interconnect are fixed at the factory, so you buy a rack, not a node. Integration fixes the ratio whether the host is Arm or x86. AMD's Helios makes the point on x86: its 18 EPYC Venice CPUs serve 72 MI455X GPUs, 4 per CPU package, and in a single-vendor rack that split is as fixed as an NVL72's.

That is valuable. The integration risk of welding 72 GPUs, 36 CPUs, NVSwitch trays and a liquid-cooling loop into one coherent domain is enormous, and the vendor has absorbed it. But the GPU:CPU ratio, host ISA and host memory are no longer yours to tune. If your agentic workload wants a fatter host than the package ships, the lever inside the rack is a different SKU, not a different host.

The lever outside the rack is to stop treating the accelerator-attached host as the whole CPU tier. Tool execution, RL environments, preprocessing and orchestration can run on a network-connected CPU pool sized independently, and NVIDIA positions Vera as standalone CPU infrastructure as well as a GPU host. A fixed CPU:GPU ratio inside a superchip does not fix the CPU:GPU ratio of the service. Latency, data locality, isolation and transfer volume decide that.

If you rent that tier rather than own it, it is increasingly hyperscaler Arm. AWS's Graviton5 packs 192 Neoverse V3 cores per chip and has been generally available since June 2026, and Amazon says Meta is deploying tens of millions of Graviton cores for CPU-heavy agentic work such as orchestration and code generation. Microsoft's Cobalt 200 and Google's Axion fill the same slot on their clouds. Price the rented CPU tier against an owned EPYC, Xeon or Vera pool before you assume it must sit beside the GPUs. The catch is distance and instruction set. Every tool call is a network round trip to the accelerators, and it crosses a wide-area link whenever the GPUs sit in your own halls or with another provider rather than in the same cloud region. Sandbox images and tool code also need Arm builds.

The owned pool is now a rack product. NVIDIA's Vera CPU rack holds 256 liquid-cooled Vera CPUs on its MGX architecture, and NVIDIA claims it can run over 22,500 concurrent sandboxed environments (vendor claim). AMD's Helion blueprint goes to 32,768 6th Gen EPYC cores in a CPU-only rack, air- or liquid-cooled, with availability planned for November 2026. AMD plans to contribute the Helion specification to OCP, and Supermicro calls the rack Open Rack V3 (ORV3) compliant (AMD release, 12 October 2026). Arm now sells its own chip for this tier, the AGI CPU, with 272 Neoverse V3 cores per two-chip blade and Meta as lead customer; systems have been orderable since March 2026. Supermicro showed rack designs on ORV3 (128 CPUs, liquid-cooled) and Open Rack Wide (336 CPUs). Arm's figure of 1.99x modeled performance per rack against AMD EPYC 9755 comes from Signal65 modeling of seven agentic workloads (Arm, 2026-10-15).

Post-training shows why the pool matters. MLPerf announced an RL post-training benchmark in September 2026, with its first round in Training v6.1. Its reference run used 64 nodes of four Blackwell Ultra GPUs each, divided into trainer and rollout workers, and the agent sandboxes ran on those same nodes. There, hundreds of rollouts at once fight over CPU, memory and storage, which makes the host tier part of time-to-train. Choose where the sandboxes run, on the GPU hosts or on a CPU rack, and if it is a rack, budget its power and cooling separately.

On the discrete path, composition stays in your hands. You build the node, choosing AMD or Intel, core count, socketed memory, GPUs per host and the ratio, and you own both the flexibility and the integration risk. For a fleet with heterogeneous workloads and a procurement strategy that refuses to single-source, this path lets composition track the workload. In 2026 most large operators run both: bundled coherent racks where the workload needs the coherent domain (training, memory-hungry inference), and composed discrete nodes where composability and dual-source supply matter more. The question is which workloads belong on which path, which returns you to the fleet-plurality discipline above. Chapter 7.11 covers procurement and fleet composition.