Topology-Aware & Rack-Scale Scheduling

Chapter 10.1 described the scheduling plane as the control loop that turns a fleet of accelerators into a service. This chapter covers the hardest constraint that loop has to respect on 2026-era hardware: the network is not flat. Any two GPUs in a modern AI cluster reach each other over one of two very different kinds of path, and the gap between them is about an order of magnitude. NVLink links GPUs at terabytes per second inside a node and, on rack-scale systems, across the whole NVLink domain, as if all of them sat on one giant board. The 2026 volume rack, the GB300 NVL72, keeps the layout of the GB200 NVL72 it replaces: 72 GPUs in 18 compute trays, with 130 TB/s of aggregate NVLink bandwidth shared across them. Once a collective leaves that domain, it drops onto the scale-out fabric, which has its own failure and congestion regime. Each GPU gets an 800G NIC on GB300 (400G on GB200), or 1.6 Tb/s on Vera Rubin. Per GPU, NVLink beats that NIC by roughly an order of magnitude: about 18× on GB200, 9× on GB300 and 7.5× on Vera Rubin. A scheduler that ignores this hierarchy will cheerfully place a tightly coupled job with eight GPUs in one rack and eight in the next. The job runs, at a fraction of its rightful throughput, forever, with no error logged.

Topology-blind versus topology-aware placement is the choice, and choosing blind costs goodput at every step of every job for the life of the run. This chapter walks the bandwidth cliffs and how to discover them. It then covers the two production defenses, Slurm block scheduling with IMEX and Kubernetes ComputeDomains over Dynamic Resource Allocation, how to map a parallelism strategy onto the fabric so the right collectives stay local, and the fragmentation problem that makes the rack, not the node, the natural unit of allocation on rack-scale systems.

Three numbers drive everything in this chapter. NVLink (scale-up, inside the coherent domain): NVLink 5 on Blackwell delivers 1.8 TB/s bidirectional per GPU, and an NVL72 rack aggregates to 130 TB/s. Scale-out NIC (inter-node, over InfiniBand or RoCE Ethernet): GB200 gives each GPU one 400 Gb/s ConnectX-7 path (50 GB/s per direction), and GB300 gives each GPU 800 Gb/s through one ConnectX-8 (100 GB/s per direction), which runs as two 400G planes on Ethernet. Against NVLink 5 at 900 GB/s per direction, the platform-scoped ratio is roughly 18× on GB200 with ConnectX-7 and 9× on GB300 with ConnectX-8. So the most bandwidth-hungry collectives must fit inside the scale-up domain, and the scheduler decides whether they do.

Inside the node sits a third link, the host link from the GPU to its CPU or NIC. On Grace-based GB200 and GB300 superchips, NVLink-C2C connects each GPU to the Grace CPU at 900 GB/s bidirectional. That works out to about 450 GB/s in each direction, about seven times what PCIe Gen5 x16 carries (NVIDIA, August 2025). Each Grace serves two GPUs, so when both spill at the same time they share that one CPU's memory. On x86 HGX nodes such as H100 and B200 the CPU sits behind PCIe Gen5 x16 at ~64 GB/s each way. KV-cache spill and host staging ride this path, so offloading to host memory is a real bottleneck on x86 HGX and much less of one on a Grace-based rack (Chapter 7.8 covers the coherent host link). The NIC hangs off PCIe on both: ConnectX-8 on GB300 has a PCIe Gen6 x16 host interface, ~128 GB/s each way.

Bandwidth does not degrade gracefully as a job spreads. It steps down abruptly at each boundary. A tensor-parallel all-reduce that fits in one NVLink domain runs at NVLink speed. Split 8-and-8 across two domains, the same all-reduce runs at NIC speed for the cross-domain portion and stalls every other rank waiting on it. NVIDIA's own Slurm guidance says that when a job crosses the NVLink-domain boundary, performance "drops sharply", which is why the block plugin treats the domain as a hard constraint rather than a best-effort preference. The failure is silent: the job completes, the loss curve descends and nothing alarms, but step time rises by whatever share of the cross-domain collective cannot be overlapped with compute, and that loss compounds across the entire run. Before you set a locality policy, measure that loss: run the same step on an in-domain placement and on a split one.

A scheduler can only defend a topology it can see, and discovery is where most topology-aware deployments fail without anyone noticing. There are two postures, and choosing between them is a trust decision. Trusted discovery reads the fabric from labels the platform asserts. NVIDIA exposes an NVLink clique identifier (the nvidia.com/gpu.clique node label in Kubernetes, or equivalent block definitions in Slurm's topology.yaml) that tells the scheduler which nodes share a coherent NVLink partition. PCIe and NUMA affinity come from the driver and from the node's hardware topology. Scale-out topology (which leaf switch a node hangs off, which rail a NIC belongs to) comes from cabling databases, LLDP, or a topology file the operator maintains. Trusted discovery is fast, and it is how production schedulers run day to day.

Verified discovery distrusts the labels and measures. At node admission and after every repair, run point-to-point bandwidth probes (nvbandwidth, NCCL all-reduce and all-gather benchmarks, p2pBandwidthLatencyTest) and compare the results with the expected matrix for the asserted topology. The labels can be right while the physical link is degraded: a marginal NVLink lane that has fallen back to a lower width, a transceiver running hot, a badly seated cable. The label still says "one clique, full bandwidth" while the hardware delivers half, and a scheduler that trusts it places a tightly coupled job onto a cliff it believes isn't there. Gate nodes into the schedulable pool only after they pass the bandwidth matrix, not merely after they boot, so topology-label drift cannot eat your fleet's goodput. Chapter 10.5 lists the NVLink bandwidth gate among its acceptance tests. Rerun it after every repair, before the node clears the return-to-service step in Chapter 10.7; the scheduler acts on the result.

Slurm is an established scheduler for large training fleets, and its answer to rack-scale topology is block scheduling (the topology/block plugin, hardened for NVL72 in 2025). You declare in topology.yaml that a set of nodes forms a block matching one NVLink domain. For the cited GB200 NVL72 profile that is 18 four-GPU compute nodes carrying 72 GPUs. The scheduler then uses the block as a locality unit, and locality does not require reserving every GPU in it. A job requesting up to 18 nodes is consolidated into a single block, provided the required TP/EP groups are mapped within that block and the selected allocation policy preserves their locality. A job larger than one block is kept within the smallest group of blocks that holds it rather than smeared arbitrarily, so cross-domain traffic stays confined to the parallelism dimensions that tolerate scale-out bandwidth.

Two refinements make this practical rather than rigid. The --segment argument lets a user declare the atomic group size the job needs. For example, --segment=4 lets a 12-node job be split into three 4-node segments that the scheduler can pack flexibly across blocks while each segment keeps its locality. This is the release valve for fragmentation: instead of demanding one contiguous 12-node hole, the scheduler can fill three smaller holes and lift utilization without abandoning topology. The second refinement is the switch/nvidia_imex plugin (Slurm 24.05+), which manages IMEX channels, the driver-level memory-export/import access control that lets GPUs in a multi-node NVLink domain address each other's memory. Slurm provisions and tears down the IMEX channel per job, so authorized memory import and export follow the job. Verify isolation with your exact driver and job teardown rather than treating channel provisioning as whole-platform security. Block scheduling without IMEX management is a security and correctness gap on shared NVL72 hardware, not merely a tuning omission.

Kubernetes arrived at rack-scale topology from the opposite direction. Its original device-plugin model exposed a GPU as a flat, countable, location-free resource, nvidia.com/gpu: 8, which alone does not express cross-node NVLink domain membership. The fix is Dynamic Resource Allocation (DRA), the API that lets a workload request a structured, parameterized resource instead of an opaque count. NVIDIA's DRA driver adds the ComputeDomain abstraction on top. A ComputeDomain represents reachability between the distributed workers of a multi-node job: when its pods are scheduled, the platform dynamically creates the IMEX domain that lets those pods' GPUs address one another's memory over NVLink, and tears it down when the job ends. The IMEX channel surfaces inside each container as a device file, so CUDA or NCCL can use the authorized memory-export path; device preparation and a real transfer still have to pass. All of that describes the default, driver-managed mode. The same driver, now a Kubernetes SIGs project (kubernetes-sigs/dra-driver-nvidia-gpu, v0.5.x), has a second, host-managed mode. It sits behind the HostManagedIMEXDaemon feature gate and relies on an IMEX service that runs permanently on the host. Every ComputeDomain on that host IMEX domain shares channel 0, so this mode isolates domains and nothing finer, and a Ready ComputeDomain says nothing about the host service's health. Find out which mode your cluster runs before you count on per-job memory-import isolation when co-tenant jobs share an NVLink domain.

Two constraints shape the design. First, placement still needs an affinity rule: pods must be steered onto nodes that share an NVLink partition by matching the nvidia.com/gpu.clique node label, or the ComputeDomain spans a boundary it cannot bridge. The clique label is the Kubernetes analogue of the Slurm block. Second, and this is the sharp edge of the 2026 implementation, the deployed ComputeDomain driver can impose one domain per node. That is an allocation constraint, not the number of channels IMEX can represent. If a ComputeDomain claims only part of a node's GPUs, the remaining GPUs on that node cannot join a different ComputeDomain. This can favor whole-node allocation when different jobs need different domains, but it does not require reserving the whole rack. Record domain ownership separately from the GPU count reserved and billed. The two install paths have different floors. The standalone DRA driver runs ComputeDomains on Kubernetes 1.32 or later, with the DynamicResourceAllocation feature gate switched on for 1.32 and 1.33 (on by default from 1.34). The NVIDIA GPU Operator 26.7 managed workflow requires Kubernetes ≥1.34.2, NVIDIA driver R580+, CDI and a fresh GPUCluster installation without ClusterPolicy, and it offers no in-place migration from standalone DRA. ComputeDomains additionally require supported Multi-Node NVLink hardware and correctly owned IMEX services. If node preparation cannot be confirmed, admission is blocked; nothing falls back on an assumption.

Topology-aware scheduling is half the bargain. The other half is the job declaring its parallelism so the scheduler can put each dimension on the right tier. 3D (and 4D) parallelism uses four collective patterns with very different bandwidth appetites, and the art is matching appetite to tier. Tensor parallelism (TP) all-reduces activations on every layer. It is the most bandwidth-hungry pattern and must live inside the NVLink domain. Expert parallelism (EP) for MoE models does all-to-all token routing. Wide EP (EP32 and beyond) is exactly what a 72-GPU NVLink domain unlocks, because the all-to-all stays on NVLink instead of hitting the scale-out fabric. Pipeline parallelism (PP) passes activations point to point between stages: low volume, latency-sensitive, and tolerant of crossing the domain boundary. Data parallelism (DP) all-reduces gradients once per step: high volume but infrequent and overlappable, so it is the canonical dimension to push onto scale-out.

The default rule that follows is TP and EP inside the scale-up domain; PP and DP across the scale-out fabric. For EP this is a preference, not a law. DeepSeek-V3 trained with 64-way expert parallelism across eight H800 nodes over InfiniBand, using node-limited routing and communication/compute overlap to keep the all-to-all off the critical path. Judge cross-node EP on measured routing volume, expert placement, overlap and available network bandwidth rather than ruling it out. A correctly mapped GB300 NVL72 job fits TP+EP within the 72-GPU domain, runs PP across racks, and lets DP all-reduce ride the rail-optimized fat-tree, where SHARP in-network reduction can absorb it. VR200 NVL72, shipping since June 2026, keeps that 72-GPU domain at higher scale-up bandwidth. Rubin Ultra Kyber (NVIDIA targets H2 2027; SemiAnalysis reported 2028 in July 2026) widens the domain itself. So the TP/EP fit is a mapping you recheck each generation, not a constant. Get the mapping backwards, with DP inside the domain and TP across racks, and you have inverted the bandwidth hierarchy: the cheap-to-cross dimension hogs NVLink while the expensive-to-cross dimension chokes on NIC bandwidth. That is why the scheduler needs the parallelism shape and not just a node count, so block and segment sizes line up with the TP/EP group instead of slicing through it. The framework that emits this shape is the subject of Chapter 10.8, and the fabric that carries it is Chapter 8.5.

Topology awareness creates a tension that flat scheduling never has: the better you defend locality, the worse your bin-packing gets. If a job can only land inside one NVLink domain, a domain with 6 of 72 GPUs busy has 66 GPUs that are useless to any job needing more than 66 NVLink-local. They are stranded by fragmentation, not by lack of capacity. Flat schedulers never see this because they place a job anywhere; they pay in silent cliffs instead. Topology schedulers make the fragmentation visible and force a policy choice.

There are three levers, and a 2026 fleet uses all three. Accept the strand: keep domains whole, leave the 66 GPUs idle until a job that wants them arrives, and price the lost utilization as the cost of full-bandwidth placement. That is the right call when goodput per job dominates, as in large training. Defragment: use preemption and migration to compact small jobs out of partially used domains and free whole domains for large ones. It works, but it is expensive on synchronous jobs because preemption forces a checkpoint and restart. Kubernetes adds a trap here. NVIDIA's GPU Operator 26.7 documentation lists a limitation: Kubernetes cannot preempt for DRA resources, so a higher-priority pod in kube-scheduler will not evict a lower-priority one to free a DRA GPU claim. On a ComputeDomain platform, compaction has to come from the batch layer evicting whole workloads (Kueue preemption is one example) or from an explicit drain. Confirm that your batch scheduler actually handles DRA claims before you plan around it. Relax locality with segments: let jobs declare smaller atomic groups (Slurm --segment, or sizing the ComputeDomain below a full rack) so they fit the holes you have, and accept scale-out traffic across the segment boundary. In Kueue, the equivalent of Slurm --segment is a slice annotation, which sets a required topology for each slice of N pods.

On rack-scale hardware the rack, meaning the NVLink domain, becomes the natural unit of allocation, not the GPU and not the node. Placement is what wants the domain as its unit. Any policy that subdivides one has to reckon with the one-ComputeDomain-per-node limit, the bandwidth cliff and the fragmentation it creates. Billing, drain and repair units are separate decisions. A job that fits inside a domain can take a subset of its nodes, and NVIDIA's own GB200 block-scheduling guidance places eight- and twelve-node jobs inside a single block. Reserve and price a whole rack where a workload or a capacity guarantee needs the whole rack, not by default.

The same mistakes recur because each one treats the GPU as fungible when the fabric says it is not:

• Flat scheduling on rack-scale hardware. Exposing nvidia.com/gpu: 8 with no clique or block awareness, then wondering why a 16-GPU job that landed 8-and-8 across two racks takes noticeably longer per step than the same job inside one domain. The hardware is non-blocking inside the domain and an order of magnitude slower across it, and a location-free resource model cannot see the difference, so it cannot avoid the cliff.
• Topology without gang scheduling. Placing the right pods on the right nodes but admitting them incrementally, so synchronous jobs deadlock holding partial allocations. Locality and all-or-nothing admission are a pair; shipping one without the other trades a silent cliff for an outright stall.
• Inverted parallelism mapping. Putting data-parallel all-reduce inside the NVLink domain and tensor-parallel across racks spends the scarcest bandwidth on the dimension that least needs it. Follow the appetite: TP/EP local, PP/DP remote.
• Trusting labels over measurements. Letting a node into the schedulable pool because it booted and its clique label is present, without verifying the NVLink bandwidth matrix, and so placing jobs onto a degraded domain the scheduler believes is healthy.
• Subdividing domains under the one-ComputeDomain-per-node limit. Designing a Kubernetes multi-tenant scheme that hands partial nodes to distinct ComputeDomains, then discovering that the one-ComputeDomain-per-node constraint strands the remaining GPUs. On rack-scale NVLink hardware, align allocations to domain boundaries and keep each ComputeDomain whole at node granularity. Sell and bill the whole rack only where the workload or the capacity guarantee requires it.

Reserve the scale-up domain when the job’s measured TP/EP communication needs it; otherwise pack compatible jobs within your isolation and SLO limits. Reserving every rack wastes paid capacity, while splitting the wrong collective buys nominal utilization at the price of a longer completion time. Keep the rank map and the measured crossover beside the allocation policy.