Orchestration Architecture & the Scheduling Plane

You can build a flawless data center, with perfect PUE, a non-blocking fabric and golden-image nodes, and still waste paid GPU-hours because the layer that places work on the hardware is badly designed. The orchestration stack is where megawatts of energized, depreciating silicon become either useful tokens or idle heat. It is the most software-defined decision in the facility and, perversely, the one most often inherited by accident. The HPC team brings Slurm because that is what they know. The platform team brings Kubernetes because that is what the rest of the company runs. The two never reconcile until a frontier training run and an inference fleet fight over the same GPUs through two schedulers that cannot see each other.

The architecture of that layer breaks into three planes that every fleet must design coherently: the workload scheduler, the node runtime and the fleet/control plane. Then come the choices. Slurm versus Kubernetes. The batch schedulers that add queue policy and group scheduling on Kubernetes (Volcano, KAI, Run:ai). The Slinky hybrids that try to run both at once. And Dynamic Resource Allocation (DRA), the substrate change that finally lets Kubernetes describe a GPU as something richer than an opaque countable unit. Each is scored by what it costs downstream in goodput, utilization, operational headcount and lock-in. The deeper mechanics (topology-aware placement, multi-tenant isolation, the node software stack, autonomous recovery) each get their own chapter in Part 10; this one is the map that says which plane owns which problem. Assign each device one allocation owner: two schedulers may share inventory visibility, but may not independently grant the same GPU.

A working AI fleet runs three loosely coupled control loops. Each works on a different timescale and, ideally, belongs to a different on-call rotation. Conflate them and you get the recurring failure mode of GPU operations: a node-health problem that looks like a scheduler bug, a fabric event that looks like a job failure, a driver mismatch that surfaces as a mysterious collective hang three layers up.

• The workload-scheduling plane answers which job runs on which GPUs, when. It owns queues, priority, fair-share, gang and co-scheduling, preemption and backfill. This is Slurm, or Kubernetes plus a batch scheduler. It works on a timescale of seconds to minutes and is the plane this chapter mostly covers.
• The node runtime plane answers can this node actually run the work correctly. It owns the GPU driver, CUDA/ROCm, NCCL/RCCL, firmware, the container runtime and device exposure (device plugin or DRA driver). It works on the timescale of a node's lifecycle and is treated canonically in Chapter 10.4. The scheduler trusts this plane. When that trust is misplaced, say by one node with a mismatched NCCL or a degrading NVLink, the scheduler happily places a 1,024-GPU job on the bad node and the whole gang hangs.
• The fleet/control plane answers is the fleet healthy, and how do we keep it that way. It owns provisioning and bring-up (Chapter 10.5), observability and health telemetry (Chapter 10.6), and autonomous fault detection and recovery (Chapter 10.7). It works over hours to days, and products such as NVIDIA Mission Control and Base Command Manager live here.

The seam that matters most is between the workload plane and the fleet plane, and one verb mediates it: cordon-and-drain. When the fleet plane detects a sick node, via DCGM, an XID error or a fabric counter, it must be able to tell the workload plane to stop scheduling onto it and to migrate or checkpoint the work already there, without a human in the loop.

On Kubernetes with DRA-managed GPUs, the cordon no longer has to cover the whole node. DRA device taints became stable in 1.37 (August 2026). The health loop, or an admin through a DeviceTaintRule, can stop new scheduling onto a single GPU and evict only the pods using it. Fleets still on the device plugin are limited to node-level cordons. Decide which faults quarantine one device and which drain the node. Either way, a synchronous job dies when it loses one rank. Inference replicas on the node's other GPUs, though, keep serving (Chapter 10.7).

Meta's Llama 3 405B pre-training, on up to 16,384 H100s, shows the cadence this seam must absorb. In a 54-day snapshot the job logged 466 interruptions, 419 of them unexpected, or about one every three hours. Significant manual intervention was needed only three times. That is one job's observed cadence, not a per-GPU MTBF or a scaling law, so size your recovery target from the whole-job interruptions you measure on your own fleet.

Two schedulers dominate the workload plane, and they were built for different problems. Slurm comes from supercomputing. It assumes a long-lived, tightly coupled batch job that wants all of its nodes at once, communicates over a fast fabric, runs to completion or fails as a unit, and is submitted by a researcher who thinks in sbatch scripts and partitions. Kubernetes comes from web services. It assumes a fleet of independent, restartable, horizontally scalable containers behind a load balancer, declaratively reconciled toward a desired state, with rolling upgrades and self-healing as first-class verbs. Neither was designed for the other's workload, and the friction at the boundary is the central operational problem of AI orchestration in 2026.

A widely repeated trade-press split puts roughly 70% of AI/HPC scheduling on Slurm, ~20% on Kubernetes, ~10% in-house. Read it as a direction, not a market share: no published sampling frame says whether it counts organizations, clusters, installed GPUs or job-hours. Training is synchronous, gang-scheduled and fabric-bound, and it gravitates to Slurm because gang scheduling is native. sbatch -N 256 gets you all 256 nodes simultaneously or the job waits, which is the all-or-nothing semantics a synchronous all-reduce job needs. Inference is bursty, loosely coupled, autoscaled and sits behind a service mesh, so it gravitates to Kubernetes, which was built for that. One operator observation points the same way. CoreWeave relayed a SemiAnalysis customer observation in April 2025: for inference, roughly 90% of GPU-cloud customers prefer Kubernetes, and for training about 50% use Slurm. It carries that sample's limits and is not a census. The two workloads pull toward opposite schedulers, which is why so many fleets end up running both.

The architecture question is whether your workload mix justifies one plane or two and, if two, whether you federate them or converge them. The downstream cost is real: headcount, stranded capacity at the boundary, and the blast radius of a control-plane outage.

Read the table plainly: Slurm fits an HPC batch operating model, Kubernetes fits an existing container service platform, and a hybrid buys access to both at the price of complexity you must staff for. Test those fits against your workload and controller-failure cases. The common mistake is choosing the hybrid by default ("we'll run both, best of both worlds") without budgeting for what that means: two schedulers, two notions of a node, two quota systems, two failure surfaces, and two on-call rotations that must agree on what "this node is unhealthy" means. Convergence tooling can coordinate the handoff, but it is not free. Treat it as a deliberate investment, justified only when the workload mix truly demands both planes on the same physical fleet.

Vanilla Kubernetes schedules pods independently, which is fatal for a distributed training job. If a 64-pod job gets 60 pods placed and then blocks waiting on the last 4, those 60 GPUs sit idle holding the gang hostage, and two such half-placed jobs can deadlock each other indefinitely. This is the partial-gang / resource-deadlock problem, and the Kubernetes batch-scheduler ecosystem exists to solve it. Each of the three main schedulers below uses gang (co-)scheduling, which places all pods of a job atomically or none, and then layers AI-specific scheduling on top. A fourth project, Kueue, governs admission rather than placement.

• Volcano is the CNCF-incubating batch system for Kubernetes and the most established open-source gang scheduler. It brings gang scheduling, queue and fair-share, preemption, backfill and topology-aware plugins, and it integrates with the training operators (PyTorch, MPI, Ray). It is the safe, vendor-neutral choice if you want to avoid lock-in.
• KAI Scheduler is NVIDIA's Kubernetes-native AI scheduler, open-sourced under Apache-2.0 in April 2025 as the engine extracted from the Run:ai acquisition. CNCF accepted it as a Sandbox project in December 2025, and the code now sits under LF Projects governance. It adds gang scheduling, hierarchical fair-share quotas, bin-packing and spread strategies, GPU sharing and fractioning, and native DRA and topology awareness. Open-sourcing it pushed these capabilities toward a commodity baseline.
• Run:ai is the commercial platform (now NVIDIA, bundled into NVIDIA AI Enterprise) layered on KAI: fractional GPU, policy and quota governance, a multi-tenant UI, and integration with the broader NVIDIA fleet stack. You buy it when you want the managed product and the support contract rather than the assembly job.
• Kueue is a Kubernetes SIG Scheduling project that decides when a workload may start rather than which node each pod lands on. It holds jobs in queues, reserves quota for a whole workload before admitting it, lends unused quota across the ClusterQueues of a cohort, and preempts between tenants by policy. Its all-or-nothing mode with ready pods evicts and requeues a workload whose pods do not all start before a timeout. Its Topology Aware Scheduling, which Kueue's documentation listed in October 2026 as beta and on by default since v0.14, also lets Kueue choose where the whole workload runs at admission. A required topology holds every pod inside one rack or block. A preferred topology lets the pods spread over several domains when no single one has room, so a job that must stay within one rack needs the required mode. Kueue then pins the pods to the domains it chose with node selectors, and within them kube-scheduler or whichever scheduler the pods name still binds each pod to a node. Kueue can admit multi-node training and inference packaged as JobSet or LeaderWorkerSet workloads. Chapters 10.2 and 10.3 build on it.

The decision is a build-vs-buy in miniature. Volcano is the open, neutral baseline you operate yourself. KAI is the more AI-specific engine. It is now a CNCF Sandbox project, but its maintainers and roadmap remain NVIDIA-led, and Sandbox is CNCF's earliest maturity tier (Volcano is Incubating). Run:ai is the supported product on top of KAI. KAI or Run:ai gives you fractional-GPU and quota machinery out of the box, which matters the moment you have multi-tenancy (Chapter 10.3). Volcano keeps you maximally portable across vendors and avoids deepening NVIDIA platform dependence at the orchestration layer. Kueue, by its own documentation, leaves pod placement to kube-scheduler, while Volcano and KAI each run queues and quotas of their own, so stacking Kueue on either means deciding which layer owns quota, or tenants hit two sets of limits.

For a decade, Kubernetes treated a GPU as a countable integer: nvidia.com/gpu: 8, an opaque unit, all alike, node-local. That model breaks the moment GPUs stop being fungible. You may care which GPUs you get (two on the same NVLink island), in what shape (a specific MIG geometry), with what attributes (≥40 GB free, a particular topology), or whether pods share them. Dynamic Resource Allocation (DRA) replaces counting with claiming. A workload describes the properties of the devices it needs in a ResourceClaim, and a vendor-supplied DRA driver plus the scheduler resolve the actual allocation. It is the same move PersistentVolumeClaims made for storage, applied to accelerators.

The timing is the news. DRA graduated to GA in Kubernetes 1.34 (27 August 2025), so the core resource.k8s.io APIs are stable. Partitionable devices and consumable capacity reached beta in Kubernetes 1.36, and extended-resource support and device taints reached stable in 1.37. MIG, time-slicing and fractional-GPU support remain driver-specific. Downstream distributions followed (DRA GA in OpenShift 4.21, February 2026). DRA changes what the scheduling plane can express natively: attribute-based selection ("2 GPUs on the same node interconnected by NVLink, min 40 GB VRAM"), driver-backed fine-grained partitioning (MIG, time-slicing, fractional GPU) without a static 1:1 pod-to-GPU mapping, and network-attached devices. That finer control can let the scheduler pack and share devices more tightly than whole-GPU counting does. Measure the gain on your own workload mix before you book it as a goodput lever.

Whether to adopt DRA now depends on your driver more than on Kubernetes. Since 1.37 (August 2026), DRA's extended-resource support is stable, so a DRA driver can answer existing nvidia.com/gpu-style requests as they are. Workloads need not be rewritten to migrate, as long as the vendor's driver publishes that mapping. Workload and PodGroup ResourceClaims are beta and off by default, and until you enable them one claim can be reserved by at most 256 pods, which matters for large gangs. What stays driver-specific is MIG, time-slicing and fractional sharing, plus how mature the vendor's driver is on your hardware. Scheduler-plugin integration and operational tooling are still settling across vendors.

For a greenfield Kubernetes cluster stood up today, DRA is the forward-looking bet and the model the topology-aware work in Chapter 10.2 increasingly assumes. For an existing production fleet with a working device-plugin setup, the conservative path is to pilot DRA on one partition and keep the proven model in production until the driver and tooling story for your specific hardware is boring. Betting a brand-new revenue cluster on a still-stabilizing path is the kind of avoidable risk that shows up as goodput loss in month one.

If you really need both Slurm's batch semantics and Kubernetes' ecosystem on the same physical fleet, the answer the industry converged on is Slinky, the open-source project (from SchedMD, now NVIDIA) that bridges the two instead of forcing a choice. It has two complementary pieces. The slurm-operator runs Slurm on Kubernetes: the Slurm controller and compute daemons run as managed Kubernetes workloads, so you get Slurm's batch and gang semantics with Kubernetes lifecycle management underneath. The slurm-bridge goes the other way and lets Slurm act as the scheduler for Kubernetes, so Slurm's allocation logic governs placement of Kubernetes-submitted work. NVIDIA’s 9 April 2026 report describes Slinky slurm-operator production clusters above 8,000 GPUs running LLM training and multinode inference. Test your own hybrid at scale before you rely on that result.

The promise is one physical fleet and one pool of GPUs serving both workload styles, with no static "Slurm half" and "Kubernetes half" stranding capacity at the boundary. In exchange, you operate, patch and debug two scheduling systems wired together, with two topology models, two quota regimes and a non-trivial integration surface where the interesting failures live. Convergence pays off for operators whose workload mix is truly bimodal and whose utilization losses from static partitioning exceed the operational cost of the hybrid. For a fleet that is ~90% training or ~90% inference, the simpler answer is usually the better engineering call: pick the one scheduler that fits the dominant workload and serve the minority case from a small dedicated partition.

Stitched together, a 2026-current reference fleet looks like this. Read it bottom-up, as planes stacked by timescale. At the base, bare metal is provisioned as code: BMC/Redfish bring-up, PXE, golden images, and a join step that registers each node with the control plane (Chapter 10.5). On every node sits the node runtime: a pinned, identical stack of GPU driver, CUDA/ROCm, NCCL/RCCL and firmware, exposed to the scheduler through the GPU Operator's device plugin or a DRA driver (Chapter 10.4). The hard rule lives here. Every node in a synchronous job runs the same tested, attested stack tuple of driver, CUDA, cuDNN and NCCL, because an untested combination is what produces collective hangs, silent bandwidth degradation or wrong results.

Above the nodes sits the workload-scheduling plane: Slurm, or Kubernetes with Volcano or KAI, or both via Slinky. It must be topology-aware. It has to know that the 72 GPUs in an NVL72-class rack (GB300 NVL72 today, GB200 NVL72 before it, VR200 NVL72 shipping since June 2026) form one coherent NVLink domain, and schedule the rack as a block, not as 72 interchangeable units (Chapter 10.2). Wrapping everything is the fleet/control plane. DCGM/NVML telemetry and XID/SXID parsing feed an observability stack (Chapter 10.6). A health-and-recovery loop cordons sick nodes, drains their work, swaps in hot spares and triggers break-fix without a human (Chapter 10.7). An IaC/GitOps layer keeps the whole thing declaratively reconciled.

The architectural test of this topology is whether a node can fail and the fleet detect, drain, recover and resume, autonomously, fast enough to meet your workload's goodput requirement. Meta reported >90% effective training time on its Llama 3 campaign; your own target comes from your workload, not from Meta’s result. If manual recovery cannot meet your recovery deadline, automate it or hold enough spare capacity to cover the gap you measured, whichever scheduler you picked.

The last fork in this chapter is whether to assemble the orchestration platform from open-source parts or buy a managed product. Assembling means Slurm or Kubernetes, plus Volcano or KAI, plus the GPU Operator, plus a Prometheus/Grafana/DCGM observability stack, plus your own IaC and recovery automation. It gives you maximal control and no per-node licensing. The cost is a standing platform-engineering team that owns the integration, the upgrades and the 3 a.m. pages when a driver bump breaks a collective. That team also owns the scheduler's security baseline. Slurm's 2 September 2026 security releases (26.05.4, 25.11.8 and 25.05.9) fixed, among other things, an sbcast credential-verification bypass, OCI-container cleanup that deleted files outside its spool directory, and a privilege escalation from operator to administrator in accounting. They reach every compute node (slurmd, slurmstepd) as well as the accounting daemon (slurmdbd), so the rollout has to be scheduled around running jobs. Buying means NVIDIA Mission Control or Base Command Manager, a hyperscaler's managed Slurm or Kubernetes, or a neocloud's turnkey cluster. It compresses time-to-first-job and hands you autonomous recovery and validated golden stacks out of the box, at the price of a license, a vendor's roadmap and less ability to customize the deep internals. A managed product shifts the patch cadence to the vendor; check its SLA for it.

The deciding variables are the three that govern most build-vs-buy calls in this guide. Scale: a large, durable fleet amortizes a platform team, and a small or short-lived one cannot. Differentiation: if orchestration is a competitive edge, as for a neocloud selling goodput, you build; if it is undifferentiated plumbing, you buy. Time-to-power: buying collapses months of integration into days. The recurring mistake is the mid-size operator who builds a bespoke platform to save license fees, then spends three senior engineers' salaries maintaining it. That costs far more than the license, and it ships lower goodput than the managed product would, because the team is re-deriving autonomous recovery that a vendor already shipped. Chapter 10.9 covers the productization view of this same fork: what operators actually sell on top of these platforms.