Multi-Tenancy, Isolation & Resource Sharing

Multi-tenancy turns a depreciating pile of accelerators into a utility. A frontier pre-training run wants the whole machine to itself. Smaller fine-tuning, batch, interactive-serving, notebook and CI workloads can under-fill a modern GPU. B200 SXM carries 180 GB of HBM and a GB200 Blackwell GPU 186 GB usable. Blackwell Ultra (B300/GB300) is specified at 288 GB of HBM3e per GPU, though usable capacity is somewhat lower and varies by SKU; Hopper parts span 80–141 GB. Recompute every slice size in this chapter against the part you actually bought. A 7B-parameter serving replica or a data-science notebook can fit in part of one of these GPUs, and concurrency, context length and the latency target decide whether useful room is left. By Deloitte's forecast, inference reaches about two-thirds of AI compute in 2026, against half in 2025. The largest models still occupy entire GPUs or racks; much of the expanding serving load, though, is small and mid-size replicas too light to fill a GPU, and that is where sharing earns its keep. An unshared, under-filled GPU still depreciates while it idles; Chapter 1.8 weighs the 2–3-year bear case for its economic life against 4–6-year useful-life estimates. This chapter is about reclaiming that stranded silicon without letting one tenant's work corrupt, starve or spy on another's.

A specific error recurs: operators pick a sharing mechanism for a utilization reason, then assume it also delivers isolation. The two are different axes that happen to share the word "partition." MIG partitions documented GPU-local compute, memory and memory paths, but shared host privilege, management and residual side channels still need a tenant threat model (Chapter 11.6). A container gives you a namespace but shares a kernel and a GPU driver with everyone else on the node. This chapter keeps three questions apart: the sharing spectrum (how the silicon is divided), quota and fairness (who gets how much, and when) and isolation (what one tenant can do to another). Get the mapping wrong and you either leave money on the floor or put two adversarial tenants behind a wall that was only ever a speed bump.

There is no single "GPU sharing" feature. Five mechanisms sit at different points on a curve that trades isolation strength against packing flexibility, and they compose: you can run MIG instances, each with MPS inside, scheduled by a fractional-GPU plugin. Read them from hardest partition to softest.

Whole-GPU is the degenerate case and the right default for tightly-coupled training. One job owns the device, the NVLink domain and the memory, so there is no sharing overhead, no noisy neighbor and full bandwidth for collectives. The cost is utilization: anything that does not saturate the GPU wastes it.

On rack-scale systems such as GB200 and GB300 NVL72, whole-GPU tenancy has a sub-rack unit: the NVLink partition. By default NVIDIA configures the rack’s 72 GPUs as one NVLink partition, and the operator can split that domain into smaller partitions with NVIDIA’s NVLink management software (NMX). GPUs in different partitions cannot open NVLink connections to each other. NVIDIA’s documentation states that partitioning does not isolate the scale-out network or storage. Physically, every partition still shares the rack’s NVSwitch trays, power and cooling. An NVLink partition is therefore the sub-rack unit you can sell to one tenant, but that tenant’s boundary still needs network and storage controls of its own. Domain discovery and IMEX are covered in Chapter 10.2, and the tenancy and SLA consequences in Chapter 10.9.

MIG (Multi-Instance GPU) is NVIDIA's hardware partition, and of these five mechanisms it alone divides the silicon in hardware. It carves the GPU into up to seven instances, each with a dedicated slice of SM compute, L2 cache, memory controllers and a fenced region of HBM. GB200's 186 GB of usable HBM per GPU does not by itself specify a supported MIG layout. NVIDIA's B200 profile table (180 GB total) lists two 3g.90gb instances, four 1g.45gb instances or seven 1g.23gb instances, with different compute fractions. Take the profile table for the SKU and driver you actually receive, and do not divide GB200 memory arithmetically. MIG dedicates documented GPU-local compute and memory, but instances still share the host, PCIe, fabric, storage and control plane. MIG can cut interference on the GPU itself. An end-to-end SLO still needs measured admission and QoS at every boundary the instances share.

MIG pays for this in rigidity, because every slice has a fixed size and a fixed placement. Once a GPU is in MIG mode, the driver can add and remove instances without stopping the rest of the GPU, and since Hopper, switching MIG mode on no longer needs a reset. On Kubernetes the usual tool is the GPU Operator's MIG Manager. It applies one layout to the whole node and stops that node's GPU pods first, so treat a geometry change as a node drain unless your tooling resizes single instances. Fragmentation can block a larger profile. A B200 job that needs 30 GB will not fit a 1g.23gb slice, so it has to take a supported profile of 45 GB or more and waste the difference.

On AMD Instinct the same job is done by two settings used together: compute partitioning (SPX/DPX/CPX) and memory partitioning (NPS modes). CPX mode presents each of the eight XCDs to software as its own logical GPU, with 38 CUs and 24 GB each on MI300X. Pairing it with NPS4 on MI300X, or NPS2 on the MI350 series, keeps each partition's memory local. Which pairings AMD validates depends on your part and driver, so check its mode matrix.

Under MPS (Multi-Process Service), each client keeps its own CUDA context and GPU address space, and all of them feed work straight into the GPU's shared scheduling resources at the same time. That is true spatial sharing of the SMs, not just interleaving. It suits several small, cooperative workloads in one trust domain, such as a fleet of tiny inference replicas that can fill a GPU together. The cost is the absence of a hard wall. MPS provides limited resource controls and error containment, and a fatal fault can disrupt every co-client sharing the affected GPU set; clients confined to other GPUs remain unaffected. With R610, static SM partitions on Ampere and newer partially contain SM-triggered faults, but not every GPU or system-level failure. MPS buys throughput by assuming the co-tenants trust each other.

Time-slicing is the GPU equivalent of a context switch: the scheduler round-robins the whole device between processes, each getting the full GPU for a slice of time. It needs no special hardware and works on any GPU, which makes it the lowest-common-denominator mode for dev, notebooks and bursty low-priority work. The cost is latency jitter and zero memory isolation: every tenant sees the full device, can over-allocate HBM and pays context-switch overhead. Nothing with a tight tail-latency SLO can live on it.

Fractional GPU happens in the scheduler: Run:ai, KAI and similar tools let a job ask for "0.5 of a GPU" or "10 GB of a GPU". The fraction means different things on different platforms. Run:ai holds each pod to its memory share through node-level software, and by default time-slices the compute. Open-source KAI's default mode just reserves the share: nothing stops a container using more memory than it requested, unless you turn on its HAMi-core or NvFractions enforcement. On both platforms, concurrent compute through MPS has to be switched on. Fractions are the most flexible way to bin-pack a mixed fleet and the friendliest for developers, but their isolation is only as strong as the layer behind them. Use the fraction for billing and packing; it is not a security boundary.

Dividing a single GPU is the easy half. The hard half is governing a 10,000-GPU fleet so that twenty teams, each convinced their work is most urgent, share it without a tragedy of the commons. Three models dominate. The right one depends on whether your tenants distrust each other and whether your workloads are elastic.

Hard partitions (static quota). Each tenant gets a fixed slice of the cluster, N GPUs, and the guarantee has to be backed by dedicated nodes or a standing reservation. A Kubernetes ResourceQuota caps what a namespace may consume, and Slurm partition membership routes jobs at a node set. Neither, on its own, reserves matching GPUs, the topology they need, or free running capacity. Sell an admission cap as a reservation and you have contracted capacity nothing enforces. Hard partitions are dead simple and perfectly predictable, and they are the right model when tenants are external customers paying for guaranteed capacity (the neocloud and colo posture; Chapter 10.9). The cost is stranded capacity. When Tenant A is idle and Tenant B is queued, the idle GPUs sit dark because the partition forbids lending. Fleet utilization caps out well below what the silicon could deliver.

On Kubernetes, GPU slices are moving to Dynamic Resource Allocation (Chapter 10.1). As of October 2026, NVIDIA's DynamicMIG, which cuts a MIG instance for each claim, is still alpha and cannot run alongside the driver's MPS support, so production fleets keep pre-configured MIG layouts. Device taints became stable in Kubernetes 1.37. After a tenant teardown or a fault, they let you quarantine one GPU and evict its pods without draining the node.

Hierarchical fair-share with preemption. This is the HPC-and-internal-platform model. Tenants get a target share, not a hard cap. The scheduler lets anyone borrow idle capacity, then preempts the borrower when the rightful owner returns. Slurm implements the share accounting as multifactor priority and Fair Tree: a tenant that has under-consumed its share recently floats to the top of the queue, and an over-consumer sinks. That only orders the queue. Evicting a borrower is a separate, explicitly configured mechanism (PreemptType/PreemptMode plus the QoS preemption rules), and it does nothing until you configure it. Kubernetes-native schedulers (KAI, Run:ai, Volcano) implement the same idea as hierarchical queues with reclaim. This model actually drives high utilization, because idle silicon is always lendable. The cost is complexity and the preemption tax: a preempted borrower must checkpoint and resume, so the model is only sane for interruption-tolerant work (Chapter 9.4).

Priority / deadline QoS. This layers on top of either model. Jobs carry a priority or a deadline, and the scheduler admits, preempts and orders accordingly: production serving outranks experiments, and a deadline batch sweep escalates as its window closes. Here you encode that an SLO-bound inference tenant must never be preempted by a speculative training run. The cost is that priority only means something if it is scarce and policed. The moment every team sets priority=high, you are back to FIFO.

One non-negotiable couples all three to the prior chapter. GPU jobs are usually gang-scheduled: a 64-GPU job needs all 64 at once or none, or it deadlocks holding resources it cannot use. Quota and fairness must therefore admit and preempt at the granularity of the whole gang, topology-aware, or the fairness model fights the placement model (Chapter 10.2).

The costliest error, and the one that shows up in real CVEs, is treating a sharing mechanism as an isolation boundary. Sharing answers "how is the silicon divided?" Isolation answers "what can one tenant do to another?": corrupt their data, starve their compute, read their memory or escape onto the host. The two axes are orthogonal, and the isolation strength you need is set by tenant trust, not by how you happen to pack the GPU.

Stack the isolation tiers from weakest to strongest.

Process isolation (bare processes, or MPS clients) shares a kernel, a driver and frequently a CUDA context. That is fine for one team's own jobs and useless across trust domains.

Container isolation adds namespaces and cgroups but still shares the host kernel and the GPU driver/runtime. The attack surface is the container runtime and the NVIDIA stack itself, which is exactly what the 2025 NVIDIAScape vulnerability (CVE-2025-23266, CVSS 9.0) punctured: a crafted container could escape to the host on shared GPU nodes.

VM isolation gives each tenant its own kernel behind a hypervisor. The GPU is passed through (full device) or virtualized (vGPU/MIG-backed). That is stronger, but the vGPU manager is itself a shared component. CVE-2025-23290 showed a guest VM reading global GPU metrics affected by other VMs' work: a cross-VM leak through that layer.

Confidential VM + GPU TEE is the top tier. The workload runs in a CPU trusted execution environment, and the GPU runs in an attested, platform-specific confidential-compute mode. In that mode on-package HBM is access-controlled, not encrypted, and the GPU package is the trust boundary. Hopper encrypts CPU↔GPU transfers but leaves GPU↔GPU NVLink unencrypted in protected-PCIe mode. Blackwell multi-GPU pass-through also encrypts the NVLink path between the GPUs of one confidential VM. The design goal is that even the operator (hypervisor, host OS, cloud admin) cannot read tenant data or weights, and that holds against software access. An attacker with physical access to the host's memory can defeat CPU-side attestation, so physical custody of the host belongs inside the boundary. Chapter 11.5 (GPU confidential computing and attestation) is the canonical home for this tier; the multi-tenant security architecture that wraps it is Chapter 11.6, and the network half, per-tenant microsegmentation and east-west zero-trust on the storage and management planes, is Chapter 11.7.

The decision rule is blunt: match the isolation tier to the trust boundary, then choose any sharing mechanism that fits inside it. Two jobs from the same team can share an MPS context behind a single container. The trust domain is one, so weak isolation is fine and you maximize packing. Two mutually distrusting external customers must never share a kernel. That is a VM boundary at minimum, and a confidential VM when your own platform must be untrusted (the regulated, sovereign-AI and model-weight-protection cases). MIG and vGPU sit awkwardly in between. They give strong performance isolation, but documented side channels (uncore counters, shared metrics, microarchitectural leakage) mean you do not sell them as a confidentiality boundary between adversaries. They are a QoS boundary that happens to live in hardware.

Even inside a single trust domain, where security is not the concern, multi-tenancy has a performance pathology: the noisy neighbor. One tenant's job saturates a shared resource and silently taxes everyone else's goodput, and on a shared GPU node the contended resource is rarely the one people watch. Four resources get contended: memory bandwidth (HBM is the bottleneck for most inference and the noisiest shared resource under MPS/time-slicing), PCIe/NVLink (a greedy host-to-device copy starves a neighbor's transfers), the shared fabric (a tenant's all-to-all floods the back-end and inflates everyone's collective time; congestion engineering is in Chapter 8.6), and shared storage and the loader (one tenant's checkpoint write or dataset scan blows the cache for the rest).

You bound the noisy neighbor at one of three layers, hardest to softest. At the hardware, MIG partitions SM and L2 slices, bounding core and cache interference for tenants with a hard SLO; shared uncore paths remain observable and still need per-tenant telemetry. At the scheduler, resource limits and requests, priority and preemption let a low-priority neighbor yield, and gang-aware admission means a job either gets its full topology-clean allocation or waits, rather than half-landing and contending (Chapter 10.2). At the fabric and storage, per-tenant QoS classes, rate limits and bandwidth reservations stop any one tenant monopolizing the network or the loader.

Skipping all three produces the failure mode that is hardest to debug, because nothing errors. Every tenant's p99 latency quietly degrades, the SLO dashboard goes amber, and the cause stays invisible unless your telemetry attributes contention per tenant. That is why per-tenant goodput and contention metrics belong in the observability plane from day one (Chapter 10.6).

The QoS guarantee you can offer follows from the mechanism. No GPU partition alone guarantees an end-to-end tenant SLO regardless of neighbors. MIG or whole-GPU can bound GPU-local contention, but the contract also needs host, PCIe, fabric, storage, control-plane and measurement boundaries. A soft guarantee ("this tenant gets a target share, best-effort, degrading gracefully under contention") is what MPS, time-slicing and fractional GPUs can deliver. Selling a hard SLO on a soft mechanism works in the demo, when the GPU is half-empty, and breaches the first time the cluster fills.

The mistake is to pick one sharing mechanism, one quota model and one isolation tier for the whole fleet. The defensible design classifies tenants and maps each class down all three axes at once.

• A frontier training tenant: whole-GPU, hard partition or a top-priority gang reservation, container isolation (one trust domain), and a noisy neighbor bounded by sole ownership.
• An external SLO-bound serving tenant: MIG where measured GPU-local partitioning helps, reserved ready capacity behind the quota, a tested VM or CVM boundary across trust domains, and end-to-end QoS plus telemetry for every shared path.
• An internal research/dev fleet: fractional GPU and time-slicing for packing, hierarchical fair-share with preemption for utilization, container isolation, and a noisy neighbor bounded by scheduler limits.
• A preemptible batch tier: whatever packs tightest, lowest priority with a measured reclaim deadline, container isolation, and explicitly soft QoS.

One tested physical fleet can run all four. The art is keeping each class's mechanism, quota and isolation tier consistent, so the guarantees you sell have capacity and a tested recovery path behind them. Fair-share without preemption changes future queue order; it does not revoke a running allocation.