Multi-Tenant & Workload Isolation Security

Multi-tenancy is the business model of the entire neocloud and most enterprise platform teams: buy one expensive accelerator, rent fractions of it to many workloads, and spread its cost across tenants whose measured workloads leave shareable headroom. The gain is measured in useful output after contention, reserve and recovery. The security cost of that sharing is routinely underpriced. The moment two tenants share a physical GPU, a memory controller, an L2 cache, an NVLink domain, or just a host kernel and a container runtime, you have a shared dependency to check. It is either a strong boundary, a soft convenience dressed up as one, or an outright leak. The work is telling them apart before a tenant's KV-cache shows up in a neighbor's framebuffer.

The chapter walks the multi-tenancy spectrum from bare metal to time-slicing, answers from the hardware whether MIG, vGPU or time-slicing is a security boundary, and treats memory hygiene on reassignment as a policy you own. The isolation-versus-utilization economics decide how far down the spectrum each data class can go. The TEE and attestation machinery lives in Chapter 11.5; here we decide when you need it.

Tenant isolation is a spectrum, and each step down it trades a stronger boundary for higher utilization. Read it top to bottom as falling isolation strength and rising density. The moment you stop giving each tenant a whole physical GPU, you share silicon that was never designed as a confidentiality boundary.

Bare-metal-per-tenant is the strongest posture. A tenant gets whole nodes, whole GPUs and ideally a whole rack or NVLink domain, with no hypervisor and no co-tenant on the same silicon, so the blast radius is the tenant's own. Serious training customers and sovereign deployments demand it, and the strongest neocloud SLAs implicitly promise it. The cost is utilization: a tenant who under-uses the hardware strands it, and you cannot backfill without reintroducing a co-tenant.

Bare-metal gives the tenant root, which means the handoff has to undo anything root could have changed. Before the next tenant arrives, reflash or attest the BMC, NIC/DPU and GPU firmware so each runs a known image, and issue new BMC credentials. Eclypsium's 2019 Cloudborne research showed why: after a rented server was reprovisioned for another customer, its modified BMC image and old BMC root password were still there. Whatever software performs the reset becomes a trust root for every tenant. NVIDIA's Infra Controller (NICo), which handles bare-metal provisioning, shipped with hard-coded credentials (CVE-2026-65113, CVSS 9.8; versions through 1.9, fixed in 2.0). Chapter 11.3 covers firmware reset on reuse and Chapter 11.4 covers BMC integrity.

VM-per-tenant (whole-GPU passthrough) keeps the GPU undivided but adds a hypervisor. Each tenant gets a strong CPU-side boundary (SR-IOV / PCIe passthrough) while one VM at a time owns the GPU end to end. The boundary you now trust is the hypervisor and the IOMMU, not the GPU. This is the workhorse of enterprise multi-tenant clouds and a genuinely defensible boundary. The residual risks are hypervisor escapes and the host-side GPU management stack.

MIG (Multi-Instance GPU) partitions one physical GPU into up to seven hardware-isolated instances, each with dedicated SM and memory-system slices. Copy and media-engine allocation depends on the product and profile, and compute instances inside one GPU instance share their parent's memory and engines; choose a +me profile where you need dedicated media engines. On NVIDIA GPUs, MIG is the only fractional-sharing mode with a hardware-enforced partition, so it is the one rung of fractional sharing there that can credibly be called a security boundary. The costs are rigidity (fixed geometries, no oversubscription) and the residual uncore side-channels discussed below.

AMD Instinct partitions draw the boundary in a different place. An MI300X, MI325X or MI350-series GPU runs as one device (SPX) or splits its compute dies into logical GPUs (DPX, or CPX with one die per logical GPU), and NPS modes split its HBM into NUMA domains. AMD offers these partitions for separating tenants, but its own partitioning guide notes that a compute die can still reach all of the GPU's memory through multi-GPU programming. On bare metal, CPX and NPS are performance partitions until a test proves that no partition can map another's memory. For hostile tenants, AMD's MxGPU host driver exposes each partition as an SR-IOV virtual function, one per GPU in SPX or eight in CPX. Each function goes to its own VM, and the hypervisor and IOMMU confine every VM's DMA to its own memory. Inside the package, though, only the GPU's partitioning logic and AMD's privileged host driver separate the eight partitions, which makes this a vGPU-style trust model rather than whole-GPU passthrough. AMD's partitioning and virtualization guides say nothing about side channels between CPX partitions, and the uncore side-channel studies below tested only NVIDIA parts. Test an eight-VF MI300X the way you would test vGPU before you sell it as fractional isolation.

vGPU (mediated virtualization) time-multiplexes a GPU across guest VMs through a host-resident vGPU Manager that schedules and mediates access. The VM boundary is real, but the GPU is now shared through a software mediator running as privileged host code, and that mediator has a real CVE record. It includes 2025 guest-to-host stack overflows (CVSS 7.8) and a low-severity cross-VM metrics leak. vGPU can be layered on MIG (MIG-backed vGPU) to combine hardware partitioning with the VM boundary, which is the strongest practical fractional mode.

Time-slicing is the bottom rung: processes share one GPU with neither memory nor fault isolation. It is a utilization tool for trusted workloads, not a tenant boundary.

MPS is a separate service for cooperative workloads. Volta-and-later clients get isolated GPU address spaces, but scheduling hardware, memory bandwidth, caches and capacity stay shared. Standard MPS provides no error isolation; R610 static SM partitioning contains some SM-triggered faults but is not a general fault-isolation guarantee. Neither mode is suitable as a hostile-tenant boundary.

The 2024–2026 disclosure record contains real, assigned CVEs at every layer of the stack, and reading them is the fastest way to calibrate which boundaries hold. The pattern is instructive: the catastrophic breaks are almost never in the GPU partitioning itself but in the control plane around it.

The control-plane escape (the real catastrophe). NVIDIAScape (CVE-2025-23266, CVSS 9.0, disclosed by Wiz in July 2025) is a three-line container escape in the NVIDIA Container Toolkit. A malicious container sets LD_PRELOAD against an OCI createContainer hook and gains root on the host, from which it can read, steal or tamper with the models and data of every other tenant on the shared machine. It affected Container Toolkit up to 1.17.7 and GPU Operator up to 25.3.1. Multi-tenant security is a full-stack property, and the runtime is the weakest, most exposed link.

The mediator leak (vGPU). NVIDIA's July 2025 GPU display-driver bulletin included two cross-VM flaws in the vGPU Manager. CVE-2025-23290 (CVSS 2.5) let a guest read global GPU metrics that work in neighboring VMs can influence, a low-severity but explicit acknowledgment that the mediator can leak co-tenant activity. CVE-2025-23285 (CVSS 5.5) let a guest access global resources, with denial of service as the impact. The same bulletin patched stack buffer overflows in the vGPU Manager (CVE-2025-23283/23284, CVSS 7.8) that enabled guest-to-host code execution. The mediated-virtualization boundary is real, but it is privileged software, and privileged software has bugs.

The memory-remanence leak (affected local-memory stacks). LeftoverLocals (CVE-2023-4969, Trail of Bits, 2024) recovered another process's data from uncleared GPU local memory across process and container boundaries on affected Apple, Qualcomm, AMD and some Imagination devices. That was enough to reconstruct an LLM's responses (≈181 MB recoverable per query against a 7B model on llama.cpp, measured on an AMD Radeon RX 7900 XT). Tested NVIDIA devices returned zeros, and the disclosure did not evaluate global HBM, MIG teardown or vGPU release. The control belongs at the local/shared-memory layer: use patched drivers and kernels that clear scratchpad on affected stacks.

The bit-flip escalation (GDDR6 Rowhammer). A University of Toronto group has shown that an unprivileged CUDA program can hammer GDDR6 GPU memory into bit flips and turn them into control of the host. GPUHammer (USENIX Security 2025) first flipped bits on an RTX A6000. GPUBreach (IEEE S&P 2026) corrupted GPU page tables and chained a driver memory-safety bug into a root shell on the host CPU, with the IOMMU switched on. GPUThor (ACM CCS 2026, public since 25 August 2026) beat ECC on four Ampere workstation GPUs, RTX A4000 through A6000. ECC flagged most multi-bit errors as uncorrectable but silently mis-corrected two triple-bit flips, and the attack still reached host root. The same patterns produced no flips on HBM, GDDR6X or GDDR7 parts; that is a test result, not immunity.

NVIDIA's answer is guidance, not a patch: run system-level ECC. Hopper and Blackwell data-center GPUs ship with it on, but no GDDR6 board is among them. On the GDDR6 boards at risk, check the setting yourself, and expect ECC to take 6.25% of memory on an A6000. In a GPU cloud, a GDDR6 inference fleet makes a weaker tenant boundary than an HBM fleet wherever tenants share one GPU or run as containers on the host's driver. Whole-GPU VM passthrough was outside these tests. Turn ECC on, keep the IOMMU on and the driver patched, alert on uncorrectable-error counts, or sell those boards only as whole-host, single-tenant capacity.

The uncore side-channels (even MIG). Spy in the GPU-box (arXiv 2203.15981) demonstrated cross-GPU L2/NVLink channels on an eight-P100 Pascal DGX-1; it did not test MIG. Veiled Pathways demonstrated uncore channels based on DRAM-frequency contention, media engines under MPS, and shared PCIe I/O between MIG instances. Even MIG's partitioned L2 is not timing-tight. Behind Bars (USENIX Security 2026) ran memory barriers in one Hopper MIG instance and watched load timing shift in another. From that shift the attacker could infer a victim's kernel-launch pattern during LLM inference. Earlier work, TunneLs (CCS 2023), found the last-level TLB shared across Ampere-generation MIG instances and used it for a covert channel. MIG's partition is genuinely hardware-enforced for memory and compute, but it is not proof of side-channel resistance. Where timing leakage is in scope, MIG is necessary and not sufficient, and a GPU TEE does not fill the gap because its deployment threat model excludes side channels. Pick the mitigation from the specific shared resource and leakage mechanism: stop sharing the contended engine, dedicate the allocation, or partition the path. Then test the chosen control against the actual attack instead of assuming a TEE gave you timing noninterference.

The most routine multi-tenant control is also the most overlooked: verify the teardown path when a resource changes hands. Whenever a whole GPU, MIG instance, vGPU slice or container allocation moves between tenants, require the vendor-defined clearing or cryptographic teardown for that mode, and test it on the deployed stack before reassignment. You decide who runs that test and whether reassignment fails closed when it does not pass.

Two postures compete: rely on the documented driver/firmware teardown, or gate reassignment on an operator-controlled verification test. The former minimizes handoff latency but inherits the vendor's implementation. The latter adds measured scheduler delay and gives the tenant a test record. For confidential workloads, the protected HBM region holds tenant data in plaintext, so verify that the platform's attested teardown scrubs that residual data before you reassign the GPU (Chapter 11.5).

When the data class cannot tolerate exposure to a co-tenant or to the operator (regulated health and financial data, sovereign workloads, frontier weights rented on infrastructure you do not own), the only sufficient answer is confidential computing. That means a per-tenant Trusted Execution Environment that spans the CPU (SEV-SNP / TDX) and the GPU. On NVIDIA Hopper and Blackwell, the GPU side combines the BAR0 decoupler with a protected HBM region that is access-controlled, firewalled rather than encrypted. Transfers between CPU and GPU are encrypted, and on Blackwell so is NVLink between the GPUs of one confidential VM. Attestation gates it, so a tenant releases keys only to a measured, verified configuration. Physical custody of the host belongs inside this boundary: memory-interposer attacks (TEE.fail, DDRop) have forged CPU attestation with parts costing under $1,000. The full machinery, including Compute Protected Region state, attestation via NRAS/RIM, TEE-I/O, NVLink encryption and the residual attack surface, is the canonical subject of Chapter 11.5. The decision here is whether you need it, because it is not free.

Strong VM isolation plus disciplined hygiene defends against an honest-but-curious neighbor and an accidental leak. It does not defend against a malicious co-tenant exploiting a side-channel, a compromised or coerced operator, or an insider with host access. Confidential multi-tenancy moves the operator out of the trust boundary and makes attestation the gate. The price is a measurable performance tax (encrypted PCIe/NVLink transfers and TEE overhead, larger for small-transfer chatty workloads than for large compute-bound ones), no fractional sharing, and real operational complexity in key brokering and attestation-policy management. Every confidential-computing mode NVIDIA supports on Hopper and Blackwell passes whole GPUs to a confidential VM, one or several at a time (NVIDIA Secure AI whitepaper, August 2025; Trusted Computing R595 release notes, April 2026). No GPU is shared among VMs, so a MIG slice or a vGPU cannot sit inside the TEE. A confidential tier is therefore a whole-GPU product priced like VM-per-tenant, and a small inference tenant who needs it pays for a full GPU. You pay goodput and ops to remove the operator from the trust model. If your tenants do not require that, you are buying nines of confidentiality the data class does not value, the security analog of over-provisioned redundancy.

Every rung you climb for stronger isolation costs utilization, and utilization is the whole economic premise of multi-tenancy. The tension is quantifiable. A bare-metal tenant who drives a $30k+ accelerator at 40% leaves ~60% of it idle and unbackfillable. MIG recovers much of that by packing up to seven isolated tenants onto one GPU. Time-slicing recovers the most by oversubscribing, at the price of being no boundary at all. Confidential computing climbs back down the utilization ladder: it hands out whole GPUs only, taxes transfers and complicates scheduling, so the same hardware serves fewer effective tenant-hours.

The decision is therefore not "how secure can we be" but "what is the cheapest posture that is sufficient for this data class's adversary model", which is why the table above is organized by adversary, not by feature. Over-isolating is a real cost: a commercial inference fleet serving public, non-sensitive content that runs everything under per-tenant confidential computing burns goodput and capex to defend against a threat its data class does not face. Under-isolating is a breach: a regulated-data tenant time-sliced onto shared silicon with a stranger is one un-scrubbed allocation away from a disclosure. SemiAnalysis grades GPU clouds in ClusterMAX 3.0 (September 2026) against itemized criteria grouped into ten categories. Security is graded as its own category, down to whether tenant network isolation is actually enforced. Buyers now price the boundary, not just the FLOPs.

In 2026 a serious multi-tenant operator should default to a tiered offering. Customers who demand bare metal or a dedicated VM, and pay for it, get one. MIG-backed vGPU with verified hygiene is the standard fractional product. Time-slicing stays strictly inside one tenant's own cooperative workloads and never crosses a trust boundary. Confidential computing is the premium tier for regulated and sovereign demand, sold by the whole GPU. Selling time-slicing as a cross-tenant boundary, or selling "isolation" without naming which rung, is the misrepresentation that turns a CVE into a liability.