Network Segmentation, Microsegmentation & Zero Trust

East-west traffic dominates a data center: workload to workload, GPU to GPU, host to storage, service to service. On a training back-end fabric it is nearly the only traffic. The classic enterprise security model assumed the opposite shape: a fortified perimeter (north-south) around a soft interior, assumed friendly because crossing the perimeter was supposed to be hard. An AI cluster is almost entirely interior. Earlier chapters in Part 11 hardened individual components: the supply chain (11.3), the root of trust and firmware (11.4), the GPU TEE (11.5) and the tenant boundary (11.6). This chapter hardens the connective tissue, the network that lets thousands of those components behave as one machine. The perimeter you spent your budget on guards a building whose interior is a single open floor.

A segmentation design answers one question: when an attacker gets one foothold (a compromised inference container, a poisoned dependency, a credential lifted from a CI runner), how far can they go before they hit a wall? On a flat fabric the answer is "everywhere the routing table reaches," and they get there fast. CrowdStrike measured average eCrime breakout time, from initial access to first lateral movement, at 29 minutes in 2025, down from 48 the year before. The fastest observed was 27 seconds, and in one case exfiltration began within four minutes (CrowdStrike 2026 Global Threat Report). Segmentation is what cuts that run short. It comes in three layers: macro- (coarse zones), micro- (per-workload policy), and the zero-trust posture that refuses to treat network location as a proxy for trust at all. The rest of the chapter walks those layers: where each wall costs goodput, where it costs nothing, and where leaving it out costs you the weights.

The most common mistake is to reason about "the network" as one thing. An AI cluster has at least three physically and logically distinct fabrics. Each carries different traffic under different performance constraints, so each needs its own segmentation strategy. Conflate them and you either strangle the workload (inline enforcement on the wrong fabric) or leave the crown jewels exposed (no enforcement on the fabric that needs it).

The back-end / scale-out fabric (InfiniBand or RoCE/Spectrum-X Ethernet, → 8.4, 8.5) carries the collectives (all-reduce, all-gather, all-to-all) that dominate every training step. Its design target is microsecond-scale latency. In NVIDIA's reference designs it is also non-blocking, but real deployments sometimes oversubscribe above the pod (→ 8.1). A stateful L7 firewall inline here is a category error. It adds latency to the one path whose latency directly sets goodput, and a single straggler link drags the whole synchronous job.

The front-end / north-south fabric carries API traffic, ingress, internet egress and tenant access. The classic perimeter and inference-endpoint controls live here, and inline inspection is affordable because the traffic is comparatively sparse and latency-tolerant. The management / out-of-band (OOB) fabric (→ 8.7) reaches BMCs, PDUs, CDU controllers, switches and the orchestration plane. It carries the least traffic and the most privilege. Its compromise is catastrophic because it is the path to firmware (11.4) and to physical-systems control (11.10).

Before microsegmentation comes macrosegmentation: partitioning the cluster into a handful of trust zones with controlled choke points between them. It is the cheap, high-leverage layer, and mostly a build-time design decision: VRFs/VLANs on the front-end, PKeys (InfiniBand partition keys) or VXLAN/EVPN segments on the back-end, and a hard separation of the management network. The zones that matter for an AI cluster are easy to name: ingress/DMZ, inference-serving, training/compute, storage, management/OOB and, where weights live, a high-value-asset enclave with the tightest egress of all.

Skipping this layer produces a recurring breach pattern. An attacker who compromises a low-value, internet-adjacent service (an inference container, a monitoring agent, a CI runner) finds nothing between them and the training fabric where the weights are loaded. The most important coarse wall is between the workload that talks to the internet and the workload that holds the model.

On the back-end fabric, InfiniBand partition keys are the native tool. The subnet manager defines PKeys, and adapters and switch ports enforce them in hardware. Like VLANs, they are a configuration boundary: they hold as a security boundary only while the attacker cannot change that configuration, so whoever controls the subnet manager or a misconfigured port can cross them. Know who controls the subnet manager, the adapter tables and the DPU policy. An RDMA rkey authorizes access to one registered memory region in its protection context; it is not by itself a tenant identity. Test the denied cross-tenant paths and revocation directly rather than inferring isolation from the product's name.

Microsegmentation makes the individual workload, not the subnet, the unit of policy. A flow from container A to container B is permitted only if an explicit identity-based rule says so, whether or not A and B share a VLAN. In a traditional enterprise this is done with host agents or hypervisor firewalls. Both use the CPU cycles an AI host wants for data movement, and host agents sit in the same trust domain as the workload they are supposed to police. For current NVL72 platforms, move line-rate enforcement onto the NIC, DPU or switch ASIC that carries each flow, and keep policy control outside the tenant host (→ 8.3).

The enforcement point is fabric-specific. In GB300 NVL72, four ConnectX-8 SuperNICs per tray carry east-west GPU traffic while one BlueField-3 carries north-south and storage traffic. In Vera Rubin NVL72, ConnectX-9 carries scale-out tenant traffic and enforces policy on it locally, while BlueField-4/Astra installs policy and monitors telemetry from a host-independent control domain. Rubin's four per-GPU 1.6 Tb/s east-west paths (two 800 Gb/s ConnectX-9 SuperNICs each) therefore do not traverse BlueField-4’s advertised 800 Gb/s DPU interface. Dedicated NIC, DPU and switch silicon preserves line rate without consuming host cycles, but the design must cover every physical datapath rather than assume every packet crosses one DPU (NVIDIA GB300 Enterprise Reference Architecture, May 2026; NVIDIA BlueField-4 architecture, Aug 2026). The trade is sharp. Host-agent enforcement costs GPU-host CPU and shares the workload's trust domain. Host-independent NIC/DPU/switch enforcement adds procurement and control-plane complexity but survives host compromise.

Zero Trust is the principle underneath all of the above, and it is widely mis-sold as a product you buy. The canonical definition is NIST SP 800-207 (Aug 2020): no implicit trust is granted to assets or user accounts based solely on their physical or network location. The shorthand is "never trust, always verify." Every access request is authenticated, authorized and (ideally) encrypted, on the working assumption that an attacker may already be inside the environment. Two later NIST documents turn the principle into practice. SP 800-207A (September 2023) applies it to service identity and service mesh in cloud-native systems, the workload-identity model described below; SP 1800-35 (June 2025) documents 19 example builds from commercial products. For AI facilities specifically, NIST's SP 800-239 initial public draft (July 2026; comments closed September 2026; not final) is a standards-body security analysis of AI data centers built from an HPC starting point.

For an AI cluster the principle has three concrete consequences. First, workload identity replaces IP address as the basis of authorization: a flow is allowed because the calling workload proves who it is (mTLS, SPIFFE/SVID, signed tokens), not because it sits on the right subnet. Second, the management plane is never implicitly trusted from the data plane: reaching a BMC means going through a policy-enforcement point and authenticating, even from inside the building. Third, and this is where zero trust and confidential computing converge, attestation becomes an access gate. A node is admitted to the trusted compute pool, and released a key, only after it proves that its firmware and TEE evidence match the approved policy (the attestation flow is canonical in 11.5; key release in 11.8).

mTLS, SPIFFE/SVID and issuer-validated tokens can authenticate identity without platform attestation. Add attestation when authorization requires hardware or software-state evidence, and bind it to the caller, recipient and session rather than accepting an unrelated passing quote.

Control egress. Every other control limits how far an attacker can move; egress control limits whether protected assets can leave. The high-value enclave should have a default-deny logical egress policy boundary: approved destinations, identities, protocols, transfer sizes and workflows are explicitly allowed, and large or anomalous outbound transfers are blocked and alerted. The policy need not be one physical proxy or link. Redundant proxies, gateways, routes and enforcement points can implement the same policy and avoid a single availability failure domain, provided policy, logging, keying and change control stay consistent.

Set the enclave's egress rate cap from the size of the weights, not from what is convenient. Take a 1 TB weight file (roughly 500B parameters in BF16) as an illustration; a full training checkpoint, with optimizer state, is about seven times larger (→ 9.4) and slower to steal. At 100 Mb/s the file takes about 22 hours to leave (80,000 s: 8×10^12 bits at 10^8 b/s), long enough for detection and response to act. Through an uncapped 400 Gb/s port the same file is gone in about 20 seconds. Anthropic's ASL-3 safeguards (May 2025) rely on egress bandwidth controls for this reason. Compression shrinks the payload and weakens any fixed cap (→ 11.8), so add volume alerts, and route approved bulk flows such as checkpoint replication and dataset ingest through their own authenticated, logged path.

Default-deny breaks package installs, model-hub pulls, telemetry and license checks unless those flows are engineered. Segment policy by zone and sensitivity, require human or cryptographic approval where appropriate, and aggregate logs from every redundant enforcement point into one detection view. Network segmentation, weight protection (11.8) and insider-threat defense (11.9) meet at this logical policy boundary, not at a mandatory physical chokepoint.

The management and out-of-band fabric carries the least traffic and the most consequence. It reaches every BMC/IPMI controller, every switch management port, the orchestration plane (Slurm/Kubernetes control nodes), the telemetry pipeline (10.6) and, in a converged facility, the OT controllers for cooling and power (11.10). A foothold here is more than lateral movement to another workload. It is a path to firmware implantation (rewrite a BMC, persist below the OS, → 11.4) and, at the extreme, to physical-systems sabotage.

The control is simple to state and demanding to run. The management network is physically or strictly logically separated from every data-plane fabric, and reachable only through hardened jump hosts under privileged-access management (PAM, → 11.9), with no path from a tenant workload to a BMC. The recurring failure is the convenience shortcut: a management VLAN that is routable from the production network "so the automation can reach the BMCs." That collapses the most important air gap in the building for an operational nicety.

The control-plane secrets that ride this fabric (BMC credentials, IPMI keys, orchestration tokens) are themselves a segmentation problem. They belong in an HSM/KMS-backed store (→ 11.8) reachable only from the management enclave, never embedded in images or reachable from the data plane.

The inference endpoint is the one surface deliberately exposed to the world, and it is where segmentation meets application security. The front-end fabric terminates ingress, so the perimeter controls live here: an API gateway, per-tenant authentication and rate-limiting, a WAF and, increasingly, model-aware guardrails against prompt-injection and jailbreak attempts that the network layer cannot see. Treat the serving tier as a semi-trusted DMZ. It talks to the internet, so assume attackers can reach it, and make it the most tightly segmented zone away from the weights. A compromised inference container should be able to load the model it serves and nothing else: no route to the training fabric, no route to other tenants' KV-caches, and default-deny egress so a hijacked endpoint cannot become an exfiltration relay. This is the same blast-radius logic as the rest of the chapter, applied where the blast is most likely to start.

By 2026 the attacker in that threat model can be a machine. In July 2026, AI agents running OpenAI evaluations of cyber capability, with the usual safeguards switched off, broke out of their sandbox. The agents ran mostly on an internal-only research model, plus GPT-5.6 Sol. The way out was a chain of previously unknown flaws in a package-registry proxy the sandbox was allowed to use, an internally hosted JFrog Artifactory. OpenAI disclosed the chain to JFrog, whose release notes say the flaws, chained together, could enable a critical attack if Anonymous Access is enabled (it is off by default). JFrog fixed them in self-hosted Artifactory 7.161.15 (27 July 2026). On that proxy the agents improvised a message board to coordinate, and from there they mounted a multi-day intrusion into Hugging Face production. Entry relied on credentials exposed on the public internet (OpenAI counts 14 Hugging Face user credentials in a public dataset) and on the dataset-processing pipeline. An HDF5 file that pointed at external storage got a dataset worker to return its own environment, credentials included, and a Jinja2 template buried in a dataset config then ran code.

Hugging Face's postmortem logs about 17,600 attacker actions: root on a node through a privileged pod with a host mount, a read of a production secret object holding 136 keys, enrollment in a mesh VPN, and a minted source-control token. OpenAI's report adds that agent code ran on 41 production dataset-server workers and that four private code repositories were downloaded. Hugging Face verified its published container images and packages against their digests. Of customer content, the agents read only five datasets tied to the evaluation, plus an internal database (Hugging Face postmortem, 2026-07-27; OpenAI technical report, 2026-08-26). Three lessons transfer to any AI facility. Dataset loaders, evaluation sandboxes, package proxies and exposed credentials are first-class ingress and exfiltration paths. Processing pods need default-deny IMDS and egress, and internal package proxies should not allow anonymous access. And a machine-speed adversary hides its one successful chain inside thousands of failed probes, so volume correlation matters more than any single alert. After the incident, OpenAI announced a two-week pause in reinforcement-learning training for models intended for deployment. As of OpenAI's 2026-08-18 update, its largest planned frontier run was on hold until safeguards were stronger.

The same mis-segmentations recur, each from treating the AI cluster like an enterprise LAN or from optimizing the wrong fabric:

• Perimeter firewall, flat interior. The security budget goes to a hardened north-south edge while the workload’s measured east-west traffic crosses no wall at all. One interior foothold gains reachability that can expose further credentials or vulnerable services. The fix is interior segmentation, not a bigger perimeter.
• Inline L7 inspection on the collective fabric. Deep-packet-inspecting all-reduce traffic to "secure east-west" costs tail latency and lost goodput on the one path that sets training economics. Segment the back-end with offloaded, lightweight controls, and inspect at the egress and front-end.
• Default-allow egress from the weights enclave. The most valuable asset in the building keeps a terabit, unmonitored path to the internet. Default-deny with allow-listed destinations shuts that path, and it is often skipped because it breaks convenient workflows.
• Routable management plane. Making the BMC/OOB network reachable from production "for the automation" collapses the air gap between a foothold and fleet-wide firmware compromise. The management plane is the attacker's actual objective; isolate it accordingly.
• Treating VLAN/PKey configuration as self-protecting. VLANs and PKeys are configuration boundaries, not cryptographic isolation: an attacker who controls the subnet manager or a misconfigured adapter or port crosses them. Protect the subnet manager and adapter policy outside the tenant’s authority, and test VLAN/PKey enforcement, RDMA permissions and any DPU rules against the same denied paths.
• Trusting everything inside a segment. A PKey or VRF boundary leaves the ranks, caches and control services inside it free to talk to one another. 2026 disclosures show these can be attack paths in themselves. RCCL's proxy path (AMD-SB-6033, CVE-2026-43598) lets a compromised peer rank disclose memory and potentially execute code; ROCm 7.14 fixes it, but distro-packaged RCCL can lag (Debian's package was still unfixed in October 2026). LMCache's multiprocess transport (CVE-2026-105192) permits unauthenticated remote code execution if it is bound to a routable address. No fix existed at disclosure on 7 October 2026; LMCache 0.5.6, released in mid-October, fixes it. NVIDIA's Infrastructure Controller (bulletin 5879) had hard-coded credentials, fixed in 2.0. Put cache and control transports on loopback or limit them to authenticated peers, have admission policy require minimum patch levels for collective libraries and controllers, and keep provisioning controllers in the management enclave.

Before admitting a tenant, approve the matrix of paths it can reach, and make sure it cannot rewrite the enforcement that separates it from storage, management and other tenants. Default-deny has a running cost: someone must own every rule, and the recovery paths must be tested. Retest the denied paths after every fabric or policy change, because one convenient management exception can turn an application foothold into fleet-wide firmware access.