Security Operations, Detection & Incident Response

The eleven chapters before this one build a fortress: a threat model and a target Weights Security Level (Chapter 11.1), concentric physical zones (Chapter 11.2), supply-chain provenance (Chapter 11.3), a hardware root of trust and firmware integrity (Chapter 11.4), GPU confidential computing (Chapter 11.5), tenant isolation (Chapter 11.6), zero-trust segmentation (Chapter 11.7), weight protection (Chapter 11.8), insider controls (Chapter 11.9), OT hardening (Chapter 11.10) and a compliance program that audits all of it (Chapter 11.11). Each control is a hypothesis: the adversary will be stopped here, slowed there, made noisy somewhere. Security operations is the function that runs the experiment. It is where the residual risk that every control chapter leaves on the table gets detected, contained and recovered, or does not.

This chapter is the operational close of Part 11. It covers four sets of decisions:

• The SOC architecture for a campus where IT, OT and physical security have converged into one attack surface.
• Detection coverage and telemetry at scale, where fabric and GPU volume defeats a naive SIEM and the most valuable asset emits the least telemetry.
• The incident-response playbooks for AI-specific scenarios that generic IR plans never anticipated: weight theft, firmware implant, kinetic strike, isolation breach, cyber-physical OT attack and the machine-speed agentic intrusion. This includes forensics on confidential systems, which are designed to be opaque even to their operator.
• The resilience, red-teaming, threat-intelligence and metrics program that keeps the whole apparatus honest.

The through-line is convergence. On an AI campus one cyber intrusion can become a kinetic, life-safety and grid-stability event in the same incident, and the SOC must recognize that transition and escalate through the incident-command authority in Chapter 14.11.

The first and most consequential decision is organizational: does one incident-command structure watch IT security, OT/facility security and physical security together, or do three separate towers each watch their slice? The legacy answer, separate towers because skills, tooling and reporting lines differ, is wrong for an AI campus. The attacks that define this environment ignore tower boundaries. A firmware implant on a CDU controller (an OT event) can drive a hall into thermal shutdown. It destroys GB200 NVL72 racks (about $3.1M each, server only, in the August 2025 model) only if the GPU and BMC thermal protections are also defeated or a pressure transient damages the loop (Chapter 11.10). Either way it is an IT-asset and availability event. Kinetic and cyber attacks can also coincide. Mandiant (2023) assessed that Sandworm's October 2022 OT attack likely tripped the breakers at a Ukrainian substation as Russia opened a wave of missile strikes on infrastructure; two days later it wiped the victim's IT systems. Mandiant had too little evidence to call the timing coordinated. A badge-reader compromise is a physical-access-control event that is really an OT/IT event, because the readers are IP-connected, and it is an insider's path to a cage. The highest-consequence incidents on an AI campus are the ones that fall into the seams between isolated towers, and a seam is where detection dies.

The converged model puts IT, OT and physical telemetry into a shared detection and incident-command plane. One duty officer can correlate a control-plane anomaly, a power-cap deviation and a door-forced alarm and recognize them as one attack. The cost is real. OT and IT analysts have different instincts (the OT engineer's reflex is 'do not touch the running process'; the IT analyst's is 'isolate and reimage'), and putting them in one room without a deliberate operating model produces friction, not coverage. The middle path is a converged detection plane with federated response: shared visibility and a single incident commander, but OT actions still gated through OT-qualified operators, who know that the wrong containment move on a cooling loop is itself a destructive act. The incident-command structure this SOC plugs into (the roles, the authority ladder, the unified command for cyber-physical events) is built out in Chapter 14.11. This chapter is the security-detection feed into it.

A SOC detects only what it can see, and an AI campus has a detection surface that is both enormous and blind in the worst places. It is enormous because a 100k-GPU cluster generates fabric telemetry, GPU and BMC health streams, firmware-attestation records, scheduler events and OT controller logs at a volume that overwhelms a 'log everything into the SIEM' approach. The ingest and retention bill alone forces hard choices about what to keep and for how long. It is blind in the worst places because the most valuable asset on the campus, the model weights, emits the least useful telemetry. The weights are encrypted at rest and decrypt only inside an attested TEE, and the confidential-computing boundary that protects them (Chapter 11.5) is opaque to the monitoring the SOC would use to watch them. You are asked to detect theft of the thing you are contractually forbidden from observing, which is the defining constraint of AI-campus detection engineering.

The resolution is to move detection to the perimeter of the opaque core. You cannot watch a weight decrypt inside a TEE, but you can watch the egress choke point it must cross to leave (the anti-exfiltration linchpin of Chapter 11.7 and Chapter 11.8). A frontier model is hundreds of gigabytes to terabytes, and that mass has to traverse a network boundary, where it shows up as a byte-budget anomaly even when the payload is encrypted. You cannot trust a GPU's self-report if its firmware is compromised, but you can demand continuous attestation against NRAS/RIM golden measurements and alert on a measurement that drifts from baseline (the firmware-integrity discipline of Chapter 11.4). You cannot see inside a confidential VM, but you can watch the scheduler and the access plane for the behaviors that precede an exfiltration: a privileged account touching a weight store it has never touched, a job submitted to read every shard, a bulk-copy pattern against the storage fabric. The strategy is to ring the opaque asset with observable behavior and treat any anomaly at the ring as the signal.

The AI software control plane belongs inside that ring too. Two critical AI-infrastructure flaws from 2026 (both CVSS 9.8) were in provisioning software for bare-metal servers and in a KV-cache transport, not in the GPU. NVIDIA Infrastructure Controller (CVE-2026-65113) shipped with hard-coded credentials, fixed in version 2.0. LMCache (CVE-2026-105192) allowed unauthenticated remote code execution over its multiprocess transport, and is reachable where multi-node setups bind that transport to a routable address (NVIDIA bulletin 5879; JFrog, October 2026). It had no fix at disclosure on 7 October 2026; LMCache 0.5.6, released in mid-October, fixes it. A compromised provisioning controller crosses every tenant boundary at once, and these services usually sit on 'trusted' internal networks that nothing logs. Alert on connections from unexpected peers, version drift from the patched baseline and admin actions outside change windows (Chapter 11.6, Chapter 11.7).

Two scale realities shape the telemetry budget. First, retention is a security control with a price tag. Reconstructing a nation-state intrusion routinely means tracing activity that began months, sometimes years, earlier. Volt Typhoon-class actors pre-position in critical infrastructure and stay quiet on purpose: in February 2024 CISA reported that Volt Typhoon had held footholds in some victims' IT environments for at least five years. IBM's 2026 report puts the mean time to identify and contain a breach at 247 days, and the average breach it studied cost $4.99M globally and $11.5M in the US (IBM Cost of a Data Breach, 2026). That cost, not the SIEM's storage rate, is what a retention budget should be set against. If your fabric-flow and access-plane logs roll off at 90 days to save money, an investigation that starts later cannot use them to reconstruct earlier activity. Second, detection at cluster scale must be tiered. High-cardinality, high-volume streams (per-flow fabric records, per-GPU health) are summarized and sampled at the edge and escalated to full retention only on anomaly. Low-volume, high-value streams (attestation failures, egress threshold breaches, OT control changes) are kept verbatim and long. Getting the tiering wrong in either direction, keeping everything and going bankrupt or keeping nothing and going blind, is a classic SOC-scaling failure on an AI campus.

A generic enterprise IR plan is built around the NIST incident-handling lifecycle, now restructured by NIST SP 800-61r3 (April 2025) to map onto the CSF 2.0 functions of Govern, Identify, Protect, Detect, Respond and Recover. It is necessary but not nearly enough here. The AI campus has incident classes that no commodity playbook anticipates, and each one inverts a default IR assumption. Six now matter, and each forces its own fork:

• Weight theft. The crown-jewel scenario. The default IR move, isolating the affected host and preserving it for forensics, is too slow when the asset is a file that can be exfiltrating in real time. The fork is to detect and contain at the egress, not the host: throttle or sever the egress path the moment a byte-budget anomaly fires, accept the goodput loss of cutting legitimate traffic, and reconstruct the access chain from the access-plane and scheduler logs you (hopefully) retained. RAND's Weights Security Level model defines what 'theft' means and which adversary tier you are defending against (Chapter 11.1, Chapter 11.8).
• Firmware implant. An implant in GPU, BMC or NIC firmware survives reimaging and can forge its own health reports. 'Wipe and restore from backup' does not help if the backup or the supply chain is the vector. The fork is to quarantine first and diagnose second, because an attestation-chain anomaly has four outcomes and only one of them is a compromised root of trust: verification unavailable (verifier or RIM unreachable), policy mismatch (a known but unapproved firmware measurement), evidence invalid (expired certificate, missing reference material) and confirmed compromise. Hold admission and key release in all four. For the first three, re-attest from the silicon RoT (Caliptra/DICE, Chapter 11.4) and restore an approved measured state; physically retire the unit only for the fourth. Know what a fresh quote proves, too: it re-signs the measurements that already exist, so it catches a changed boot-time image but not a runtime implant that never altered a measured object.
• Kinetic / drone strike. In March 2026 drone strikes damaged AWS facilities in the UAE and Bahrain. Media linked to the IRGC claimed the strikes. They were likely the first publicly confirmed military attack on a US hyperscaler's data centers. In a September 2026 status update, AWS said its Bahrain region cannot be restored. Data kept in only one UAE availability zone was unrecoverable too, because the damage went beyond what regional and multi-AZ services are built to withstand. Plan recovery around cross-region restore, not just AZ failover (Chapter 12.3). Aerial attack has moved from tail risk to design case, and most legacy physical controls assume ground intruders. The fork is that the security incident and the facility incident are the same incident: structural and power damage demand the unified cyber-physical command (Chapter 14.11), and the SOC's job is to determine whether the kinetic event is cover for a simultaneous cyber intrusion.
• Isolation breach. A shared-GPU boundary leaks or fails. NVIDIA's July 2025 vGPU bulletin lists the real cases. CVE-2025-23290 (low severity) let a guest read global GPU metrics that co-tenants influence. CVE-2025-23285 let a guest starve shared resources. CVE-2025-23283 and -23284 were vGPU Manager flaws (CVSS 7.8) that could let a guest execute code on the host. Uncore side channels add a separate route for tenants to leak data to each other, one that MPS and MIG partitioning does not close (Chapter 11.6). The fork is blast radius first: identify every tenant that shared the affected isolation boundary, not just the two named in the alert, because a side channel does not respect the alert's scope.
• Cyber-physical / OT attack. The most dangerous class, treated at length in Chapter 11.10: a compromised control plane can force a synchronized load step that trips the grid, shut down CDUs until rising temperature trips the whole hall, or drive a BESS into runaway. This is the one incident where the playbook must accept evidence destruction to protect life and the grid, which is the escalation trigger above.
• Agentic intrusion. The newest class, seen in production in July 2026. An autonomous agent (a frontier-lab eval escapee or an adversary-operated one) attacks at machine speed and machine volume, so the fork is preparation, not reaction. Pre-position an on-prem forensic model: Hugging Face reconstructed the July 2026 campaign with open-weight GLM-5.2 after hosted frontier models refused the analysis as "cyber". Correlate on action volume rather than single alerts, and treat dataset, eval and package-proxy surfaces as the ingress tier.

The other clock. Most legal notification deadlines (NIS2, CIRCIA, CRA) start when you become aware of an incident, and the SEC's starts at the materiality determination; all of them run alongside containment, not after it. NIS2, as national laws transpose it, gives EU data-center service providers 24 h for an early warning, 72 h for a notification and one month for a final report. US-listed companies have four business days to file SEC Form 8-K Item 1.05 once they judge an incident material. CIRCIA's statutory 72 h incident report and 24 h ransom-payment report apply only after CISA publishes its final rule, which had not happened by mid-October 2026. Manufacturers run a clock of their own. Since 11 September 2026 the EU Cyber Resilience Act has made them report any product vulnerability that attackers are actively exploiting, and any severe incident affecting a product, in three steps: an early warning within 24 h, a notification within 72 h, and a final report 14 days after a fix or mitigation is available for an exploited vulnerability, or one month after the notification for a severe incident (Chapter 11.11). Write into supplier contracts who notifies whom, and that the supplier shares its final report, which is where the root cause appears. Name a legal and notification owner in the incident-command roster, and make starting the clock a triage step.

Confidential computing creates a forensics paradox unique to this era. The point of a GPU TEE is that no one, not the cloud operator, not the SOC, not a privileged insider, can observe the workload's memory. NVIDIA's Compute Protected Region places most GPU memory (the paper's example is 90%) behind hardware access control. HBM is access-controlled, not encrypted. Encryption covers the transfers across the untrusted host boundary and, on Blackwell, NVLink between the GPUs of one confidential VM. In confidential-computing mode, 99.78% of the GPU's memory-mapped register fields read back as zero through the BAR0 decoupler (an H100 measurement, arXiv 2507.02770, 2025). That is the property a tenant pays for, and it is the property that blinds the responder. When the incident is inside the confidential boundary, the operator's normal forensic toolkit (memory capture, process inspection, packet capture of the workload) is unavailable by design. You cannot dump the memory of a TEE when the hardware will not let you read it.

Design the forensic strategy before the incident, around the boundary rather than across it. Three principles apply. Attestation logs prove platform state. The GPU device-certificate chain and RIM-validated measurements prove device and platform integrity, not the tenant application, model or job. To establish what ran, preserve separately signed scheduler and job records, workload manifests and hashes, CVM/CPU attestation where applicable, image provenance, key-release decisions and tenant-authorized application telemetry. Metadata leakage is your friend here. The residual metadata that weakens confidentiality (plaintext queue headers and physical-address tables, the staging of encrypted bounce buffers across the untrusted PCIe boundary) is also the operator's only observability into an otherwise opaque workload, so instrument the SOC to capture it. The contract is the constraint. Multi-tenant confidential workloads carry contractual and regulatory limits on what the operator may inspect. The IR plan must pre-negotiate with tenants and with legal exactly what evidence the operator may collect and under what incident conditions the TEE may be torn down, because the moment to discover you may not image a tenant's enclave is not during the breach. Forensic readiness on confidential systems is a design decision made at onboarding (Chapter 10.10 on data governance; isolation and CC internals in Chapter 11.5 and Chapter 11.6), not a capability you can improvise.

Detection and response are the reactive half of security operations. The proactive half finds the gaps before the adversary does and measures whether the whole apparatus is improving. It has four functions.

Resilience and continuity. The SOC is itself an asset an adversary will target, and a campaign frequently begins by blinding or degrading the defenders. Plan for the SOC's primary tooling, its telemetry pipeline or its incident-command channel to be the thing under attack, with out-of-band communications and a documented fallback to manual operation. This dovetails with the facility-resilience program of Part 12 (Chapter 12.3 on disaster recovery and continuity): a security incident and a reliability incident increasingly share a root cause and a recovery path, so co-design the runbooks instead of keeping them in separate binders.

Red-teaming. The only honest test of the Part 11 fortress is an adversary actually attacking it. AI-campus red-teaming must exercise the seams the converged SOC was built to cover. A purple-team exercise that starts with a simulated firmware anomaly and walks it through to a cyber-physical escalation tests the detection plane, the playbook, the escalation trigger and the incident commander in one run. The output is a set of detection gaps and a measured time-to-detect for each attack path, fed back into the telemetry-coverage map above, rather than a vulnerability list.

Threat intelligence. Generic threat feeds under-serve this environment; the relevant intelligence is specific to AI infrastructure. The dominant nation-state pattern is pre-positioning: living-off-the-land actors use legitimate built-in tools to evade detection and dwell for the long term in critical infrastructure, aiming at disruption during a future conflict rather than immediate theft (CISA/Microsoft on Volt Typhoon, 2023–2026). An adversary that patient, stealthy and OT-aware, already inside the walls (see the years-long footholds under retention), is the reason for this program's long log retention, its alerts on attestation drift and the OT telemetry it collects, which a theft-focused program would skip. Intelligence about which frontier labs and which silicon are being targeted feeds directly into the target Weights Security Level (Chapter 11.1).

Metrics. What you measure the SOC against determines what it optimizes, and the wrong metric is expensive. Borrowing the facility's availability nines as the SOC's target rewards uptime and ignores the slow, silent exfiltration that never causes an outage. Measure detection and response instead: MTTD and MTTC per attack class, detection coverage as the fraction of mapped techniques that have a tested detection, and the percentage of playbooks rehearsed in the last quarter. Map techniques against public taxonomies: MITRE ATT&CK for the IT estate, ATT&CK for ICS for the controllers in Chapter 11.10, and MITRE ATLAS for attacks on models and training pipelines. A shared taxonomy is what lets this quarter's red-team result be compared with last quarter's. Add this guide's recurring lens, goodput preserved per incident. A fast, surgical containment that severs one tenant's egress and keeps the cluster running preserves far more return than a slow, blunt response that halts the campus. A SOC that measures itself on goodput preserved rather than incidents closed makes the right trade between containment and continuity.

The sharing layer gained an AI-specific venue in mid-2026. NVIDIA launched OSAA, the Open Secure AI Alliance, on 2026-07-27, a few days after the Hugging Face agent intrusion. By Black Hat, press tallies put it past 120 members. Microsoft, CrowdStrike and Hugging Face were among them, along with Red Hat and the Linux Foundation. The alliance moved under Linux Foundation governance in September 2026. The proposed SAFE (Shared AI Findings Exchange) would give members a confidential channel for AI security incidents, near misses and control failures. Comments on the RFC closed on 2026-09-21; as of mid-October 2026 no working exchange had been announced, so watch what ships. NVIDIA open-sourced NOOA too. It is a framework for building agent harnesses that are easier to test, trace and audit. Despite the name, OSAA's SAFE is unrelated to the firmware-audit program OCP S.A.F.E. (Chapter 11.3).

Scope containment to the affected asset and the threat, keep the evidence that justified each action, and return capacity only through recovery and admission checks. Keeping a compromised path live buys continuity at the price of further exposure; isolating the whole campus without a scoped reason throws away useful output. Decisions that combine safety, continuity and customers go up through the incident command in Chapter 14.11 while cyber responders close off the compromised path.