Security Operations, Detection & Incident Response

The preceding eleven chapters of Part 11 build a fortress: a threat model and a target Weights Security Level (Chapter 11.1), concentric physical zones (Chapter 11.2), a hardware root of trust and firmware integrity (Chapter 11.4), GPU confidential computing (Chapter 11.5), tenant isolation (Chapter 11.6), zero-trust segmentation (Chapter 11.7), weight protection (Chapter 11.8), insider controls (Chapter 11.9), OT hardening (Chapter 11.10), and a compliance program that audits all of it (Chapter 11.11). Every one of those controls is a hypothesis: that the adversary will be stopped here, slowed there, made noisy somewhere. Security operations is the function that runs the experiment. It is where prevention's assumptions meet a live adversary, where the residual risk that every control chapter explicitly leaves on the table gets detected, contained, and recovered — or does not.

This chapter is the operational close of Part 11. It treats four decisions and their consequences: the SOC architecture for a campus where IT, OT, and physical security have converged into one attack surface; detection coverage and telemetry at scale, where the volume of fabric and GPU signal defeats a naive SIEM and the most valuable asset emits the least telemetry; the incident-response playbooks for the AI-specific scenarios that generic IR plans never anticipated — weight theft, firmware implant, kinetic strike, isolation breach, cyber-physical OT attack — including forensics on confidential systems that are designed to be opaque even to their operator; and the resilience, red-teaming, threat-intelligence, and metrics program that keeps the whole apparatus honest. The unifying theme is convergence: the AI campus is the first environment where a cyber intrusion can become a kinetic, life-safety, and grid-stability event in the same incident, and the SOC must be built to see that transition before it happens.

The first and most consequential decision is organizational, not technical: does one incident-command structure watch IT security, OT/facility security, and physical security together, or do three separate towers each watch their slice? The legacy answer — separate, because the skills, the tooling, and the reporting lines are different — is exactly wrong for an AI campus, and the reason is structural. The attacks that define this environment do not respect the tower boundaries. A firmware implant on a CDU controller (an OT event) is the prelude to thermal runaway that throttles or destroys a $3M+ rack of allocation-constrained silicon (an IT-asset and availability event). A drone strike on a substation (a physical event) is increasingly paired with a cyber intrusion timed to the chaos. A badge-reader compromise (a physical-access-control event that is really an OT/IT event, because the readers are IP-connected) is an insider's path to a cage. The highest-consequence incidents on an AI campus are precisely the ones that fall into the seams between isolated towers — and a seam is where detection dies.

The converged model puts IT, OT, and physical telemetry into a shared detection and incident-command plane, with one duty officer who can correlate a control-plane anomaly against a power-cap deviation against a door-forced alarm and recognize them as one attack. The cost is real: OT analysts and IT analysts have genuinely different instincts (an OT engineer's reflex is 'do not touch the running process'; an IT analyst's reflex is 'isolate and reimage'), and forcing them into one room without a deliberate operating model produces friction, not coverage. The honest middle path most large operators are converging on is a converged detection plane with federated response: shared visibility and a single incident commander, but OT actions still gated through OT-qualified operators who understand that the wrong containment move on a cooling loop is itself a destructive act. The canonical incident-command structure that this SOC plugs into — the roles, the authority ladder, the unified command for cyber-physical events — is built out in Chapter 14.11; this chapter is the security-detection feed into it.

A SOC detects only what it can see, and an AI campus presents a detection surface that is simultaneously enormous and blind in the worst places. Enormous, because a 100k-GPU cluster generates fabric telemetry, GPU and BMC health streams, firmware-attestation records, scheduler events, and OT controller logs at a volume that overwhelms a naive 'log everything into the SIEM' approach — the ingest and retention bill alone forces hard choices about what you keep and for how long. Blind in the worst places, because the single most valuable asset on the campus — the model weights — is, by design, the asset that emits the least useful telemetry: it lives encrypted at rest, decrypts only inside an attested TEE, and the confidential-computing boundary that protects it (Chapter 11.5) is opaque to the very monitoring the SOC would use to watch it. You are asked to detect theft of the thing you are contractually forbidden from observing. That paradox is the defining feature of AI-campus detection engineering.

The resolution is to move detection to the perimeters of the opaque core, not into it. You cannot watch a weight decrypt inside a TEE, but you can watch the egress choke point the weight must cross to leave (the anti-exfiltration linchpin of Chapter 11.7 and Chapter 11.8): a frontier model is hundreds of gigabytes to terabytes, and that mass has to traverse a network boundary, which is detectable as a byte-budget anomaly even when the payload is encrypted. You cannot trust a GPU's self-report if its firmware is compromised, but you can demand continuous attestation against NRAS/RIM golden measurements and alert on a measurement that drifts from baseline (the firmware-integrity discipline of Chapter 11.4). You cannot see inside a confidential VM, but you can watch the scheduler and the access plane for the human and machine behaviors that precede an exfiltration: a privileged account touching a weight store it has never touched, a job submitted to read every shard, a bulk-copy pattern against the storage fabric. The detection strategy is to ring the opaque asset with observable behavior and treat any anomaly at the ring as the signal.

Two scale realities shape the telemetry budget. First, retention is a security control with a price tag. Forensics on a nation-state intrusion routinely requires reconstructing activity from months ago — Volt Typhoon-class actors pre-position in critical infrastructure and dwell quietly, with the explicit goal of remaining undetected for as long as possible (CISA/Microsoft, 2023–2026). The industry's measured breach lifecycle — mean time to identify and contain — sat at 241 days in 2025, the lowest in nine years but still eight months of adversary dwell (IBM Cost of a Data Breach, 2025). If your fabric-flow and access-plane logs roll off at 90 days to save money, you have guaranteed that the most serious intrusions will be un-investigable by the time you notice them. Second, detection at cluster scale must be tiered: high-cardinality, high-volume streams (per-flow fabric records, per-GPU health) are summarized and sampled at the edge and only escalated to full retention on anomaly, while low-volume high-value streams (attestation failures, egress threshold breaches, OT control changes) are kept verbatim and long. Getting this tiering wrong in either direction — keeping everything and going bankrupt, or keeping nothing and going blind — is the most common SOC-scaling failure on an AI campus.

A generic enterprise IR plan — built around the NIST incident-handling lifecycle and now restructured by NIST SP 800-61r3 (April 2025) to map onto the CSF 2.0 functions of Govern, Identify, Protect, Detect, Respond, and Recover — is necessary but radically insufficient here. The AI campus has incident classes that no commodity playbook anticipates, and each one inverts a default IR assumption. The five that matter, with the fork each forces:

• Weight theft. The crown-jewel scenario. The default IR move — isolate the affected host and preserve it for forensics — is too slow when the asset is a file that can be exfiltrating in real time. The playbook fork is detect-then-contain at the egress, not the host: throttle or sever the egress path the moment a byte-budget anomaly fires, accept the goodput loss of cutting legitimate traffic, and reconstruct the access chain from the access-plane and scheduler logs you (hopefully) retained. The reference framework for what 'theft' means and which adversary tier you are defending against is RAND's Weights Security Level model (Chapter 11.1, Chapter 11.8).
• Firmware implant. An implant in GPU, BMC, or NIC firmware survives reimaging and can forge its own health reports. The default 'wipe and restore from backup' does not help if the backup or the supply chain is the vector. The fork is to treat any attestation-chain anomaly as a compromise of the hardware root of trust itself: quarantine the node from the fabric, re-attest from the silicon RoT (Caliptra/DICE, Chapter 11.4), and — if the measurement cannot be re-established — physically retire the unit, because you can no longer trust anything it reports.
• Kinetic / drone strike. The March 2026 IRGC drone strikes on AWS facilities in the UAE and Bahrain moved aerial attack from tail-risk to design case (domain research, 2026); most legacy physical controls are built for ground intruders, not aerial ones. The fork here is that the security incident and the facility incident are the same incident — structural and power damage demand the unified cyber-physical command (Chapter 14.11), and the SOC's job is to determine whether the kinetic event is cover for a simultaneous cyber intrusion.
• Isolation breach. A documented multi-tenant escape — the 2025 vGPU and uncore-side-channel CVE class (Chapter 11.6) — means tenant A read tenant B's data or denied them service. The fork is blast-radius first: identify every tenant who shared the affected isolation boundary, not just the two named in the alert, because a side-channel does not respect the alert's scope.
• Cyber-physical / OT attack. The most dangerous class, treated at length in Chapter 11.10: a compromised control plane forcing a synchronized load step to trip the grid, disabling CDUs to cook a hall, or inducing BESS runaway. This is the one incident where the playbook must accept evidence destruction to protect life and the grid — the escalation trigger above.

Confidential computing creates a forensics paradox that is unique to this era. The whole point of a GPU TEE is that no one — not the cloud operator, not the SOC, not a privileged insider — can observe the workload's memory: NVIDIA's Compute Protected Region encrypts roughly 90% of GPU memory and the BAR0 decoupler hides ~99.78% of registers in confidential mode (vs ~7.94% in normal mode) (NVIDIA / arXiv 2507.02770, 2025). That is exactly the property a tenant pays for, and exactly the property that blinds the responder. When the incident is inside the confidential boundary, the operator's normal forensic toolkit — memory capture, process inspection, packet capture of the workload — is structurally unavailable. You cannot dump the memory of a TEE you do not hold the keys to.

The forensic strategy must therefore be designed before the incident, around the boundary rather than across it. Three principles. Attestation logs are the forensic record. The 5-certificate device-identity chain and the 64 structured measurement records validated against NRAS/RIM (domain research, 2026) are, in a confidential incident, often the only operator-side evidence of what ran and whether the platform was intact — so they must be collected, signed, and retained as first-class forensic artifacts, not transient health checks. Metadata leakage is your friend here. The same residual metadata that is a confidentiality weakness — plaintext queue headers and physical-address tables, the bounce-buffer staging across the untrusted PCIe boundary — is also the operator's only observability into an otherwise opaque workload, and the SOC should be instrumented to capture it. The contract is the constraint. Multi-tenant confidential workloads come with contractual and regulatory limits on what the operator may inspect; the IR plan must pre-negotiate, with tenants and with legal, exactly what evidence the operator is permitted to collect and under what incident conditions the TEE may be torn down — because the moment to discover you are not allowed to image a tenant's enclave is not during the breach. Forensic readiness on confidential systems is a design decision made at onboarding (Chapter 10.10 on data governance; isolation and CC internals in Chapter 11.5 and Chapter 11.6), not a capability you can improvise.

Detection and response are the reactive half of security operations; the proactive half is the program that finds the gaps before the adversary does and measures whether the whole apparatus is improving. Four functions.

Resilience and continuity. The SOC is itself an asset an adversary will target — a campaign frequently begins by blinding or degrading the defenders. The operational-security plan must assume the SOC's primary tooling, its telemetry pipeline, or its incident-command channel could be the thing under attack, with out-of-band communications and a documented fallback to manual operation. This dovetails with the facility-resilience program of Part 12 (Chapter 12.3 on disaster recovery and continuity): a security incident and a reliability incident increasingly share a root cause and a recovery path, and the runbooks should be co-designed rather than maintained in separate binders.

Red-teaming. The only honest test of the Part 11 fortress is an adversary actually attacking it. AI-campus red-teaming must exercise the seams the converged SOC was built to cover: a purple-team exercise that starts with a simulated firmware anomaly and walks it through to a cyber-physical escalation tests the detection plane, the playbook, the escalation trigger, and the incident commander in one run. The output is a set of detection gaps and a measured time-to-detect for each attack path, fed back into the telemetry-coverage map above, rather than a vulnerability list.

Threat intelligence. Generic threat feeds under-serve this environment; the relevant intelligence is AI-infrastructure-specific. The dominant nation-state pattern is pre-positioning: living-off-the-land actors using legitimate built-in tools to evade detection and dwell for the long term in critical infrastructure, with the goal of disruption during a future conflict rather than immediate theft (CISA/Microsoft on Volt Typhoon, 2023–2026). That threat model — a patient, stealthy, OT-aware adversary already inside the walls — is what justifies the long retention, the attestation-drift alerting, and the OT telemetry that a theft-focused program would skip. Threat intelligence about which frontier labs and which silicon are being targeted feeds directly into the target Weights Security Level (Chapter 11.1).

Metrics. What you measure the SOC against determines what it optimizes, and the wrong metric is expensive. Borrowing the facility's availability nines as the SOC's target rewards uptime and ignores the slow, silent exfiltration that never causes an outage. The right metric set is detection-and-response-centric — MTTD and MTTC per attack class, detection coverage as a fraction of the mapped attack surface, percentage of playbooks rehearsed in the last quarter — and, in keeping with this guide's recurring lens, goodput preserved per incident: a fast, surgical containment that severs one tenant's egress and keeps the cluster running preserves far more return than a slow, blunt response that halts the campus. The SOC that measures itself on goodput-preserved rather than incidents-closed makes the right trade between containment and continuity, which is the trade this entire chapter has been about.