Model & Weight Protection (At-Rest, In-Transit, In-Use)

Every other asset in an AI data center can be replaced. A failed GPU comes from the spare pool, a tripped transformer is re-energized, and a tenant's data is restored from backup. The weights are the exception. They are the distilled output of a training run that may have cost hundreds of millions of dollars and tied up an interconnection slot you cannot get back. Unlike hardware, they can be copied perfectly, instantly and silently. The crown jewels are anywhere from under one terabyte to a few terabytes of floating-point numbers. Once a copy leaves, you cannot claw it back, and you have no way to know for certain that it left.

The chapter follows the weights through three states: at-rest (on disk, in checkpoints, in object storage), in-transit (across the fabric, between regions, to and from storage) and in-use (decrypted into HBM, where a GPU can compute on it). It is organized by RAND's Securing AI Model Weights and its five Weights Security Levels, the one published framework that ties controls to a named adversary tier rather than to a compliance checklist. Egress is the choke point, attestation-gated key release is the in-use answer, and key management and PKI are a discipline rather than a feature. The chapter closes on crypto-agility, which long-lived weights force on you whether you want it or not. Who is allowed to train on what is a different problem, covered in Chapter 10.10.

You cannot scope weight protection until you name the adversary, and the industry's reference for that is RAND's framework. It defines five Weights Security Levels (SL1–SL5) and five operational-capacity tiers of attacker, from OC1 amateurs to OC5 top-priority nation-state operations, and it catalogs 38 distinct attack vectors. RAND's August 2026 follow-up proposes a 262-control catalog for SL3 (see the ladder below). RAND defines each level by the attacker capability its safeguards are meant to thwart. SL2 addresses professional opportunistic efforts and explicitly excludes insiders; SL3 adds cybercrime syndicates and insider threats. Eight of the 38 vectors are likely infeasible for OC1–OC3 but likely feasible for OC4–OC5. Moving from opportunistic attackers to top-tier operations changes the kind of defense you need, not just its amount.

A cross-industry SL5 Task Force formed in 2025 in response to that gap. Its goal is frontier infrastructure that resists nation-state attackers by 2028/2029, and its SL5 standard is written as an overlay on NIST SP 800-53. In release 0.1.2 (June 2026) the standard picks 60 controls with long lead times, grouped into 12 families across network, physical, machine, personnel and supply-chain streams. The timeline runs years out for the same reason siting and interconnection do: the controls with the longest lead time (facility construction, hardware procurement, organizational capability) must be planned before the steel is cut. SL5 is a building decision, not a software patch, which is why it belongs in this guide and not in a security-tooling appendix.

At-rest protection is the part of the problem that mature practice has nearly solved, which is why operators over-index on it. Weights and checkpoints sit encrypted on disk and in object storage under envelope encryption. A data-encryption key (DEK) encrypts the bytes, a key-encryption key (KEK) held in an HSM or KMS wraps the DEK, and the HSM root never leaves tamper-resistant hardware. Take illustrative models of 175 billion and 1 trillion parameters. The 14-byte recipe of BF16 weights, FP32 master weights and two FP32 Adam moments gives exactly 2.45 TB and 14 TB (decimal), so inspect the real serializer before sizing encryption. At-rest encryption is bulk symmetric crypto (AES-256-GCM/XTS), and its cost is negligible next to the storage bandwidth you already provisioned for checkpointing (Chapter 9.4).

Size the theft target from the deployed weights, not the checkpoint. Without optimizer state, a 1-trillion-parameter model takes about 2 TB at BF16 precision. In 4-bit NVFP4 it takes roughly 0.6 TB, and more where some layers keep higher precision. The copies on inference servers are the smallest and the most numerous.

The fork that matters at rest is not whether to encrypt but where the key lives and who can release it. Disk-level transparent encryption with keys resident on the host protects against a stolen drive and nothing else: once an attacker has code execution on the host, the cleartext is theirs. The defensible posture binds DEK release to an attestation of the requesting node, so a checkpoint decrypts only on a machine whose firmware, kernel and GPU state match a known-good measurement. That turns at-rest protection from 'encrypted until someone logs in' into 'encrypted until a verified machine asks'. The in-use decision reuses the same machinery, so getting the key hierarchy right here pays off twice. At rest you have the most time and the fewest constraints, which makes it the place for your strongest cryptography and strictest access policy. The harder exposures come later, when the weights are decrypted for use and when they cross the egress path.

An egress-rate limit or upload limit buys response time on the paths it actually controls. Take assumed operands: a full 1,000 GB uncompressed artifact divided by 800 GB/day of aggregate allowed egress takes 1.25 days. That constructed ratio holds only for the complete payload on that one aggregate channel. A partial, compressed or adapted payload, parallel endpoints or a bypass can leak useful capability sooner. To give defenders time to detect and interdict, the cap has to be aggressive, and that is where it collides with the workload.

The collision is the inference-server paradox. A production inference fleet exists to emit token output to users outside, continuously, at a volume that scales with traffic and revenue rather than with any security budget. A cap below the required aggregate inference output breaks the revenue service. A cap above that demand still needs controls on payload content, bypasses and parallel endpoints. Egress limiting therefore works well for a training cluster, which has no reason to send weight-sized payloads outward, and poorly for an inference cluster whose normal behavior is high-volume output.

The 2026 frontier has a defender's side and an attacker's side. The defender's side is inference-output verification, which tries to show that what leaves is plausibly model output and not encoded model weights. In a March 2026 preprint it held the share of output usable for exfiltration under 0.5% on a 30B open-weight model, which would slow a thief by a factor of 200 or more. It remains research, not settled practice. The attacker's side is aggressive weight compression. Demonstrated work compresses weights to about 1.15 bit/parameter with minimal downstream loss, and exploratory results reach about 0.1 bit/parameter. In a theft context, that shrinks the payload and undercuts any fixed-rate cap. Egress limiting is necessary; on its own it is not sufficient.

In-transit protection on the wire is the least controversial of the three states: mutual TLS or IPsec for north-south and control-plane traffic, link-layer or fabric encryption for the high-bandwidth east-west path, and signed, hash-anchored manifests so a tampered checkpoint is detected before it is loaded. The decision is where the encryption boundary sits relative to the fabric you are trying to keep fast. Encrypting the back-end collective fabric, the non-blocking InfiniBand or Spectrum-X plane that exists for all-reduce bandwidth, adds latency and may not be terminable at line rate without offload. The common posture is therefore to encrypt storage and cross-region paths hard and rely on physical and segmentation controls (Chapter 11.7) for the in-rack scale-up fabric. The GPU TEE (below) helps only inside small groups. On Blackwell, confidential mode encrypts the NVLink traffic among as many as eight GPUs assigned to one confidential VM. A full NVL72 rack is not one CVM. Rack-scale confidential computing for Vera Rubin exists so far as an NVIDIA announcement rather than a measured capability (Chapter 11.5).

The subtler in-transit problem is provenance and integrity, not confidentiality. A weight blob that arrives intact but wrong (a poisoned checkpoint, a backdoored fine-tune, a substituted base model) is an integrity failure that encryption does nothing to catch. The control is a signed chain: every checkpoint and released weight artifact carries a cryptographic signature and a measurement that ties it to the training run and the build pipeline, validated before load. This is the measured-boot and golden-measurement machinery used for firmware integrity (Chapter 11.4), applied to the model artifact. Skip it and you have protected the weights from being read but not from being silently replaced, and for many threat models a replaced model is worse than a stolen one.

Treat external checkpoints as executable input. Pickle-based formats and conversion tools can execute code on load. In September 2026 NVIDIA fixed a batch of untrusted-deserialization bugs in its Megatron Bridge checkpoint-conversion library (CVE-2026-61750 and siblings). Accept weights in non-executable formats such as safetensors. Do conversions in an isolated environment that has no network path or key access, and sign the artifact into your registry only afterward.

This is the hardest state, and the real decision lives here: to compute on weights, a GPU must at some instant hold them in usable form in HBM. The classic posture is plaintext-in-use behind a perimeter. You decrypt the weights into HBM and trust that the host OS, hypervisor, BMC, network segmentation and personnel controls keep everyone else out. It is fast and simple and adds no TEE encryption path, but the cleartext sits in HBM for any principal whose privileges reach that memory. Against an OC4–OC5 adversary, or an insider with root, test which of those privileges the boundary actually excludes.

The alternative is attestation-gated key release into a GPU TEE. The weights stay encrypted until a confidential-computing enclave on the GPU proves, through hardware-rooted attestation, that its firmware and configuration match a known-good measurement. Only then does the KMS release the key, to the confidential VM. Plaintext then exists only within the CVM and in the GPU's protected memory, which neither root on the host nor the operator's other software privileges can read. Release weight keys only after the required CPU/CVM evidence and every assigned GPU's evidence pass, bound to the authorized tenant, the workload image and configuration, the challenge and the recipient key. Physical attacks on the host are a different problem. Chapter 11.5 owns them, along with the boundary, attestation, performance and residual side-channel details, while this chapter owns the key-release decision.

Every control above (at-rest envelope encryption, in-transit mTLS, attestation-gated in-use release) collapses to the same root: a key hierarchy and the discipline that operates it. Get the keys wrong and the strongest cryptography is one stolen credential away from irrelevant. The hierarchy is conventional in shape and unforgiving in operation. A hardware root of trust (an HSM cluster, FIPS 140-3 Level 3 or equivalent) holds the top-level KEKs, which never leave tamper-resistant silicon. Those wrap per-domain and per-tenant keys in a KMS, and those in turn wrap the DEKs that encrypt the weight blobs. Where the HSM sits is a fork. A network-attached cluster is the mature, certified option, but network latency and availability dependencies push teams to cache or export keys outside it, the limitation Microsoft cites for its workload-attached Azure Integrated HSM. Marvell, Microsoft and Oracle formed an OCP working group on 13 October 2026 to specify a FIPS 140-3 Level 3 in-server HSM, and Microsoft's Tessera project open-sources the Azure Integrated HSM firmware and software. The companies said an initial draft would appear at the October 2026 OCP Global Summit, with version 1.0 to follow, but no draft had been published by 19 October 2026. No product is certified against it, so judge an in-server HSM by its own FIPS certificate. Three decisions separate a real program from a checkbox: who can release, how fast you can revoke, and what a single compromise unlocks.

Authorization to release is where two-person control earns its keep. A weight-key release for the crown-jewel model should require an attestation and a policy that no single human or service credential can satisfy alone. Chapter 11.9 treats this separation-of-duties and least-privilege discipline as the dominant unaddressed vector.

Rotation and revocation at fleet scale is the operational nightmare nobody budgets for: re-wrapping DEKs across tens of thousands of accelerators, propagating a revocation before an attacker can use a leaked key, and doing it without stalling training goodput. A KMS that can issue a key in milliseconds but takes hours to revoke one across the fleet fails exactly when it matters. Be precise about what each operation buys. Re-wrapping an unchanged DEK protects nothing from someone who already holds that DEK, and a revoked grant cannot recall a key already copied. Compromise response is re-encryption of the affected ciphertext under a fresh DEK, denial of future release, and termination or bounded leasing of the jobs already holding plaintext, with a defined recovery objective and a rehearsed checkpoint/restart path (Chapter 12.3).

Blast radius is the last test. If one compromised KMS credential or one over-scoped IAM role can release every weight key, you have built a hierarchy with no interior walls, the same anti-pattern Chapter 11.7 warns about for the network, now applied to keys. The control-plane secrets that operate this machinery (KMS credentials, attestation-service trust anchors, signing keys) are crown-jewel-adjacent themselves and belong under the same custody discipline as the weights they protect (Chapter 10.6, Chapter 11.7).

The last decision is the easiest to defer and the most expensive to get wrong, because its consequence lands years after the choice. Frontier weights are long-lived secrets: a model trained in 2026 may keep strategic and economic value for a decade. The asymmetric cryptography that wraps key material today (RSA, ECC) is what a future cryptanalytically relevant quantum computer threatens, and the threat does not wait for that computer to exist. Harvest-now-decrypt-later means an adversary can exfiltrate today's RSA/ECC-protected key material and weight blobs now, store them, and decrypt them when the capability arrives. A one-week session token may expire, but captured traffic from that session can contain long-lived secrets, so token lifetime does not settle harvest-now-decrypt-later exposure.

Nobody can date that quantum computer, but the regulatory clock is already set. NIST published the post-quantum standards in August 2024: FIPS 203 (ML-KEM) for key establishment, and FIPS 204 (ML-DSA) and FIPS 205 (SLH-DSA) for signatures. A November 2024 draft, NIST IR 8547, sets out the transition: it proposes deprecating RSA and elliptic-curve schemes at 112 bits of security strength (the RSA-2048 class) after 2030 and disallowing all of them after 2035. NSA's CNSA 2.0 FAQ (December 2024) sets an earlier date for US National Security Systems, whose new acquisitions must be CNSA 2.0 compliant from 1 January 2027. A weight key wrapped with RSA or ECC in 2026 is still worth protecting when those algorithms are retired.

The decision is therefore crypto-agility now. Design the key hierarchy and the in-transit and at-rest envelope so the algorithms can be swapped without re-architecting, and run hybrid classical-plus-PQC during the transition. Move key establishment and key wrapping to ML-KEM first, because harvest-now-decrypt-later attacks the key exchange you run today. Signatures (ML-DSA) guard against a different failure: forged firmware, attestation or artifact signatures once a quantum computer exists. That threat starts when the computer does and cannot reach back to decrypt what was captured earlier.

Keys fused into silicon are the exception to putting signatures second. If a GPU, BMC or root of trust you buy in 2026 verifies firmware and signs attestation reports only with classical keys, it keeps doing so for its whole service life. Attestation-gated key release, this chapter's main in-use control, would then rest on classical signatures that a future quantum computer could forge. Ask each vendor when its hardware roots of trust move to post-quantum signatures. Buy HSMs, KMS and attestation services that support FIPS 203 and FIPS 204 now or by firmware update; one that cannot becomes a forced replacement before 2035.

The June 2026 release of the SL5 standard sets a stricter bar for the network. Inline network encryptors must run post-quantum algorithms, and each endpoint needs two encryptor layers in series whose algorithms rest on different hardness assumptions, so a single cryptanalytic break cannot open both. NIST's backup KEM, HQC (selected March 2025, built on error-correcting codes rather than lattices, final standard expected in 2027), is one candidate for that second family. Symmetric bulk encryption (AES-256) is already quantum-resistant enough at full key length, so the migration concentrates in the key-wrapping and signing layers, which is where attestation and PKI live. Deferring has one consequence you cannot undo: weights already harvested under classical crypto cannot be protected retroactively.

Climbing the Weights Security Level ladder is not a matter of buying a product. Each level demands controls with different lead times and irreversibility, and the most consequential ones are facility decisions that must be made before construction. In practice SL1–SL2 is operational hygiene you can add late, SL3 is a serious program you can retrofit with effort, and SL4–SL5 is a design basis you largely have to build in from the start.

• SL1 → SL2 (stop amateur and professional opportunistic efforts): at-rest encryption, RBAC and least privilege, MFA, audit logging, standard cloud hygiene. Cheap, late-bindable, reversible. This is the floor for any non-open-weight model.
• SL2 → SL3 (stop cybercrime and trained insiders): egress-rate limiting and upload caps, HSM-backed key custody, attestation-gated DEK release, two-person controls on weight-key release, behavioral monitoring, hardened offboarding. This is the first RAND level whose threat model includes insiders. It is mostly retrofittable, though the egress and key-custody architecture is easier built in. RAND's August 2026 SL3 report proposes 262 controls adapted from NIST SP 800-53. They target 31 high-feasibility attack vectors and are designed to be phased in over 6–12 months. Use that catalog, not this summary, as your checklist.
• SL3 → SL4 (stop standard operations by leading cyber-capable institutions): attestation-gated in-use protection as the default (GPU TEEs, confidential-by-default compute), hardware supply-chain assurance and firmware integrity below the root of trust, network microsegmentation enforced on the NIC, DPU or switch that carries each flow (Chapter 11.7), and personnel-security programs. This starts to dictate the hardware you buy and the topology you build, and it is increasingly irreversible.
• SL4 → SL5 (stop top-tier nation-state operations): release 0.1.2's 60 long-lead-time controls across 12 families, including air-gapped or strongly isolated compute for the highest-value weights, secure facilities, end-to-end supply-chain provenance, and organizational capability that takes years to build. The SL5 Task Force targets this level for 2028/2029 because, like interconnection, it is gated by lead time, not by will.

The recurring failure is the one that governs density and power: designing the irreversible substrate to today's level and being unable to reach tomorrow's. If there is any chance a facility will host weights that attract OC4–OC5 attention, reserve headroom at scoping time for confidential-compute capability, HSM placement, segmentation topology and secure-room provisions. Unlike egress policy, you cannot retrofit them after the weights arrive.