Data Governance, Privacy & the Training-Data Legal Regime

An AI data center turns data into a model, then turns user data into outputs. The rest of this guide treats data as a payload: bytes to store, move and compute on. This chapter treats it as a legal and regulated object. It has a provenance, an owner, a jurisdiction, a retention clock and a set of attached rights, and which rights apply depends on the operator’s purpose, role and jurisdiction. In 2025 these decisions became the difference between a defensible asset and a stranded one. A model trained on the wrong corpus can bring a settlement-scale bill, such as the $1.5 billion fund approved in Bartz v. Anthropic on 20 July 2026. A customer-data hall sited in the wrong country can be uncontractable. A deletion request you cannot honor can be a regulatory finding.

Five governance surfaces follow: training-data provenance and the copyright/fair-use frontier; the PII and privacy program; data residency by jurisdiction and cross-border transfer; retention, deletion and the right-to-erasure plumbing; and the customer-facing contract stack (DPAs, sub-processors, tenant isolation). Each legal determination needs an architecture that can enforce it. You cannot bolt residency onto a global fleet, or erasure onto a model with no lineage graph, after the concrete is poured and the weights are trained. This chapter governs the data. Protecting the model and its weights (at rest, in transit, in confidential execution) is a different threat model, covered in Chapter 11.8.

The legal regime around training data went from speculation to case law in 2025, and it is jurisdiction-specific in a way that breaks any single global policy. Four major regimes show the spread. The corpus has to survive the most demanding one its model will be deployed under.

The US: fair use, but the acquisition matters. The watershed is Bartz v. Anthropic. Judge Alsup’s June 2025 summary-judgment ruling split the question in two. Training an LLM that returns new text on lawfully acquired books was held "among the most transformative" uses imaginable, and so fair use under §107. But Anthropic had also downloaded millions of books from shadow libraries (LibGen, PiLiMi) to build a permanent "central library," and the transformative training did not excuse that acquisition.

The settlement is a minimum of $1.5 billion across roughly 500,000 works. That is approximately $3,000 gross per covered work before deductions, not a net claimant payout. It also requires destruction of the covered pirated LibGen/PiLiMi copies, subject to legal preservation. The court gave final approval and entered judgment on 2026-07-20. The release is narrow by design: it covers only past acquisition and copying (inputs) up to 2025-08-25. Claims over model outputs, and over conduct after that date, are not released. Two appeals have been filed, both by law firms left out of the attorney-fee award. They contest only that award, so the settlement itself proceeds. Class counsel’s status report to the court says first distributions to authors begin by 15 November 2026, for works with agreed rightsholders and complete payment details. Anthropic’s final $450 million installment is due by September 2027. It remains the most expensive lesson in the industry: under Bartz, fair use can protect the training act while the data-acquisition act remains independently infringing.

Two days after Bartz, a second judge in the same district reached fair use by a narrower road. In Kadrey v. Meta (25 June 2025), Judge Chhabria granted Meta summary judgment on training Llama with the plaintiffs’ books, but only because they had not shown market harm. He wrote that the ruling does not establish that Meta’s training is lawful, and that market dilution, a flood of AI-generated works competing with the originals, could win the market-harm factor on a better record. He also declined to treat the shadow-library downloads as a separate use, the opposite of Alsup’s approach. In June 2026 the plaintiffs sought leave for an early Ninth Circuit appeal of the downloading ruling. Judge Chhabria said no on 8 July 2026. He conceded the question is hard and that Alsup’s opinion has a point, but he expects to enter final judgment soon in at least one of his Meta cases, and the appeal can wait for it. The split between the two judges on downloading will reach the Ninth Circuit only after that judgment. The US Copyright Office’s Part 3 report on generative-AI training (pre-publication version, 9 May 2025) leans toward Alsup, but it folds acquisition into the fair-use analysis of training instead of treating it as a separate infringement. Knowingly training on works that were pirated or illegally accessed, the report says, “should weigh against fair use without being determinative.” For a Tier 2 corpus, a fair-use win on training is a finding on one record, so build the acquisition record that would satisfy the stricter judge.

The content classes keep widening. Round Hill Music sued both Anthropic and Suno in the Northern District of California on 2026-08-17. The Anthropic complaint pleads lyrics from at least 500 songs and says Round Hill will amend toward 10,000. Round Hill seeks up to $1B per case and states an intent to go to trial. This is a complaint, not a ruling, and publishers have been here before. Anthropic was sued over song lyrics in 2023 by Concord, Universal Music Publishing and ABKCO. For an operator the consequence is concrete. Provenance is per document, not per corpus, and the acquisition method is a load-bearing fact you must be able to prove for every source.

The UK: copies, not weights. In Getty Images v. Stability AI (UK High Court, November 2025), Getty abandoned its primary training-copy claim because it had no evidence that Stable Diffusion training and development took place in the UK. It lost on secondary infringement: on the evidence for the Stable Diffusion versions at issue, the court held that those final model weights were not infringing copies for the CDPA secondary-infringement claim. The narrow win that survived was a trademark finding for generated images bearing Getty’s watermark. The holding on the weights rests on that record and comes from a first-instance court, and Getty has permission to appeal it. Permission came in December 2025, and the Court of Appeal hearing is reported for November 2026. Liability for the primary training copies still turns on where they were physically made, so that exposure is decided by siting as much as by law.

The EU: a statutory regime, not a fair-use doctrine. The EU has no fair use. Its route is the text-and-data-mining (TDM) exceptions of the 2019 DSM Directive. For the Article 4 route, the exception allows effective rights reservations, including appropriate machine-readable reservations for online content (robots.txt, the TDM Reservation Protocol, metadata). Qualifying Article 3 research mining and licenses have different conditions. Article 4 permits mining of lawfully accessible works whose rightsholders have not reserved TDM rights. Honor the reservation, keep copies only while the mining needs them, and handle any personal data in the corpus under GDPR.

Article 4 may protect the mining step without protecting the model. In GEMA v. OpenAI (decided November 2025 and on appeal) and GEMA v. Suno (July 2026, appealable), Germany’s Munich Regional Court treated protected works memorized in model parameters as reproduced inside the model, which the TDM exception does not cover. The court enjoined the use, and in the Suno case that reached training done in the US. Both are first-instance rulings from Germany, and the CJEU has yet to rule on training and TDM for the first time (Like Company v. Google, C-250/25). Until it does, treat extractable memorization of protected works as an infringement risk in Germany and a live one elsewhere in the EU. Test for memorization before release, and file the dedup and extraction-test results next to the opt-out record.

Layered on top are the EU AI Act’s GPAI obligations, which have applied to new models since 2 August 2025. Providers within the applicable GPAI scope must (a) maintain a copyright policy that respects opt-outs and (b) publish a public summary of training content using the Commission’s mandatory template (finalized July 2025). The template requires detailed descriptions of public and private datasets, identification of large public datasets, and additional crawler, domain and collection details for scraped online sources, while protecting trade secrets. This makes "we don’t disclose our training data" a non-option for providers of GPAI models placed on the EU market (from 2 August 2025 for new models; by 2 August 2027 for models already on the market).

The defensive artifact that ties this together is dataset documentation: a per-corpus record of source, acquisition method, license terms, opt-out status, collection date and any filtering applied. Datasheets for datasets and data cards are no longer academic hygiene. They are what you produce when a plaintiff's expert asks how a specific book entered your weights, and they are the raw material for the EU training-content summary. An operator who cannot say where an arbitrary training sample came from, and whether the company was allowed to use it, has in effect chosen Tier 3 by omission.

Copyright governs creative works in the corpus. Privacy law governs personal data. The two regimes are orthogonal, so a corpus can be copyright-clean and privacy-dirty at the same time. Under GDPR and its global cousins (CCPA/CPRA, LGPD, PIPL, India’s DPDP), the binding questions are whether you have a lawful basis to process personal data for training, and whether the resulting model still "contains" that personal data.

The lawful-basis fork. The EDPB’s Opinion 28/2024 (17 December 2024) set the European frame. A controller that relies on legitimate interest (Art. 6(1)(f)) to train on personal data must pass a three-step test: a legitimate interest, the necessity of the processing, and a balance against data-subject rights. The Commission’s 2025 Digital Omnibus proposes writing this basis into the GDPR as a new Article 88c. As of October 2026 that amendment was still in negotiation, and the EDPB and EDPS say it would not remove the three-step test, so build the assessment to today’s text. The Opinion also sets a deliberately high bar for calling a trained model "anonymous": you must show that extracting personal data from the model, or regurgitating it, is not reasonably likely. The sting in the tail: if a model was trained on unlawfully processed personal data, that can taint the lawfulness of deploying the model, unless it has been genuinely anonymized. This is the privacy analogue of the Bartz acquisition problem. An upstream data sin can follow the model downstream.

The enforcement is already here. Italy’s Garante fined OpenAI €15 million in December 2024 for training ChatGPT without an adequate legal basis and for inadequate transparency, and ordered a six-month public-awareness campaign. The Court of Rome annulled that decision on 18 March 2026, but on jurisdictional grounds only. It held that the Irish DPC had become OpenAI’s lead supervisory authority in February 2024, before the Garante’s final ruling. The court never reached the merits, and the decision is appealable. Regulators are not waiting for the AI Act to mature before applying ordinary GDPR principles to training pipelines.

The engineering response is minimization at ingest, not cleanup at audit. The cheapest place to handle PII is before it enters the corpus: PII detection and redaction in the data pipeline, de-duplication (which both improves the model and reduces the memorization that drives privacy risk), and aggressive filtering of high-sensitivity categories. The expensive place is after training, when removing a person’s data may mean retraining. Build the privacy program as a pipeline stage with a documented lawful basis, a record of processing activities (ROPA) where Article 30 requires it, and a data-protection impact assessment (DPIA) where Article 35’s likely-high-risk test is met. One assessment can cover similar processing operations with similar risks. Keep per-run records of which assessment and lawful basis cover each run. A DPA will ask for the same artifacts, so produce them before the run, not after the complaint.

Residency is where data governance becomes a fleet-architecture decision. The question a customer or regulator asks sounds simple: "where does our data live and who can reach it?" The answer decides whether a workload can run on shared global silicon or must be pinned to in-region hardware.

The fork is logical versus physical residency. Logical residency is a contractual, software-enforced promise. Data is tagged with a region and the control plane routes it to in-region storage and compute, but the fleet underneath is global, and a misconfiguration can leak data across a border. Physical residency places processing and storage on silicon in the jurisdiction. Separate clusters, key custody and operational staff must also constrain remote administration, replication, support and backups, because the hall’s address alone does not keep bytes or access in-region. Physical residency is dramatically more expensive. It fragments your fleet, strands capacity in low-utilization regions and forecloses the global load-balancing that makes inference economics work. But sovereign, government and regulated-industry customers increasingly require it. Choosing it means a smaller, more expensive, less elastic deployment, and that choice cascades into siting (you need a hall in that jurisdiction) and capacity planning (you cannot pool demand across borders).

Cross-border transfer mechanisms. Where personal data subject to GDPR goes to a third country or international organization, Chapter V requires an applicable transfer route: an adequacy decision (the EU-US Data Privacy Framework, for participating US importers), Standard Contractual Clauses with a transfer impact assessment, or Binding Corporate Rules. China’s PIPL requires qualifying handlers to store covered personal information in China and pass a security assessment before exporting it. India’s DPDP §16, operationalised by Rule 15 of the DPDP Rules 2025 with effect from 13 May 2027, instead permits transfers unless the government restricts a destination, and stricter sectoral rules continue to apply. So "the model is served from a US region" can be a compliance event for an EU customer’s prompts, and the cross-border posture must be designed per data class, not per company. The deeper sovereignty stack (export controls, control of the stack, the geopolitics of where compute may sit) is a siting problem treated in Chapter 3.12. This chapter covers the customer-data residency obligations that sit on top of it.

The controller/processor line governs what you may do with the data at all. When you serve another company’s users, you are typically a processor. You may process their data only on documented instructions, and you may not reuse it for your own training merely because the contract is silent. Your own training purpose needs a controller-role assessment, a lawful basis, transparency and the necessary permissions. When you collect data for your own model, you are a controller with the full weight of lawful-basis, transparency and data-subject-rights obligations. Misplacing this line is the most common governance failure in multi-tenant AI. Silently training the foundation model on tenant prompts is a controller act dressed up as a processor convenience, and it is exactly the kind of act that turns a customer’s data into a competitor’s model.

The right to erasure (GDPR Art. 17, mirrored by CCPA deletion rights and others) is the governance obligation AI architecture handles worst. The naive implementation, "delete the row," does not reach the place the data actually lives. Personal data in an AI system sits in at least four places, and a deletion request must be reasoned through for each: (1) the operational store (databases, prompt logs, vector indexes), (2) the training corpus (the dataset snapshot the model was trained from), (3) the model weights (where the data may have been memorized), and (4) downstream artifacts (caches, backups, derived datasets, fine-tunes).

Locations (1) and (2) are tractable with disciplined plumbing. Location (3) is the hard one. You cannot surgically delete an individual’s contribution from a trained weight tensor, because the data is spread across billions of parameters in a way that is not addressable. The response starts with an applicability decision: does a valid erasure ground apply, is the material personal data, and is it present in or inferable from the model? If erasure reaches the weights, separate immediate mitigation from removal:

• Immediate output mitigation. Use filters and guardrails to suppress regurgitation while the request is assessed. This is cheap and fast, but it is not erasure, because the data remains in the model.
• Machine unlearning. Apply an algorithm that approximately removes a data point’s influence without a full retrain. This is an active research area as of 2026, not yet a turnkey production control, and it is hard to prove to an auditor’s satisfaction.
• Verified removal, retraining, or deletion. Exclude the data from the corpus and verify that the affected model no longer contains or permits inference of it. If that cannot be demonstrated, retrain without the data or delete the model. This is the costly response when erasure reaches the weights, and it is why erasure obligations push toward not memorizing in the first place (de-duplication, minimization) and toward retention windows that bound how long raw data is kept before the corpus is frozen.

What makes any of this possible is data lineage: a graph that records, for every model and dataset, which sources fed it, when, and on what basis. That graph answers "is this person’s data in this model?" and therefore "what is the cheapest valid response to their erasure request?" Without lineage, every erasure request is unanswerable and every audit is a fishing expedition. With it, you can limit the blast radius of a deletion to the smallest set of artifacts that must change.

The contract stack is where governance becomes enforceable against the operator. Where GDPR applies and you process personal data on a controller’s behalf, a Data Processing Agreement (GDPR Art. 28) defines the relationship: the scope and purpose of processing, the prohibition on using data outside instructions (the no-training clause again), the security measures, the data-subject-rights support you must provide, and the breach-notification timeline. The DPA is what makes "we will not train on your data" legally binding rather than a marketing claim, because a processor must follow documented controller instructions under Article 28(3)(a). Training for the operator’s own purpose needs its own controller-role and Article 6 lawful-basis assessment. Permission in a contract alone does not supply that basis.

Sub-processor management is the part operators underestimate. A downstream party that processes that personal data on your behalf can be a sub-processor: the cloud region, the GPU neocloud you burst to, the vector-database SaaS, the content-moderation API. Assess independent controller roles separately. Article 28(2) requires prior specific or general written authorization, and under general authorization you must give notice of intended additions or replacements and an opportunity to object. Article 28(4) requires equivalent data-protection obligations in the downstream contract and leaves the initial processor liable to the controller for that sub-processor’s performance. A published sub-processor list is a contractual requirement where the DPA makes it one. An AI serving stack assembled from a dozen vendors needs a role and data-flow determination for each, and a tenant’s right to object to a new one can constrain your own procurement. Your supply chain is part of your compliance surface, and a neocloud burst-out (see Chapter 1.8) is a sub-processor event that must be papered before the traffic flows.

Tenant data isolation is where data governance meets the multi-tenancy architecture. The requirement is simple to state and hard to guarantee: one tenant’s data must never be readable by another, never leak through a shared cache or vector index, and never enter cross-tenant model improvement without its own valid purpose, permissions, lawful basis and transparency. The enforcement mechanisms are namespace isolation, per-tenant encryption keys, isolated KV-cache and embedding stores, and strict separation of any data used for model improvement. They are the data-plane half of a problem whose compute-plane half (MIG, vGPU, confidential computing as security boundaries) lives in Chapter 11.6. Keep the two objectives apart. Isolating tenants means the data does not cross the boundary. Protecting the model weights from extraction is a different objective, handled in Chapter 11.8. An operator who treats them as one problem builds controls that satisfy neither the privacy auditor nor the security auditor.

The same governance mis-scopes recur, each from treating data as a payload instead of a regulated object:

• Tier-3 by omission. Crawling and training without a provenance inventory, then discovering under subpoena that you cannot prove lawful acquisition for the sources a plaintiff cares about. The fix is upstream: tag provenance at assembly, not at audit.
• The silent no-training violation. Training the foundation model on tenant prompts without documented controller instructions and a lawful basis for the operator’s own training purpose. It is a controller act performed under the cover of a processor role, and the fastest way to turn a customer’s data into a competitor’s model and a regulatory finding.
• Promising surgical erasure from frozen weights. A DPA or privacy notice that commits to on-demand deletion of memorized data the architecture cannot deliver. Determine where the erasure obligation reaches, then execute verified removal, retraining or model withdrawal inside the statutory clock, which a retraining cadence does not extend.
• Keep-everything logging. Retaining prompt and output logs indefinitely "for debugging," then discovering they are a 20-million-record discoverable liability. Keep minimal retention by default, and treat legal holds as exceptions.
• Conflating data governance with weight security. Building one control set for "protect the data" that an auditor reads as also covering "protect the weights," and satisfying neither. They are different threat models, covered by this chapter and Chapter 11.8 respectively.

Admit a source only when its intended use has a defensible rights and personal-data basis, then carry that decision through every derived artifact. Quarantine delays training, but an undocumented corpus can strand the weights, embeddings and releases built from it. Choose erasure mechanisms that can execute the obligation, not a promise to forget on a schedule the architecture cannot meet.