Privacy, Data Governance, and Security

Privacy and data governance are critical pillars of AI governance because AI systems consume and generate large volumes of data, often including personal and sensitive information. Compliance with privacy laws — GDPR in the EU, CCPA/CPRA and state-level laws in the US, PIPEDA in Canada, PIPA in Korea — is the starting point, but a mature AI governance programme also addresses data quality, model security, third-party risk, and incident response specifically calibrated to AI.

Foundational privacy law

The EU General Data Protection Regulation (GDPR) applies to in-scope processing of personal data, including training and inference. Article 22 restricts solely automated decisions with legal or similarly significant effects, subject to exceptions and safeguards. It is not a general ban on automation. Related information duties appear in Articles 13–15. Establish the lawful basis, assess special-category data and complete a DPIA when required; adding a nominal human reviewer does not resolve every issue.¹

In the United States, CCPA/CPRA provides California residents with rights concerning covered businesses, including access, correction, deletion, opting out of sale/sharing and limiting certain uses of sensitive personal information. Coverage, exceptions and the conditions on each right matter.² Other state privacy laws differ in thresholds, exemptions, automated-decision rules and effective dates. Maintain a jurisdiction-specific register rather than assuming California compliance covers every state.

Sector-specific regimes may also apply: HIPAA for covered entities and business associates, GLBA for covered financial institutions, FERPA for covered education records, and COPPA for covered online collection from children. Working in a sector does not automatically trigger every statute; map the entity, data and activity to the relevant rule. See Sectoral regulation.

International regimes include Brazil’s LGPD, Japan’s APPI, Singapore’s PDPA and Korea’s PIPA; their automated-decision rights are not interchangeable with GDPR Article 22. India’s DPDP Act and Rules have phased commencement following November 2025 notifications: some provisions commenced then, with later tranches after 12 and 18 months. Do not describe the full substantive regime as already effective in 2024–2025.³

Data quality and lineage

Data quality influences model performance, but sound data alone does not ensure safe outcomes. A practical data-governance record includes:

• Provenance tracking — for each dataset, document source, acquisition method, licensing, and any consent obtained.
• Quality assessment — measure completeness, accuracy, freshness, representativeness; document known limitations.
• Datasheets for datasets — a documentation approach covering collection, composition, intended uses and limitations.⁴ This can support evidence for applicable data-governance duties; EU AI Act Article 10 does not mandate the particular “Datasheets for Datasets” format.
• Schema and version control — treat training data with the same rigour as code.

Assess the scope and transitional dates of California AB 2013 and EU AI Act Article 53 before determining which public training-data documentation is required — see Copyright & IP.

Data minimisation and access control

Use data proportionate to the purpose and risk. Pseudonymisation, encryption and access controls can reduce exposure but do not automatically make personal data anonymous or establish a lawful basis. Consider:

• Role-based access controls restricting training-data access to authorised personnel and processes.
• Data tokenisation for training where individual records are not required.
• Differential privacy to bound privacy loss under a specified mechanism and privacy budget, with implementation and composition reviewed.⁵
• Output-side restrictions to reduce, not eliminate, the risk of revealing memorised personal data.

Privacy-enhancing technologies (PETs)

Privacy-enhancing technologies solve different problems. Select them against a documented threat model and measured performance constraints, not a blanket “privacy-preserving” label:

• Differential privacy — mechanisms such as DP-SGD or private aggregation; specify the protected unit, privacy parameters and cumulative privacy loss.⁵
• Federated learning — distributed training without centralising raw datasets. Model updates can still leak information; secure aggregation or differential privacy may be needed.
• Homomorphic encryption — computation on encrypted data; performance has improved but still constrains practical use to specific inference workloads.
• Secure multi-party computation (MPC) — computation designed to limit input disclosure under specified assumptions about the participants and protocol.
• Trusted execution environments (TEEs) — isolated execution with hardware and attestation dependencies; examine side-channel, administrator and supply-chain risks.

None of these techniques alone establishes anonymity, legal compliance or protection against every attack. For differential-privacy claims in particular, use NIST SP 800-226 to examine assumptions and implementation hazards.⁵

Retention and purpose limitation

Data governance policies should define retention schedules and purpose-limitation controls for training and inference data. GDPR requires data not be kept longer than necessary; CCPA permits consumer-initiated deletion. Practically:

• Document retention windows per dataset; automate deletion at end-of-window.
• Re-purposing review — if a dataset is reused for a new model or use case, evaluate consent and purpose limitation before training.
• Deletion engineering — identify affected corpora, retrieval stores, logs, backups and model artefacts. Evaluate applicable rights and exceptions, and document what can actually be removed. Machine unlearning is not a universally reliable substitute for retraining or a guarantee of legal erasure.

Model security

AI models themselves are attack targets. Threat categories include:

• Model extraction — adversaries query a model to approximate its behaviour or recover model information. Training-data extraction and membership inference are related but distinct privacy attacks.
• Adversarial examples — inputs crafted to cause misclassification.
• Data poisoning — tampering with training data to embed backdoors or biases.
• Prompt injection — manipulating LLM behaviour through crafted input, particularly via untrusted data in retrieval pipelines.
• Model evasion — bypassing safety filters or content controls.

The NIST AI 600-1 Generative AI Profile was published in July 2024. The separate NIST AI 100-2 E2025, published in March 2025, provides an adversarial-machine-learning attack and mitigation taxonomy. These are complementary voluntary resources, not a security certification.⁶⁷

Defensive measures:

• Adversarial training — train on adversarial examples to harden the model.
• Input and output filtering — useful layers, not an authorisation boundary. Treat retrieved documents and tool results as untrusted data; enforce permissions in the application and tool layer.
• Rate limiting and behavioural monitoring — detect extraction and reconnaissance attempts.
• Red-teaming — scoped adversarial evaluation, with findings linked to mitigations and retests. Legal duties and voluntary commitments differ (see Frontier Models).
• Provenance verification of upstream components (base models, weights, fine-tuning datasets).

Data security

AI data pipelines must be secured with general infosec hygiene plus AI-specific considerations:

• Encryption in transit and at rest for training data and model weights.
• Identity and access management for systems handling training data and models.
• Integrity verification of training datasets (cryptographic hashing, version control).
• Poisoning detection — statistical anomaly detection in training data.
• Backup and recovery for model weights and training pipelines as critical assets.

Third-party and supply chain risks

Most AI systems incorporate third-party components: pre-trained foundation models, open-source libraries, cloud AI services, vector databases, datasets. Governance must extend to these:

• Vendor assessment for security, privacy, and AI-specific compliance posture.
• Model provenance — document source, version, and license of every model component.
• License compliance for open-source models with restrictive terms.
• Contractual allocation of responsibility for incident response, data handling, and regulatory cooperation.
• Cross-border data transfer mechanisms (Standard Contractual Clauses, adequacy decisions) where data crosses jurisdictions.

EU AI Act Article 25 explicitly addresses provider obligations through the value chain for high-risk systems.

Incident response

AI incident response should be a dedicated discipline within general incident response, with playbooks for:

• Safety incidents — AI causes physical or financial harm.
• Bias incidents — AI is found to produce discriminatory outcomes.
• Privacy incidents — AI reveals or is alleged to reveal personal data.
• Security incidents — AI is compromised or used to attack other systems.
• Misuse incidents — AI is used by adversaries for harmful purposes.

Map each reporting regime separately: the reporting entity, threshold, authority, clock start, deadline, initial-notice content and follow-up requirements. GDPR personal-data breach notification can involve a 72-hour supervisory-authority deadline, subject to its risk threshold. AI Act high-risk-system incident rules are not interchangeable with systemic-risk GPAI reporting, and California’s critical-incident rules have their own timing. Do not use a generic “15-day AI incident” timer. See Frontier Models and Sectoral regulation.¹

Independent reviews and red-teaming

Choose assessments that answer a defined question; distinguish legal requirements from procurement preferences:

• Bias audits — NYC LL 144 requires an independent audit for covered automated employment decision tools, not all hiring software.
• Privacy and security reviews — scope these to applicable law, contracts and the actual system; an annual review is not a universal statutory rule.
• Adversarial evaluations — selected frontier regimes require them; record method, coverage and residual risk.
• ISO/IEC 42001 certification — concerns a defined AI management-system scope, not a guarantee that each model is safe or compliant. ISO/IEC 42006 specifies requirements for certification bodies; it did not create the first possibility of certification. See ISO standards.

Coordination across functions

A practical programme coordinates privacy, security, data governance, AI/ML engineering, legal, compliance and product functions. A council or office is one organisational option, not a universally mandated structure. Assign decision authority, escalation and evidence ownership clearly; ISO/IEC 42001 offers a management-system approach where appropriate.

Footnotes

1. GDPR, Articles 5, 13–15, 22, 33 and 35. ↩ ↩²

2. California Attorney General, CCPA rights and scope. ↩

3. MeitY, DPDP Rules 2025 and enforcement notifications; NIST, phased DPDP commencement overview. ↩

4. Gebru et al., Datasheets for Datasets. ↩

5. NIST, SP 800-226: Guidelines for Evaluating Differential Privacy Guarantees, March 2025. ↩ ↩² ↩³

6. NIST, AI RMF resources and AI 600-1 publication date. ↩

7. NIST, AI 100-2 E2025: Adversarial Machine Learning, March 2025. ↩