Technical Aspects of AI Safety

AI governance needs technical evidence as well as policy. Engineering practices can reduce and reveal risk, but no checklist guarantees that an AI system is safe, reliable or aligned in every setting. This chapter offers a practical menu: choose controls for the system’s capabilities, use context and potential harm, and record the limits of the evidence.

Robustness and reliability

Robustness begins with rigorous validation on test data that simulates real-world variability and edge cases. Practical techniques:

• Stress testing — computer vision systems tested in varying lighting and noise; text systems tested on out-of-distribution inputs and adversarial paraphrases.
• Adversarial training — training on adversarial examples to harden the model.
• Ensemble methods — multiple models or sensors may reduce some failures; correlated errors and shared dependencies can defeat the expected benefit.
• Operating-domain documentation — explicitly state what conditions the model was designed for; reject or flag inputs outside the domain.
• Capability evaluations — for frontier models, structured pre-deployment evaluations of dangerous capabilities (cyber, CBRN, autonomy). See Frontier Models for the structural pattern.

The NIST AI 600-1 Generative AI Profile dates to July 2024. Use it alongside the separate March 2025 NIST AI 100-2 E2025 adversarial-machine-learning taxonomy; neither is a mandatory testing standard or an exhaustive account of current attacks.¹

Alignment with human values

Alignment research and engineering seek to make AI behaviour better match intended objectives and constraints. Techniques include:

• Reinforcement learning from human feedback (RLHF) — widely deployed for instruction-following and value alignment in large language models.
• Constitutional AI / RLAIF — AI-assisted critique and revision against an explicit set of principles, reducing reliance on human raters at scale.
• Direct preference optimisation (DPO) and successors — offline preference learning that avoids the reinforcement-learning loop.
• Specification through constraints — encoding behavioural rules directly (e.g., refuse-list rules, scoped tool access).
• Tool-use scoping — for agentic systems, explicit permissions and human-in-the-loop checkpoints for high-impact actions.

For applications beyond instruction-tuned LLMs, alignment thinking still applies: define objective functions carefully (not just optimise the metric, but the metric subject to fairness and safety constraints), and peer-review model objectives during development.

For agentic AI systems — AI that takes actions through tools or other interfaces — add controllability: interruption, bounded permissions and recovery paths. Not every external action can be undone. The International AI Safety Report 2026, published 3 February 2026, reviews capabilities, risks and the limitations of safeguards; it is scientific synthesis, not product approval.²

Interpretability and transparency tools

Interpretability sheds light on “black box” models:

• SHAP and LIME — per-prediction feature attribution; still standard for tabular and structured models.
• Saliency maps and integrated gradients — per-input attribution for vision models.
• Attention visualisation — can reveal attention patterns, but is not by itself a faithful causal explanation of an output.
• Probing — lightweight classifiers attached to model internals to test for specific learned representations.
• Sparse autoencoders and circuit analysis — research methods for examining internal features and computations. Human-readable interpretations can be incomplete or wrong and should be tested, not treated as a complete account of a model.
• Model cards — documentation in plain language describing intended use, performance, and limitations.³

Documentation requirements differ from the research convention of a model card. Relevant duties can include:

• EU AI Act Articles 11–13 for covered high-risk systems, subject to applicable dates (technical documentation, records and instructions).
• GDPR Articles 13–15 information duties and Article 22 safeguards for qualifying automated decisions.
• US sector-specific notice and explanation duties, including FCRA and ECOA where applicable.

A generic model card does not automatically satisfy any of these. See EU AI Act, Privacy and Sectoral regulation.

Bias and fairness mitigation

Measuring and mitigating bias is a discipline with multiple technical entry points:

• Pre-processing — examine representation, labels and sampling; reweight or resample when justified by the problem and evaluation evidence.
• In-processing — add fairness penalty terms or constraints to the objective.
• Post-processing — adjust model outputs to reduce disparity.

Evaluation libraries can help calculate metrics, but their defaults do not select a legally or ethically appropriate fairness definition. A useful fairness review records the population, protected groups where lawful to analyse, sample limitations, metrics, uncertainty, thresholds and remediation decisions.

Specific legal regimes now codify fairness expectations:

• NYC Local Law 144 — bias audits for automated employment decision tools (see US State Laws).
• EU AI Act Article 10 — data governance requirements for high-risk systems including measures to detect and prevent bias.
• EU AI Act Article 27 — fundamental rights impact assessments for specified deployers and uses, not every high-risk deployment.

Engineering, legal, product and affected stakeholders should develop fairness criteria together. Metrics embed choices and can conflict under common conditions, such as different base rates; passing one numerical test is not proof that a system is lawful or free from discrimination. Record trade-offs and who approved them.⁴

Performance monitoring and drift detection

Deployment is not a one-time event. Governance requires ongoing monitoring:

• Performance metrics in production: accuracy, calibration, fairness metrics across subgroups, business metrics impacted by the AI.
• Drift detection — distribution shift in inputs (covariate drift), in labels (label drift), or in the input-output relationship (concept drift).
• Out-of-distribution detection — identify inputs unlike training data; route to human review or fall back gracefully.
• Grounding and factual evaluation for LLM applications — measure answer correctness and source support; retrieval can introduce inaccurate or malicious material and does not eliminate confabulation.
• Misuse detection — identify abuse patterns and adversarial attempts.

When monitoring signals a problem, governance triggers a defined response process (NIST RMF’s “Manage” function): retrain, roll back, restrict, escalate. Document each incident and decision; feed findings back into design.

Safety constraints and testing in simulation

For AI interacting with the physical world (robots, autonomous vehicles, medical devices), pre-deployment safety testing is essential:

• Simulation — explore rare and dangerous scenarios while checking whether the simulation represents the operating environment. More simulated runs alone do not prove real-world safety.
• Formal verification — prove specified properties under explicit assumptions where tractable; a proof applies to its model and boundaries, not automatically to the complete deployed system.
• Redundancy and fail-safes — engineering patterns from aviation and industrial control extended to AI systems.
• Guardian systems — an independent monitor that can override or shut down the main AI if it detects unsafe behaviour.

For non-physical AI, apply defence-in-depth: pre-deployment evaluation, runtime controls, monitoring and incident response. The architecture in Frontier Models is one application of this pattern, not a universal safety benchmark.

Agentic systems: put controls around actions

For agentic AI — systems that take multi-step actions via tools, browsing, code execution or robotic control — govern what the system can do, not only what it can say:

• Permission scoping for tools and APIs the agent can call.
• Human-in-the-loop checkpoints for high-impact actions.
• Action records capturing relevant requests, tool calls, approvals and outcomes, with redaction and retention limits. Do not assume a model-generated rationale is a faithful account of its internal reasoning.
• Sandboxing of code execution and environment access.
• Rate limiting and anomaly detection at the action layer.
• Reversibility analysis — categorise actions by reversibility; require stronger controls for irreversible actions.
• Identity and authorisation — bind each action to a principal, scope and expiry; enforce permissions outside the model.
• Untrusted-content separation — documents, webpages and tool responses are data, not new authority to change the task or send secrets.
• Transaction boundaries — use idempotency, previews and explicit approval for consequential or irreversible actions; test partial failure and retries.

These are engineering recommendations, not a statement that every jurisdiction mandates the same agent controls. Test misuse and failure scenarios end to end, including permissions, downstream APIs, rollback and human escalation.

Verification, validation, and release evidence

A safety argument makes a bounded claim and exposes the evidence, assumptions and gaps supporting it. A safety case is one useful form; it is not identical to the transparency report required by California SB 53. Requirements and voluntary commitments must be checked separately. A practical structure includes:

1. The claim (e.g., “this model can be deployed for use case X with acceptable residual risk”).
2. The evidence — evaluations, independent reviews, monitoring data, third-party assessments.
3. The argument linking evidence to claim — explicit reasoning about how the evidence supports the claim.
4. The counterarguments — what could undermine the claim and how those risks are addressed.

Engineering and governance functions both contribute. Name the decision-maker accepting residual risk and the conditions that trigger reassessment. ISO/IEC 42001 can support this work as a voluntary management-system standard; adopting it or obtaining certification does not itself prove a particular system safe.

Footnotes

1. NIST, AI RMF resources and AI 100-2 E2025. ↩

2. International AI Safety Report 2026, 3 February 2026. ↩

3. Mitchell et al., Model Cards for Model Reporting, 2019. ↩

4. Chouldechova, Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments, 2017. ↩