Ethics and Responsible AI

Responsible AI is not a values statement to be filed and forgotten. It is an engineering discipline with concrete practices: bias testing, fairness metrics, explainability mechanisms, documentation standards, and regulatory compliance requirements. Teams that treat ethics as a philosophy discussion rather than a technical practice will find their systems fail audits, produce discriminatory outcomes, and create legal liability under frameworks including the EU AI Act and the UK Equality Act 2010.

The distinction matters particularly for AI agents. An agent that makes or contributes to consequential decisions (who gets a job interview, who is offered a loan, who receives a medical referral) is exercising a form of institutional power. The ethical obligations that apply to humans exercising that power apply equally to the AI systems that assist or replace them, and in many jurisdictions they now apply as a matter of law.

Bias is a systematic error in an AI system's outputs that unfairly favours or disadvantages particular groups, topics, or outcomes. Bias can enter an agent system at multiple points in the pipeline, not just in the training data.

Training data bias is the most widely known source. When historical decisions reflect historical inequalities, a model trained on those decisions learns to replicate them. The Amazon hiring algorithm is the canonical example. A credit scoring model trained on historical lending data in a market where certain groups were offered worse terms will learn to offer those groups worse terms.

Prompting bias is specific to LLM-based agents. System prompts encode assumptions about what a “normal” user looks like, what “professional” writing sounds like, and what constitutes a good outcome. A tone-correction agent instructed to make text “more professional” may flag linguistic features associated with specific cultural or regional dialects as deficiencies, even when the text is grammatically correct and communicatively effective.

Tool selection bias arises when agents default to tools or data sources that are better calibrated for some user groups than others. A search agent that retrieves predominantly English-language results for all users, regardless of the user's preferred language, is exhibiting tool selection bias. So is a summarisation agent whose abstractive quality degrades for technical domains that are underrepresented in its training data.

Output calibration bias is the subtlest form. Quality scores, confidence estimates, or summary lengths may be systematically different for different groups even when the underlying task difficulty is equivalent. This is difficult to detect without explicit measurement across demographic groups.

Testing for bias requires constructing demographically varied evaluation sets and comparing outcomes across groups. For a hiring screening agent, this means creating test CVs that are identical in qualification and experience but vary in details correlated with protected characteristics (names associated with different ethnic groups, degree institutions with different demographic profiles, gaps corresponding to parental leave). A disparity in scores across these groups is evidence of bias that must be investigated and mitigated before deployment.

ISO/IEC 42001:2023, the international standard for AI management systems, requires organisations to establish processes for identifying and assessing the impacts of AI systems on individuals and groups, including impacts on fairness and equality. This formalises bias testing as a management system requirement, not just a technical recommendation.

Explainability is the degree to which an AI system's outputs can be explained in terms that a user, auditor, or regulator can understand and verify. It is distinct from interpretability (understanding the model's internal workings) and from transparency (openness about the existence and nature of the AI system). An agent can be transparent (users know they are interacting with AI) without being explainable (users cannot understand why they received a particular score or recommendation).

Agents are more explainable than end-to-end black-box models because the reasoning trace is visible. The sequence of tool calls, the content retrieved, and the intermediate reasoning steps are all potentially loggable. This means that, with appropriate instrumentation, you can produce an explanation of the form: “The candidate scored 3.2 out of 5. The agent searched the knowledge base for the required skills (Python, SQL, project management), found evidence of Python and SQL in the CV, found no explicit evidence of project management, and downweighted accordingly. The search results used were: [list].”

The EU AI Act requires this kind of explanation for high-risk systems. Article 13 specifies that high-risk AI systems must be transparent: users must be able to understand how the system works at a level sufficient to exercise meaningful oversight. For consequential decisions (hiring, lending, education, healthcare triage), this means users must be able to understand why a particular output was produced, not just what the output was.

The EU AI Act entered into force on 1 August 2024. It is the world's first thorough legal framework specifically for AI systems, and it has extraterritorial effect: it applies to AI systems that operate in or affect people in the European Union, regardless of where the provider or deployer is located. A company based in the United Kingdom or the United States deploying an AI agent to EU users is subject to the Act.

The Act uses a risk-pyramid structure with four tiers.

Unacceptable risk (prohibited). AI systems in this category are banned outright. Examples include social scoring systems used by governments to evaluate citizens, real-time biometric identification in public spaces (with narrow exceptions), AI systems that exploit psychological vulnerabilities to manipulate behaviour, and AI used to infer political opinions, religious beliefs, or sexual orientation from biometric data.

High risk (strict obligations). AI systems used in eight domains listed in Annex III: critical infrastructure, education and vocational training, employment (including CV screening and performance evaluation), access to essential services (credit scoring, social benefits), law enforcement, migration and border control, administration of justice, and democratic processes. Agent systems deployed in any of these domains fall into the high-risk category.

Limited risk (transparency obligations). Systems including chatbots, deepfakes, and emotion recognition tools must disclose to users that they are interacting with AI. No additional technical obligations apply.

Minimal risk (no obligations). Spam filters, AI-powered video game characters, and similar systems fall here. No mandatory requirements apply, though voluntary codes of practice are encouraged.

The high-risk obligations under Articles 9 to 16 are extensive. Providers must establish a documented risk management system (Article 9), implement data governance practices to ensure training data does not introduce bias (Article 10), maintain technical documentation sufficient to assess conformity (Article 11), implement logging to enable post-hoc reconstruction of events (Article 12), ensure transparency to users about the system's capabilities and limitations (Article 13), design for human oversight including the ability to override or stop the system (Article 14), and achieve and document the accuracy and robustness needed for the intended use (Article 15). High-risk systems must also be registered in the EU AI database before being placed on the market (Article 71).

The implementation timeline for high-risk obligations is August 2026 for new systems, and August 2027 for AI embedded in existing regulated products. General-purpose AI (GPAI) model obligations (covering foundation models like Claude and GPT-4) took effect in August 2025.

The NIST AI Risk Management Framework (AI RMF 1.0), published by the National Institute of Standards and Technology in January 2023, provides a voluntary framework for identifying, assessing, and managing AI risk throughout the lifecycle of an AI system. It is organised around four core functions that operate as a continuous cycle: Govern, Map, Measure, and Manage.

Govern establishes the organisational conditions for responsible AI: policies, accountability structures, roles and responsibilities, and a culture that treats AI risk as a first-class concern. For an agent deployment, Govern means having documented policies about which agent capabilities require senior approval, who is accountable when an agent causes harm, and how concerns about agent behaviour are escalated and resolved.

Map categorises AI use cases and identifies the relevant risks. For a hiring screening agent, Map requires identifying: what types of discrimination risk exist, what the consequences of a false positive or false negative are, who is affected by the system's outputs, what applicable laws and standards apply, and what the intended and foreseeable misuse scenarios are.

Measure evaluates AI performance, bias, robustness, and fairness using quantitative and qualitative methods. This is where bias testing (Section 18.2) and explainability assessment (Section 18.3) are applied. Measure also includes adversarial testing: attempting to elicit discriminatory or harmful outputs through edge cases, out-of-distribution inputs, and deliberate adversarial prompts.

Manage prioritises and responds to identified risks. For risks that cannot be mitigated to an acceptable level, Manage may mean limiting the agent's scope, adding human oversight requirements, or deciding not to deploy. NIST AI RMF Manage 2.4 specifically requires that risk treatments be applied, monitored for effectiveness, and adjusted when evidence suggests they are not working.

ISO/IEC 42001:2023 is the international standard for AI management systems. Published in December 2023, it provides a framework for establishing, implementing, maintaining, and continually improving an AI management system: the organisational infrastructure for responsible AI development and deployment. It is analogous to ISO 27001 (information security management) and ISO 9001 (quality management), and it forms the basis for third-party certification and audit.

The standard requires organisations to define the scope of their AI management system, assess the impacts of AI systems on individuals and groups (including bias and fairness), establish controls for managing identified risks, maintain documented records sufficient for audit, and conduct periodic internal and external reviews. For organisations deploying high-risk AI under the EU AI Act, ISO/IEC 42001 certification provides a structured path to demonstrating compliance with the Act's documentation and risk management requirements.

Practically, the most important implication for agent builders is the documentation requirement. ISO/IEC 42001 requires that the model version, training data sources, evaluation results, system prompt, tool configuration, and risk assessment be documented and version-controlled alongside the code. This is not a bureaucratic exercise: it is the foundation for debugging discriminatory outcomes, responding to regulatory enquiries, and ensuring that changes to any component are assessed for their impact on the overall system's behaviour.

Constitutional AI (CAI) is Anthropic's approach to training AI models to be helpful, harmless, and honest. Rather than relying solely on human labellers to identify harmful outputs, the approach uses a set of principles (the “constitution”) to guide reinforcement learning from AI feedback (RLAIF): the model critiques its own outputs against the constitutional principles and produces revised outputs that better comply.

Constitutional AI is relevant for agent builders for three reasons. First, Claude models trained with CAI are relatively resistant to certain categories of harmful request, which reduces (though does not eliminate) the risk that an injection attack will cause the model to produce dangerous outputs. Second, the approach demonstrates that safety constraints can be built into model training rather than only applied at runtime through filtering. Third, the constitutional principles are published, which means they are auditable: you can evaluate whether a model's behaviour in practice is consistent with its stated principles.

The limitation of Constitutional AI as a safety mechanism is that it operates at the model level, not the system level. It reduces the likelihood that the model will comply with a harmful instruction, but it does not prevent injection attacks from routing harmful instructions through the model, does not enforce least privilege on tool access, and does not substitute for audit logging or human approval gates. It is one layer in the defence-in-depth approach described in Module 17.

Before deploying an agent that makes or contributes to decisions affecting people, the following questions should have documented answers:

Affected populations. Who is affected by this system's outputs? Have you tested for differential impact across the relevant demographic groups? Is the disparity in outcomes across groups within acceptable limits?

Intended use and misuse. What is the system intended to do? What are the foreseeable ways it could be misused, and have mitigations been implemented for each?

Human oversight. Is there a clear mechanism for a human to review, override, or correct consequential decisions? Is that mechanism tested and documented?

Feedback channel. Is there a way for users or affected individuals to report errors or harm? Who is responsible for responding to those reports?

Documentation. Are the model version, system prompt, tool configuration, evaluation results, and risk assessment documented and version-controlled alongside the code?

Review date. When will this system be re-evaluated? AI models change (through provider updates), deployment contexts change, and regulatory requirements change. A system that was compliant at launch may not remain compliant without periodic review.

Regulatory compliance. Have you confirmed which regulations apply (EU AI Act, UK Equality Act 2010, GDPR, sector-specific rules) and that the deployment meets the applicable requirements?