Security and ethics stage summary

A language model that produces text carries a content risk. If it is wrong, the organisation that published it is embarrassed and has to correct the record. An agent that takes actions carries an operational risk, and the difference is not a matter of degree. An agent that sends an unauthorised email, deletes a record or executes a payment instruction has caused harm that may not be reversible, and no amount of apologising afterwards puts the money back or unsends the message. The threat model therefore has to be built around consequences in the world rather than around the quality of the text.

The practical consequence is that an agent's attack surface is proportional to its capabilities, not to its model. An agent that can only answer questions from a knowledge base has a small surface. The same model wired to email, file storage, external APIs and code execution has a large one, because every tool it can call is a route an attacker can try to reach. That makes the tool list the first artefact to look at in any agent security review, before the prompt, before the model choice and before the infrastructure diagram. If you cannot say what the agent may do, you cannot say what a successful attack would cost.

It also means security is a design constraint rather than a feature to be added later. Retrofitting content sanitisation, credential scoping and approval gates onto a deployed agent is expensive, usually incomplete, and sometimes impossible without rebuilding the parts of the system that assumed a general-purpose agent with every tool available. The stage treats this as its opening claim and the rest of it as the working out: decide what the agent may do before you decide how clever it needs to be.

Prompt injection is an attack in which adversarial instructions arrive inside content the agent reads, and override or supplement the instructions it was configured with. The name is borrowed from SQL injection, and the analogy is exact where it counts: in both cases the system cannot tell a value from a command because both travel down the same channel. That is why the module treats injection as a structural property of the current architecture rather than as a defect in one product, and why the phrase to be suspicious of is any claim that a firmer system prompt has closed it.

Direct injection is the visible form. An attacker types the override into the input field, using roleplay framing, a fictional wrapper, a multi-step setup or simply an instruction to comply with everything. The module opens on a dealership customer service assistant that was talked into agreeing to sell a vehicle for one dollar, and the point of the case is not that the model was weak but that the attacker never had to leave the chat box. Models are trained to resist the well-known forms of this, but resistance is not immunity and the phrasings move faster than the training does.

Indirect injection is the form that should shape your design. Here the attacker does not interact with the agent at all. They place instructions in content the agent will retrieve as part of a legitimate task: a web page it searches, a document it summarises, a database record it queries, an email it reads. The instructions arrive through a channel the system was built to trust, and they can be hidden from the person who would otherwise notice them, by markup that renders invisibly or by zero-width characters that a human reader never sees and the model processes normally. You control what your users may type. You do not control every page your agent will read, and that asymmetry is the whole reason the indirect form is harder.

Jailbreaking is a neighbouring attack with a different target, and the two are worth keeping apart. Injection redirects the agent's task towards the attacker's goal. Jailbreaking attempts to strip the model's safety constraints so that it produces output it was trained to refuse, using roleplay, fictional framing, translation into languages where safety coverage is thinner, or long runs of apparently compliant examples before the real request. Both are arms races, each widely known technique eventually gets addressed in training and new ones appear, which is exactly why neither model safety training nor a hardened system prompt can be the entire defence.

Data exfiltration happens when an agent is manipulated into sending sensitive information somewhere it should not go. For agents the risk is much higher than for a static assistant, because the agent has real data sources on one side and real transmission tools on the other. The most direct route is a tool call: manipulate the agent into invoking a message-sending or outbound-request tool with sensitive content as the payload, and the data leaves through a channel the agent is explicitly permitted to use. Nothing in the infrastructure fires, because nothing abnormal happened at that layer. The published research behind this stage demonstrated exactly that against an assistant with inbox access, where a single crafted incoming message caused the agent to forward the rest of the mailbox to an outside address.

The subtler route needs no tool permission at all. If the agent can put a URL into its output and the interface renders it, then encoding sensitive data into that URL's parameters means the data is transmitted the moment the client fetches the resource. The only capability being used is the ability to emit text containing a link. That is worth sitting with, because it defeats the intuition that restricting the tool list is sufficient: output is a channel too, and it needs its own constraints.

Agent hijacking is the same mechanism applied mid-task. If an attacker can influence the content of a tool result, by compromising an API, by placing content where the agent will read it, or by writing to a data store the agent queries, then they can plant instructions in what comes back and redirect everything the agent does next. A price lookup that returns a legitimate JSON object with one extra field carrying an instruction is enough. The agent reads the whole response. The user sees an assistant that appears to be working normally, and may not notice until the damage is done. The habit this builds is to treat every tool result as untrusted input on the same footing as a user message, including results from services you believe you control, because control of the service is not control of the content when the service takes its data from somewhere else.

An agent system is assembled from components other people wrote, and each one is a surface. Model weights can carry behaviour introduced by poisoned tuning data, where a hidden response is triggered only by a specific phrase, so the model passes every ordinary evaluation and misbehaves only when the trigger appears. Package registries are the long-standing route, through outright compromise of a published package and through typosquatting against the names of widely used machine learning libraries, which is how credential-harvesting code reaches machines that never downloaded anything suspicious.

Two surfaces are more particular to agents. Third-party protocol servers, the connectors that expose tools and data to the agent, run with the permissions of the host they were installed on, which means a malicious one can read files, inject instructions into what it returns, or execute code within its granted scope without needing to defeat anything. The same due diligence you would apply to any dependency applies here, and it is applied far less often because the install is a line of configuration rather than a code change. Content APIs are the fourth: news, search and market data feeds where the provider controls what arrives, and where injected instructions reach your agent through a channel that looks entirely legitimate to any filter watching for suspicious user input.

The stage organises all of this with the OWASP Top 10 for Agentic Applications, which sits alongside the older list for language model applications rather than replacing it. The ten are ASI01 Agent Goal Hijack, ASI02 Tool Misuse and Exploitation, ASI03 Identity and Privilege Abuse, ASI04 Agentic Supply Chain Vulnerabilities, ASI05 Unexpected Code Execution, ASI06 Memory and Context Poisoning, ASI07 Insecure Inter-Agent Communication, ASI08 Cascading Failures, ASI09 Human-Agent Trust Exploitation and ASI10 Rogue Agents, taken from the published document rather than from a summary page. Two of them do most of the work in this stage. Goal hijack is where an attacker gets to choose what the agent is trying to do. Identity and privilege abuse is the multiplier: it does not cause an incident on its own, it decides how expensive every other entry turns out to be. An injection against an agent that can only read is an inconvenience. The same injection against an agent that can also pay is a loss.

Input validation is two jobs, and teams reliably do the easy one. Checking user messages against known injection patterns and enforcing a maximum length is cheap and catches automated and unsophisticated attempts before any inference cost is incurred, but pattern matching alone will always trail the phrasings being invented against it. The neglected job is sanitising retrieved content before it enters the context at all: strip markup that can hide instructions from a human reader, remove zero-width characters used for the same purpose, and truncate over-long documents with a visible marker so the agent knows it did not see the whole thing. Alongside that sits instruction and data separation, which means wrapping retrieved material in a structural container and telling the model in its configuration that anything inside the container is material to process rather than instructions to obey. It raises the bar without being a guarantee, which is the right expectation to hold about every control here.

Output validation is the mirror image and is easier to make reliable, because you are checking structure rather than intent. Require the agent to emit output conforming to a declared schema, validate it before it reaches anything, and reject or escalate when it does not conform. A tool call allow-list applies the same idea at the point of action: define what the agent may call and refuse everything else, so a successful injection produces a rejected call rather than a payment. The value of validating output as a separate step is that even a compromised reasoning process only gets as far as producing something malformed, and never reaches the file system, the mail server or the payments API.

Least privilege is the control with the largest effect and it works in three dimensions at once. Tool scope means each agent instance is created with only the tools its task needs, rather than one general-purpose agent holding everything and a hope that the model will choose sensibly. Credential scope means a separate credential per role with only the rights that role needs, so a support agent reads a replica rather than holding write access to the primary store. Data scope means the agent sees only the records in scope for the task rather than the whole document store. The common objection, that this makes agents too weak to be useful, misreads it: least privilege does not mean a minimal agent, it means the right tools for the right task, and a support agent, a research agent and a finance agent can each be fully capable within their own boundary.

The last two controls cover the actions and the code. Human approval gates require explicit confirmation before an irreversible or financially consequential action executes, which is the answer to overreliance and the control that would have stopped the mailbox exfiltration at the moment of sending. Classify every tool by risk: read-only lookups usually run unattended, reversible writes may need a confirmation, and deletions, payments, unreviewed outbound messages and bulk communications should always wait for a person, with a time limit after which the pending action is cancelled rather than allowed through. Sandboxing covers code execution: run it in a fresh container with no network access, a read-only filesystem apart from one temporary directory, explicit memory and processor limits, a timeout, and a minimal base image, then destroy the container. Even successfully injected code then has nowhere to send anything and nothing to persist to.

The most common audit failure in agent systems is not that logging is absent but that it records only errors. That inverts the threat model, because exfiltration through an agent runs entirely on successful calls. The attacker's goal is that the send succeeds. If only failures are written down, the one event you most need to reconstruct is the one event that left no trace, and the incident review has nothing to work with. Log every action that touches the world, including and especially the ones that returned success.

An entry that is actually useful afterwards answers a fixed set of questions: what did the agent do, when, on whose behalf, with what inputs, with what result, and who authorised it. That means a timestamp in a single stated time zone, the session identifier, the agent name and version, the tool called, a summary of the inputs, the result status, and the identity of any human approver. Note the deliberate omissions. Secrets, full personal data and complete tool payloads should not be in there, because a log that contains everything the agent ever processed is a single high-value target that an attacker would rather have than the system itself. Log summaries and truncated representations, and set retention and access to match the data protection rules that apply to you.

Tamper-evidence is the part that gets skipped and it is what makes the log worth keeping. An attacker who can delete or rewrite entries can remove the record of what they made the agent do, so the log has to be written somewhere it cannot be quietly edited: append-only or write-once storage, with each entry carrying a hash of the one before it so that any deletion or alteration breaks the chain visibly. Logging is also the input to monitoring rather than an archive to be filed. A jump in tool call frequency, an unfamiliar recipient address, or a run of rejected approval requests are all signals that something is being attempted, and they are only available if the successful calls were recorded in the first place.

Responsible AI in this stage is treated as an engineering discipline with testable outputs rather than as a values statement, and the reason is that an agent contributing to a consequential decision, about a job, a loan, a place or a referral, is exercising institutional power. The obligations that attach to a person exercising that power attach to the system too, and in a growing number of jurisdictions they now attach as law. The module's opening case makes the mechanism plain: a recruitment screening model trained on a decade of an organisation's own past hiring decisions learned that the pattern in those decisions was a preference, and began downgrading applications that mentioned women's activities and institutions. The model was optimising correctly. The objective was the problem, and the only reason anyone knows about it is that the team looked.

Bias enters an agent pipeline at four points, and only the first is widely discussed. Training data bias reproduces historical inequality because the model learned from decisions that contained it. Prompting bias is specific to language model agents: a configuration that asks for output to be made more professional encodes an assumption about whose register counts as professional, and can mark features of a regional or cultural dialect as deficiencies in text that is correct and clear. Tool selection bias appears when the agent defaults to sources calibrated for some groups and not others, such as retrieving predominantly English results for every user whatever their language. Output calibration bias is the hardest to see, where confidence scores, quality ratings or response lengths differ systematically across groups on tasks of equivalent difficulty.

Testing for this means constructing evaluation sets that vary only the attribute under study and hold everything else constant, then comparing outcomes across the groups. For a screening agent, that is candidate records identical in qualification and experience which differ in details correlated with a protected characteristic. A disparity is evidence to be investigated and mitigated before deployment, not a result to be explained away as a reflection of the real world. The important trap here is proxies. Deleting the protected field does not delete the signal, because a postcode, an institution name or a gap in employment history each correlate with characteristics you thought you had removed, and the model will find them. This is why the test has to be for differential outcomes rather than for the presence of an input field.

Explainability is the companion requirement and it is distinct from both transparency and interpretability. Transparency is telling people they are dealing with an AI system. Interpretability is understanding the model's internals. Explainability is whether a user, an auditor or a regulator can understand and check why a particular output was produced. Agents are better placed here than an end-to-end model, because the sequence of tool calls, the content retrieved and the intermediate steps can all be instrumented and shown. That is what turns a bare score into an account a person can challenge, and it is the mechanism that makes human oversight real rather than nominal, because a reviewer who cannot see why the system decided something cannot meaningfully override it.

The European Union's AI Act sorts systems into four tiers by the risk they pose, and the obligations rise with the tier. Unacceptable risk covers practices banned outright, including government social scoring, real-time remote biometric identification in public spaces subject to narrow exceptions, systems that exploit psychological vulnerability to manipulate people, and inference of protected attributes from biometric data. High risk covers use in listed domains including critical infrastructure, education, employment such as screening applications and evaluating performance, access to essential services such as credit and benefits, law enforcement, migration and border control, the administration of justice and democratic processes. Limited risk covers systems people interact with or that generate synthetic content, and carries disclosure duties. Minimal risk carries none.

The point to carry away is that the tier attaches to the deployment, not to the architecture. The same underlying model can be minimal risk in one product and high risk in another, and nothing about the weights changed between the two. It also follows that a human making the final call does not lower the classification of the system that shaped it: an agent that ranks candidates and hands a shortlist to a recruiter is still the high-risk component in that process. The Act reaches beyond the Union's borders as well, applying to providers placing systems on the Union market and to deployers within it, and to parties outside when the output is used inside, so a team elsewhere serving European users is inside its scope.

For a high-risk system the module maps the duties onto the Act's articles: a continuously maintained risk management system rather than a one-off assessment, data governance intended to keep historical bias out of training data, technical documentation sufficient to assess conformity, logging that allows events to be reconstructed afterwards, transparency adequate for a user to interpret the output, human oversight including the ability to override or stop the system, and demonstrated accuracy and robustness for the intended use, with registration before the system goes to market. The schedule by which these bind phases in and carries dates that move, so read the current ones in the module rather than committing them to memory, and plan against the earliest one that binds the thing you are building.

Two instruments make that practicable. The NIST AI Risk Management Framework is voluntary and organises the work into four functions that operate as a cycle: Govern establishes policy, accountability and escalation; Map identifies the use case, who is affected, which laws apply and what misuse is foreseeable; Measure produces the quantitative evidence, which is where bias auditing and adversarial testing sit; and Manage prioritises, treats and monitors, up to and including the decision to narrow the agent's scope or not to deploy it. Knowing which function an activity belongs to matters more than it sounds: a bias audit is a Measure activity, Govern is what made it mandatory, Map decided which groups to test, and Manage acts on the result. Alongside it, the international standard for AI management systems turns the same work into an auditable organisational capability, and its practical demand on agent builders is documentation: the model version, the system prompt, the tool configuration, the evaluation results and the risk assessment held under version control with the code, because that is what lets you debug a discriminatory outcome, answer a regulator, and see what a change to one component did to the behaviour of the whole.