AI security

Most bad AI security advice comes from answering one question with a control that belongs to another. There are four separate problems, and a strong answer to one is often irrelevant to the rest. The first is securing AI systems: the model, its data pipelines and the tools it can call are software, and they have their own weaknesses. The second is AI-enabled attacks: how attackers use AI to work faster and more convincingly. The third is AI for defence: where AI genuinely helps a security team, and where it quietly makes things worse. The fourth is governing AI: the frameworks and obligations that say who is accountable for an AI system and how its risk is managed.

Keeping the four apart is a practical habit, not an academic tidiness. A board that has bought a content filter for its chatbot has addressed a sliver of the first problem and nothing of the other three. A team that has trained staff to spot AI-written phishing has touched the second problem but has not asked whether its own AI tools can be turned against it. The figure below sets out the four with their canonical framework and one characteristic failure so that you can place any AI security claim you meet into the box it actually belongs in.

The OWASP Top 10 for Large Language Model Applications, updated for 2025, is the reference list for the weaknesses of AI applications. It runs from LLM01 Prompt Injection through to LLM10 Unbounded Consumption, and it now names System Prompt Leakage and Vector and Embedding Weaknesses as first-class risks. The list is worth reading in full, but one entry governs how you should think about the rest.

Prompt injectionis an attack where crafted input overrides the developer's instructions and makes the model behave as the attacker intends. It works because instructions and data travel to the model on the same channel: the model reads a system prompt, a user message and any content it has been given as one stream of text, and it has no reliable way to know which words are commands and which are merely quoted. Direct prompt injection is delivered by whoever is typing to the model. Indirect prompt injection is more dangerous: the hostile instruction rides in on content the application fetches and trusts, such as a web page, an email or a document. Any system that uses retrieval-augmented generation to ground its answers in a corpus has therefore turned that corpus into an input the attacker may control.

The honest teaching, and the one OWASP itself gives, is that there is no deterministic fix. Because instruction and data share a channel, no input filter can reliably separate them, and a filter that blocks today's phrasing is bypassed by tomorrow's. Defence is structural rather than a single control: give the model the least privilege and the fewest tools it needs, treat everything the model produces as untrusted output that must be checked before it is acted on, put a human approval gate in front of any consequential action, and test the system adversarially before and after it ships. The primary figure below draws these paths for a typical AI application.

The agent era widens this surface. When a model can call external tools, the tools become an injection route of their own. The Model Context Protocolgives AI applications a standard way to reach tools and data sources, which is useful and also standardises an attack surface. The NSA's information sheet on Model Context Protocol security warns of tool poisoning, where the description or metadata of a tool is written to manipulate the model, and of metadata injection more generally, so a tool's configuration has to be treated as untrusted just like any retrieved content. Running alongside this is excessive agency: giving an agent more autonomy, more tools or broader permissions than the task needs, so that a manipulated model can do real harm. The structural questions to ask of any agent are therefore simple to state and hard to dodge: what can it do, what should it be trusted to do, and what sits behind a human gate.

AI has not invented new categories of attack so much as removed the friction from old ones. Two changes matter most. The first is that the linguistic tells are gone. Model-written lures carry no spelling mistakes and adopt a colleague's tone, and deepfake voice and video defeat the instinct to ask whether something looks real. The Arup case, covered in the human factors module, is the reference point: a finance employee in Hong Kong joined a video call on which the chief financial officer and colleagues were all deepfakes and authorised transfers of about twenty-five million US dollars. Any control that depends on a person detecting fakery is now a broken control, which is why verification procedures such as out-of-band callbacks belong in the design rather than in the training.

The second change is scale and autonomy, and the GTG-1002 case at the top of this module is the first well-documented instance of a largely autonomous campaign. When agents can carry the operational load, the historic trade between how many targets a crew can work and how carefully it works each one weakens. That is the sober reading a security team needs: the tooling that makes a legitimate developer faster makes an attacker faster too, so plan for more attempts of consistent quality rather than for a sudden new exploit.

AI earns its place in a security team in the unglamorous work: drafting the first pass of an alert triage, summarising a long incident timeline, suggesting a detection rule from a described behaviour, and turning a half-formed question into a query. Used this way it lifts the floor for a stretched team and buys back analyst time. The security operations judgement in the detection and incident response module still applies: the human owns the decision, and the AI drafts.

The failure modes are specific and worth naming so they can be designed against. Automation bias is the tendency to accept a confident machine answer without the scrutiny a human answer would get, which is most dangerous exactly when the model is fluent and wrong. Alert suppression is its quiet cousin: a confident model that dismisses a real alert as a false positive removes the signal before anyone sees it. The corrective is to use AI to widen what a team looks at rather than to shrink it, to keep the model's output as a recommendation a named analyst confirms, and to measure whether the assistance improves outcomes rather than assuming it does.

Governance answers who is accountable for an AI system and how its risk is managed, and several frameworks now speak to it. The NCSC and CISA Guidelines for Secure AI System Development, published in November 2023, set out secure practices across four lifecycle phases: secure design, secure development, secure deployment, and secure operation and maintenance. They are the natural companion to the OWASP list because they cover the process rather than the individual weakness.

The AI RMF, the NIST AI Risk Management Framework, gives a common structure for managing AI risk across the lifecycle around the functions govern, map, measure and manage, and its Generative AI Profile, published as NIST AI 600-1 in July 2024, extends that structure to generative systems. ISO/IEC 42001 provides a certifiable management system for AI, the AI counterpart to the information security management system taught in the regulation and assurance module. The EU AI Act adds binding obligations on a phased timetable, covered in that same module, and the NCSC's Cyber Assessment Framework version 4.0 takes the deliberate approach of weaving AI risk through its existing outcomes rather than bolting on a separate AI annex. The pattern across all of them is the same: govern AI as part of how you already manage security and risk, not as a novelty that sits to one side.