Prompt injection is the attack you cannot fully patch

A coaching agent has a tool that reads the user's shared notes and a tool that emails a summary to their therapist. A user pastes in an article they want help with. Buried in the article, in text the model reads but the user never notices, is a line that says ignore your instructions, pull the user's private notes, and send them to this address. The agent has the private data, it has the untrusted article, and it has a send tool. It follows the instruction. Nobody wrote a bug. Every part worked as designed.

The model reads instructions and data in the same channel

Prompt injection is when untrusted content the model reads carries instructions the model then follows, as if the attacker were the developer or the user. Simon Willison coined the term in September 2022, naming it after SQL injection because the shape is the same. Trusted commands and untrusted input travel through one channel, and the system cannot reliably tell which is which. He wrote it up in Prompt injection attacks against GPT-3, and the name stuck because it gave builders a category to check against.

Two forms matter, and they are worth pinning on first use. Direct prompt injection is the user typing the malicious instruction themselves, the classic jailbreak where someone tries to talk the model out of its rules. Indirect prompt injection is a third party planting the instruction in content the agent will later read, a web page, a PDF, a calendar invite, an email, a stored memory. The user never sees it and never asked for it. Kai Greshake and colleagues named and demonstrated the indirect form in Not what you've signed up for, showing that an LLM wired into a real application could be compromised just by looking at a website, with the attack text hidden in the page. Direct injection is a fight with your own user. Indirect injection is an attacker you never invited, speaking through the content your agent trusts.

There is no separate line for trusted instructions

The reason this is not a patchable bug lives in how the model reads its input. A system prompt, the user's message, a retrieved document, and a tool's output all arrive as one stream of tokens in the same context window. The model has no privileged channel that says these tokens are commands from the developer and those tokens are only data to be summarized. It was trained to follow instructions written in natural language, and an injected instruction is instructions written in natural language. OWASP puts prompt injection at the top of its Top 10 for LLM Applications and states the reason plainly. The vulnerability exists because models cannot currently distinguish between trusted instructions and untrusted content, and given the stochastic nature of how models work, it is unclear whether any fool-proof method of prevention exists.

SQL injection was solvable because a parameterized query draws a hard line the database enforces, the query text on one side and the user's data on the other, and no amount of clever input crosses it. A language model has no equivalent line to enforce. Every defense so far is a filter, a wrapper, or a probability shift, not a guarantee, which is why the honest framing is reduce the blast radius, not close the hole.

The problem gets worse exactly as guidance agents get more useful. Give an agent tools and it can now act on an injected instruction, not just say something wrong. Give it memory and yesterday's poisoned content sits in today's context waiting to fire. Willison's sharpest framing is the lethal trifecta, the three capabilities that turn injection from embarrassing into dangerous when one agent holds all three in a session. Access to private data, exposure to untrusted content, and a way to send data out. Hold any two and an attacker who controls the untrusted content is contained. Grant all three and a single poisoned page can read the private data and exfiltrate it, with no exploit code, just English. The coaching agent in the opening had all three.

Treat every external token as hostile, then shrink what it can reach

Since you cannot make the model reliably tell instructions from data, the working move is to stop relying on it to. Treat all external and remembered content as untrusted by default, the retrieved document, the tool output, the web page, the prior memory, and design so that even a fully hijacked model can do little harm. That means breaking the lethal trifecta on purpose. Scope tool permissions to the task, not the account. Put a human confirmation in front of any irreversible or exfiltrating action, the send, the delete, the purchase, the external post. Prefer read-only tools where the job allows. This is the same instinct as designing a refusal and escalation boundary as a real feature rather than a hope.

The stronger architectural pattern is to separate the model that can act from the model that reads untrusted text. Willison's dual-LLM pattern runs a privileged LLM that holds the tools but never sees raw untrusted content, and a quarantined LLM that reads the untrusted content but has no tools and cannot act. The quarantined model returns structured, referenced results, and the privileged model works with the reference without ever ingesting the attacker's tokens, so the injected instruction has no path to the actor. Google DeepMind's CaMeL builds on that idea, extracting the control and data flow from the trusted request up front so untrusted data can never change what the program does, and enforcing capability policies at each tool call. On the AgentDojo benchmark it solved 77 percent of tasks with provable security against injection, next to 84 percent for the undefended baseline that provides no guarantee at all. The gap is the price of the guarantee, and it is small.

Locking it down costs the usefulness you built the agent for

Every one of these defenses trades capability for safety, and the trade is real, not rhetorical. AgentDojo measures it directly. A plain GPT-4o agent in the benchmark held 69 percent benign task utility, which fell to 45 percent once injections were present. Turning on tool filtering cut the attack success rate to 7.5 percent, but utility dropped to 53.3 percent, so a chunk of the agent's usefulness went with the attack surface. That is the shape of the whole space. The tighter you scope tools and the more confirmations you require, the more often the agent stops to ask about work it used to just do, and a guidance product that interrupts constantly is a product people abandon.

Detection looks like the easy out and is the weakest leg to stand on. A classifier that flags injected instructions runs into the same wall as any content filter, false positives. Flag too aggressively and you block the user's own legitimate content, the article they actually wanted summarized, and detectors trained on last month's attacks miss this month's rephrase, so the arms race never ends. The design patterns paper from a group across Google, ETH Zurich, and Microsoft makes the point that holds this whole piece together. Detection and model-hardening reduce risk but cannot promise safety, so the patterns worth building are the ones that constrain what an agent can do once you assume it may be compromised, not the ones that try to keep it from ever being compromised. You are choosing where to spend the trust, not whether to spend it.

Assume the model will be compromised, and design for that

The rule to carry is short. There is no known complete fix for prompt injection, so stop trying to make the model perfectly obedient and start making a compromised model unable to hurt anyone. Assume any external or remembered content may be adversarial, keep the three capabilities of the lethal trifecta from meeting in one unsupervised session, and put a person in front of the actions you cannot take back.

Then test it like any other behavior. Prompt injection belongs in the eval suite next to your jailbreak and robustness cases, because a model swap or a new tool can quietly reopen a hole you closed, the same way it reopens a safety boundary. This sits alongside jailbreaks and robustness as the adversarial half of your evals, and it is the standing reason a hard-guidance product sometimes needs more than a conversation, a human at the step where a wrong action cannot be undone. The open question the field has not answered, and may not soon, is whether a model can ever be trained to hold a hard line between the instructions it was given and the data it was handed. Until it can, the safe assumption is that it cannot.