Prompt Injection: The Web's New Untrusted-Input Problem

Twenty years ago the web learned, painfully, that user input is not code. LLM applications are relearning it: any text your model reads — a user message, a fetched web page, a PDF, an email — can contain instructions, and the model cannot reliably tell content from command.

The two flavors

Direct injection is the user telling the model to ignore its instructions. Indirect injection is nastier: the attacker plants instructions in content the model will process later — a web page your summarizer visits, a resume your screening bot reads, a calendar invite your assistant parses. The victim never typed anything.

Why filtering is not enough

Blocklists fail against paraphrase; instruction-detection models are themselves persuadable. Treat 'the model will refuse' as a soft mitigation, never a boundary. The real boundaries live OUTSIDE the model.

Defenses that hold

Least-privilege tools: the model can only call the actions the current user could take themselves — injection then steals nothing the user did not already have. Human confirmation on irreversible actions: sending money, deleting data, emailing externally. Provenance separation: keep system, user, and fetched content in separate labeled channels and never promote fetched text to instruction status. Egress control: if a tool result must never leave the tenant, enforce that in the gateway, not in the prompt.

The mental model

Treat the LLM as a brilliant, suggestible intern: enormous capability, zero security judgment. You would not give that intern production credentials and unsupervised email. Give the model exactly what you would give the intern — sandboxed tools, scoped access, and a supervisor for anything dangerous — and injection attacks become disappointments instead of disasters.

Why models cannot simply tell instructions from data

Traditional software separates code from data structurally: a database query has a defined place for parameters, and properly built applications never let user input change the query's meaning. A language model has no such separation. Its instructions, the user's message and any content it reads all arrive as one stream of text, and the model decides what to do based on patterns in that text. Clever wording, formatting tricks or text hidden in documents can therefore change its behavior, and no prompt can fully prevent this.

This is why prompt injection is best treated as an architectural problem. The question is not "how do I stop the model from being persuaded?" but "what is the worst that can happen if it is?"

Real-world injection paths

Indirect injection can arrive through almost any content an application processes. A summarization tool reads a web page containing hidden text that tells it to include a phishing link in its summary. An email assistant processes a message instructing it to forward the inbox to an outside address. A support chatbot with access to order records reads a product review that asks it to reveal other customers' details. A coding assistant opens a repository whose documentation tells it to add a malicious dependency. In each case the user did nothing wrong; the danger came from content the system trusted implicitly.

Data exfiltration through outputs

Even an application without dangerous tools can leak information if its outputs are rendered carelessly. A classic technique asks the model to include sensitive data inside a link or an image address pointing to the attacker's server; when the interface renders the image, the browser sends the data automatically. Defenses include refusing to render images and links from untrusted domains, displaying links as plain text that users must choose to follow, and filtering outputs for content that should never leave, such as access tokens or personal identifiers.

Designing tools with injection in mind

Tools are where injection turns into impact, so design them defensively. Give each tool the narrowest capability that serves its purpose: a tool that reads one customer's order, not one that queries the whole database. Execute tools with the permissions of the user on whose behalf the model acts, not with a powerful service account. Separate tools that read untrusted content from tools that perform sensitive actions, and require explicit user confirmation, showing exactly what will happen, before any action that sends data outside, spends money or changes records irreversibly.

Isolating untrusted content

Some architectures reduce risk by splitting work between models with different privileges. A quarantined model with no tools reads untrusted content and produces a structured result — for example, a list of extracted fields with strict validation — while a privileged model that can call tools never sees the raw content. This does not eliminate all risk, but it limits how far injected instructions can travel. Even simpler measures help: label fetched content clearly, keep it separate from system instructions and avoid placing it where the model treats text as guidance.

Monitoring and testing

Log every tool call with the context that triggered it, and watch for patterns that suggest manipulation: unexpected tool use after reading external content, attempts to access other users' data or outputs containing unusual links. Test applications regularly with known injection techniques before release and after changes, including injected content in documents, web pages and emails relevant to your use case. Treat findings like any other security bug, with an owner, a fix and a regression test.

What users and administrators should know

People using AI tools should understand that content can manipulate the assistant, and that a request to send, share, pay or delete deserves the same scrutiny whether it comes from a colleague or from an AI summary. Administrators should know which tools each AI application can call, which data sources it reads and who approved those permissions. A short internal guide covering these points prevents many problems before they start.

A defense-in-depth checklist

Assume any content the model reads may contain instructions. Limit each tool to the minimum capability and run it with the user's own permissions. Require confirmation for irreversible or external actions. Keep untrusted content separate from instructions and, where possible, process it with a model that has no tools. Control what outputs can render or send. Log tool use and review anomalies. Test with injection attempts before every release. No single measure is sufficient, but together they turn a fundamental weakness of language models into a manageable engineering risk.

Retrieval systems are injection surfaces

Applications that answer questions from a document collection deserve special attention. Every document added to the index becomes content the model may read and, potentially, obey. Control who can add documents, record where each one came from and prefer sources whose authorship you can verify. Content scraped from the public web or submitted by customers should be treated as untrusted even after it is indexed. When answers cite their sources, reviewers can quickly see whether an unexpected instruction came from a particular document and remove it.

Agents raise the stakes

Autonomous agents that plan and execute multiple steps amplify both the usefulness and the risk of injection. A single injected instruction early in a task can steer every later step. Constrain agents with explicit goals, a limited set of tools, budgets on the number of actions and spending, and checkpoints where a person reviews the plan before anything consequential happens. Keep a complete, readable trace of every step, so that when something goes wrong it can be understood and the weakness fixed.

Policy and accountability

Organizations deploying AI features should decide in advance who approves new tools and data connections, who reviews security testing results and who is accountable when an AI feature misbehaves. Document these decisions alongside the application's other security controls. Regulators and customers increasingly expect this kind of governance, and it ensures that security questions are asked before an assistant receives access to email, payments or customer records rather than after an incident.

Keeping up with a moving field

Prompt-injection techniques evolve quickly, and so do model behaviors. Follow the security guidance published by your model providers and by community projects that catalogue risks for language-model applications, such as the OWASP list of top risks for these systems. Revisit your defenses when you change models, add tools or connect new data sources, because each change can open a path that did not exist before.

The bottom line

Prompt injection cannot be patched out of language models, but its consequences can be engineered down to an acceptable level. Assume the model will sometimes follow the wrong instructions, and design every tool, permission and output channel so that when it does, the damage is small, visible and reversible. Applications built on that assumption can use the remarkable capabilities of these models without handing attackers the keys.

Questions to ask before launch

Before releasing an AI feature, ask: what content can reach the model, and who controls it? Which tools can the model call, and with whose permissions? Which actions require a person's confirmation? What can the output render or send automatically? How are tool calls logged and reviewed? And what is the plan if an injection succeeds in production? If every answer is clear and written down, the feature is ready for real users.