Treat Model Output the Way You Treat User Input: As Untrusted.

Most teams that deploy a language model have learned to treat its input as hostile. Fewer have internalised the mirror image: its output is hostile too. A model's response is text assembled from a prompt that may contain anything, steered by whoever can influence that prompt, and produced by a process nobody can fully predict. The moment that text leaves the model and enters a database, a browser, a shell, or an HTTP client, it is untrusted input to that component and deserves the suspicion you would give a form field.

The failure is not subtle, only easy to overlook, because the output looks like the system's own work. It arrives from inside the trust boundary, through an API you configured, in a format you specified. Nothing about it announces that an attacker may have written part of it. That is how the oldest injection classes return through a new door.

The output is just more untrusted input

Consider what determines a model's output. The system prompt, the user's message, retrieved documents, tool results, and any memory the agent recalls all sit in one context, and the model weighs them together with no hard boundary between instructions and data. Anyone who can place text in any of those channels can influence what comes out the other end. Users can do it deliberately; third-party content can do it without the user's knowledge, through indirect prompt injection.

So the output cannot inherit the trust of the system that produced it. It carries the trust of its least trusted contributor. This is the principle that governs any pipeline: data derived from untrusted input stays untrusted until something deterministic has checked it for the purpose it is about to serve. A language model is not such a check. It is a transformation, and a persuadable one.

The injections you already know, reborn

Every classic injection reappears the same way: a string produced by one component is interpreted as code by the next. Concatenate model text into a SQL statement and you have SQL injection with an unusually articulate attacker. Render model text as HTML and you have cross-site scripting delivered by a chatbot. Hand model text to a shell or an interpreter and you have command injection, or its cousin, template injection. Let model text become a URL that a server fetches and you have server-side request forgery.

None of these is new, and none requires the model to be malicious or even wrong. A model that is merely helpful — that quotes the user's filename faithfully, reproduces a snippet from a retrieved page, or completes the query the user's phrasing suggested — will pass attacker-chosen bytes through intact. The dangerous property is fidelity, not misbehaviour. The model does exactly what it was built to do while the application treats its output as if a trusted author had written it.

The diagram shows the shape of the problem: a gate, and a path around it. Nobody builds the bypass on purpose. It appears when a developer wires the response straight into a convenience call — a database helper, a templating shortcut, a function that shells out to a tool — because the response was assumed to be the system's own data.

Structured output still needs validation

A common defence is to ask the model for structured output, JSON that conforms to a schema, and to feel that the problem has been constrained away. Schema-guided generation improves the odds that the shape is right. It does not make the content trustworthy, and it is not an enforcement mechanism. Treat the model's JSON the way you treat a JSON body from an anonymous client.

That means parsing strictly with a real parser, validating against a schema in your own code, rejecting unknown fields rather than ignoring them, and checking types, ranges, lengths, and enumerations. It also means validating semantics: an identifier the model returns must belong to the caller, a quantity must sit within business limits, and a field that selects an operation must name one this path is allowed to perform. A well-formed object can still be a hostile one. A string field that is syntactically a string can still carry a SQL fragment, a script tag, or an internal address.

Generated code and the execution boundary

Code is the extreme case, because the output is meant to be executed. Agents that write and run code, analysis tools that generate queries or scripts, and assistants that produce shell commands for a user to paste all extend trust from the text to its effects. The rule is simple to state: generated code is untrusted code, and it runs only inside a boundary designed for untrusted code.

In practice that means an isolated sandbox with no ambient credentials, a restricted filesystem, constrained network egress, resource and time limits, and no path back to production systems except through narrow, audited interfaces. For generated queries, it means a database role that can read only what the task requires, with write and administrative statements simply not granted, so a hostile query fails at the permission layer rather than at a filter you hoped would recognise it. For commands shown to a human, it means treating the display as part of the attack surface: a person approving a command they cannot fully read is not a control.

Escape per sink, allowlist per destination

Escaping is not a property of a string; it is a property of a string at a destination. The encoding that is correct for HTML body text is wrong inside an attribute, a script block, or a URL, and none of it helps in a SQL statement. Output handling therefore has to be applied at the sink, using the mechanism that destination offers for keeping code and data apart.

• SQL — Parameterised queries and prepared statements, with model-supplied values bound as parameters. Never build statements by concatenation, however trustworthy the generating component appears.
• HTML and the DOM — Contextual output encoding for the exact context: element content, attribute, script, or URL. Render model text as text, not markup, unless a vetted sanitiser has processed it, and keep a content security policy as a backstop.
• Shells and interpreters — Avoid the shell altogether. Invoke programs with an argument vector, never a command string, and validate each argument against an allowlist of expected forms.
• HTTP clients — Treat any URL the model supplies as attacker-chosen. Resolve and check the destination against an allowlist of hosts and schemes, block internal and cloud-metadata address ranges, and do not follow redirects blindly.
• Logs and downstream systems — Neutralise control characters and newlines, and remember that today's output is tomorrow's input to another pipeline that trusts it.

Model output is not the system talking to itself; it is untrusted input arriving by a longer route, and it must be validated for the place where it lands.

A defensible output path

Put the pieces together and a defensible output path has a recognisable shape. The model proposes; deterministic code disposes. Every consumer of model output sits behind a validation step specific to that consumer: a schema check, a sink-appropriate encoder, a credential limited to what the task needs. No downstream component trusts a string merely because it arrived from the model.

One principle deserves its own line because it removes a whole class of problems: do not let the model choose the destination. A URL to fetch, a host to connect to, a file path to write, a table to query, a recipient to email — these should come from application logic and allowlists, with the model at most selecting among identifiers you defined. The moment a model-generated string decides where a privileged operation lands, you have lent your servers' network position and credentials to the most persuadable input channel in the system.

Reviewing this is less a matter of testing prompts than of tracing data flows. When 6QURE reviews an LLM-integrated or agentic system, one of the first exercises is to enumerate every sink that model text can reach — queries, templates, shells, fetchers, other agents — and ask, for each, whether a deterministic check sits between the two, specific to that sink and enforced in code rather than in the prompt. The gaps tend to be found in the convenience paths nobody thought of as an integration boundary.