Every generative system deployed to real users will be probed by some of them. Some probing is curiosity, some is mischief, and some is deliberate attack. The systems that hold up are not the ones with the most guardrails but the ones whose guardrails were tested by people actively trying to defeat them before launch. This guide covers prompt injection, why guardrails fail more easily than teams expect, and what structured red teaming actually involves.
Prompt Injection: The Core Problem
Prompt injection exists because of a structural property of language models: they receive instructions and data in the same channel, as text. A model given a document and asked to summarise it has no reliable way to distinguish "this is content to summarise" from "this is an instruction to follow" if the document itself contains instruction-shaped text.
Direct injection is a user typing something designed to override the system's instructions: asking it to ignore previous directions, adopt a different persona, or reveal its configuration.
Indirect injection is more serious and less understood. Instructions are planted in content the system will retrieve: a web page, a document in a knowledge base, an email, a code comment. The user never sees it. The system retrieves the content as part of normal operation and follows the embedded instruction. For agentic systems that take actions, this is the highest-severity risk in the category, because the consequence is not a bad answer but an unwanted action.
Why Guardrails Fail
Teams typically add guardrails through system prompts, filters, and refusal training, and then are surprised by how readily they fall.
System prompt instructions are text, and text can be argued with. Filters match patterns, and paraphrase defeats patterns. Refusal training generalises imperfectly, so a request refused in one framing succeeds in another. And multi-turn interaction erodes constraints: a model that refuses at turn one frequently complies at turn eight, after context has accumulated that makes the request seem reasonable.
The pattern behind all of these is that guardrails built on the model's own compliance are only as strong as the model's judgment under adversarial pressure, which is weaker than it appears in testing that is not adversarial.
What Red Teaming Actually Covers
Instruction override. Attempts to displace the system's instructions through direct argument, role-play framing, hypotheticals, or claimed authority.
Indirect injection. Planting instructions in retrievable content and confirming whether the system executes them. Essential for any retrieval or agentic system.
Boundary erosion over turns. Testing whether constraints hold across a long conversation rather than a single exchange, since single-turn testing systematically overestimates robustness.
Information extraction. Attempts to surface system prompts, other users' data, or training content.
Harmful output. Probing whether the system can be led into producing content it should refuse, including through indirect framings that avoid the obvious triggers.
Action safety. For agentic systems, whether it can be induced to take actions outside its intended scope. Our work onAI agent workflow validation covers agentic behavior assessment.
Domain-specific harm. In regulated contexts, whether it can be led into unlicensed advice, unsupported claims, or disclosure that carries specific legal consequence.
Why Automation Is Not Sufficient
Automated adversarial testing exists and is useful: known attack libraries, fuzzing, and model-generated attack attempts all scale and catch the well-documented cases cheaply. They should be part of any programme.
They are insufficient alone for a straightforward reason. Automated attacks are variations on known patterns, and the attacks that succeed against a well-defended system are usually novel, contextual, and specific to that system's domain and purpose. Producing those requires people who understand the system, think adversarially, and are willing to be creative about it. The most effective attacks in practice tend to be socially framed rather than technically clever, and that is exactly what automation generates worst.
Structuring a Red Team Exercise
A useful exercise starts with a defined scope: what would constitute a serious failure for this system specifically, since the answer differs completely between a customer service assistant and a clinical tool. It covers both direct and indirect vectors, because teams routinely test the first and skip the second. It tests multi-turn, not just single prompts. It documents findings with reproduction steps, since a finding that cannot be reproduced cannot be fixed or verified. And it re-tests after fixes, because mitigations frequently displace an attack rather than eliminate it.
Findings should become permanent regression tests. An attack that worked once and was fixed should be tested forever, since model updates reopen old holes surprisingly often.
Guardrails That Hold Up Better
Red teaming usually points toward architectural rather than prompt-based mitigations. Constraining what actions a system can take limits the consequence of a successful injection more reliably than instructing it not to misbehave. Separating retrieved content from instructions structurally helps more than asking the model to ignore instructions in documents. Requiring confirmation for consequential actions bounds the damage. And output filtering independent of the model catches some of what the model itself will not refuse. Ourcontent moderation for generative AI output safety work covers the output layer.
Continuous, Not One-Time
Attack techniques evolve, models change underneath, and new capabilities create new exposure. A system red-teamed at launch and never again is defended against last year's attacks. Periodic re-testing, plus testing after significant changes, is the workable cadence. OurAI quality assurance function covers ongoing production assessment.
Common Questions From US Teams
What is prompt injection?
Text designed to override a system's instructions, exploiting the fact that models receive instructions and data in the same channel. It can be typed directly by a user or planted in content the system retrieves.
What is indirect prompt injection and why is it worse?
Instructions planted in retrievable content such as a web page or document. The user never sees it, the system retrieves it in normal operation and may follow it. For agentic systems the consequence is an unwanted action, not just a bad answer.
Why do guardrails fail so easily?
Because most are built on the model's own compliance. Prompt instructions can be argued with, filters are defeated by paraphrase, refusal training generalises imperfectly, and constraints erode across long conversations.
Can red teaming be automated?
Partly, and automated libraries and fuzzing should be part of any programme. But successful attacks against well-defended systems are usually novel and contextual, which is what automation generates worst.
What should a red team exercise cover?
Instruction override, indirect injection, multi-turn boundary erosion, information extraction, harmful output, action safety for agents, and domain-specific harm in regulated contexts.
Why test multi-turn rather than single prompts?
Because a model that refuses at turn one often complies at turn eight after context accumulates. Single-turn testing systematically overestimates how robust a system is.
What mitigations actually hold up?
Architectural ones: constraining available actions, structurally separating retrieved content from instructions, requiring confirmation for consequential actions, and independent output filtering.
How often should red teaming happen?
Periodically and after significant changes. Attack techniques evolve and model updates reopen previously fixed holes, so findings should also become permanent regression tests.
Working With Prudent Partners
Prudent Partners Private Limited provides structured adversarial testing for US teams deploying generative and agentic systems: direct and indirect injection testing, multi-turn boundary probing, harmful output and action safety assessment, with documented reproduction steps and findings converted into permanent regression cases. See ourcontent moderation for generative AI output safety andAI quality assurance work.
The first conversation is a 30-minute scoping call about your system, what would constitute a serious failure for it, and what has already been tested. No commitment to go further.