Traditional application security assumes you control what data reaches which user, because the application decides. Retrieval and generative AI systems complicate that assumption in a specific way: the system retrieves material dynamically and synthesises an answer, so a user can receive information they were never explicitly granted, expressed in a form that no access rule anticipated. The document was protected. The sentence summarising it was not. This guide covers how AI systems leak, and what prevents it.

The Ways AI Systems Expose Data

Retrieval crossing access boundaries. The most common and most preventable. A knowledge base is assembled from many sources with different sensitivity, the retrieval layer treats it as one corpus, and a user's question surfaces content they should not see. Ourretrieval-augmented generation overview covers the architecture where this occurs.

Synthesis across permissions. Subtler. Each retrieved fragment might be individually permissible while the combination reveals something that is not, such as assembling a picture of an unannounced organisational change from separately innocuous documents.

Training data surfacing in output. Models can reproduce content from what they were tuned on, which is a reason to be deliberate about what enters fine-tuning data rather than treating it as a private input.

Prompt injection. Instructions embedded in retrieved content or user input that redirect the system's behavior, potentially causing it to reveal material or take actions it should not. This is the AI-specific attack class with no clean equivalent in conventional software.

Logging and traces. Prompts, retrieved context, and outputs are commonly logged for debugging, which means sensitive content ends up in log systems governed far less carefully than the source data.

Third-party model providers. Data sent to an external model leaves your environment. What the provider retains and whether it is used for training is a contractual question that should be settled explicitly.

Access Scoping as a Security Control

The central design principle: in retrieval architectures, access control belongs at the retrieval layer, not only at the document store.

That means retrieval is filtered by the requesting user's permissions before ranking, not after, so restricted material never enters the candidate set. It means access metadata is attached to content during knowledge base preparation rather than bolted on later, which makes preparation a security activity and not just a data-modelling one. And it means the system's behavior is tested with users at different permission levels asking the same questions, which is the only way to know the scoping actually holds.

Teams that treat access as a post-processing filter, removing restricted content from an answer after generation, are building on sand. By then the model has already seen the material and can leak it through paraphrase.

Governing What Enters the System

Much of the exposure is decided before any query runs, at the point where content enters the knowledge base or the training set. Practical governance means knowing what is in the corpus, which requires a real inventory rather than a folder that has accumulated. It means classifying content by sensitivity as part of ingestion. It means deliberately excluding material that has no business being retrievable, since the safest content is content that is not in the system. And it means keeping the corpus current, because stale sensitive content carries the same risk as current sensitive content with none of the value.

Prompt Injection and Adversarial Testing

Prompt injection deserves specific attention because it is genuinely new. Content the system retrieves may itself contain instructions, and a model that does not distinguish between instructions and data can follow them. The mitigations are architectural, including clearly delineating retrieved material from instructions and constraining what actions the system can take, but they need testing to be trusted.

Adversarial testing means deliberately attempting to make the system misbehave: embedding instructions in documents it will retrieve, probing for content the tester should not be able to reach, attempting to extract training or system material. This is human work requiring people who think adversarially, and it is the only reliable way to know whether the defenses hold.

Regulated Data in AI Systems

For health, financial, and legal data the questions become concrete. Which regulated material is in the corpus at all, and does it need to be? Can access scoping enforce the relevant restrictions reliably? What is logged, and is the log store governed to the same standard as the source? Where do model calls go, and what does the provider retain? For healthcare specifically, ourmedical data annotation work covers the handling requirements for protected health information.

Monitoring After Deployment

Security posture at launch decays. Content is added to the corpus without classification, permissions change, models are updated, and new attack techniques appear. Ongoing monitoring means sampling real outputs for inadvertent disclosure, re-testing access scoping periodically, and re-running adversarial tests after significant changes. This connects to broader production quality work covered in ourAI quality assurance function.

Where Human Review Fits

Two parts resist automation. Classifying content sensitivity accurately requires understanding what the content means in context, which is judgment work. And detecting subtle disclosure in generated output, where nothing is quoted verbatim but the substance is revealed, requires a reviewer who knows what should not have been said. Automated scanning catches the obvious cases and misses precisely these.

Common Questions From US Teams

How do RAG systems leak sensitive data?

Most commonly when retrieval crosses access boundaries, surfacing content a user should not see. Also through synthesis across permissions, prompt injection, logging of retrieved context, and data sent to third-party model providers.

Where should access control live in a RAG system?

At the retrieval layer, filtering by the requesting user's permissions before ranking, so restricted material never enters the candidate set. Filtering after generation is unsafe because the model has already seen the content.

What is prompt injection?

Instructions embedded in retrieved content or user input that redirect the system's behavior, potentially causing disclosure or unintended actions. It is the AI-specific attack class with no clean conventional equivalent.

Can synthesis leak information even when each source is permitted?

Yes. Individually innocuous fragments can combine to reveal something that is not permitted, which is why access design has to consider combinations rather than only documents.

Does logging create a security risk?

Frequently. Prompts, retrieved context, and outputs are commonly logged for debugging, placing sensitive content in log systems governed far less carefully than the source data.

How do I test whether access scoping actually works?

Have users at different permission levels ask the same questions and compare what surfaces. Adversarial testing should also attempt to reach content the tester is not entitled to see.

What should I govern at ingestion?

Know what is in the corpus through a real inventory, classify content by sensitivity as part of ingestion, exclude material that has no business being retrievable, and keep the corpus current.

Can AI security be automated?

Partly. Automated scanning catches obvious disclosure. Classifying sensitivity accurately and detecting subtle paraphrased disclosure both require human judgment from reviewers who know what should not have been said.

Working With Prudent Partners

Prudent Partners Private Limited supports US teams securing AI systems at the data layer: content inventory and sensitivity classification during knowledge base preparation, access metadata structuring, adversarial testing including prompt injection attempts, and human review of production output for inadvertent disclosure. See ourretrieval-augmented generation overview andAI quality assurance function.

The first conversation is a 30-minute scoping call about your system, your corpus, and the data governing your obligations. No commitment to go further.