The RAG Answer Was Correct. The Employee Was Not Allowed to See It.

The answer nobody should have received

Consider an internal assistant launched for an HR operations team. Its first demonstrations are convincing. Employees ask about leave, travel, and relocation. The assistant returns concise explanations with citations, and the rollout expands to another department.

Then someone asks whether the company will cover temporary accommodation during a move. The answer includes an unusually generous allowance. The cited document is real. It is also an executive employment agreement that ordinary employees should not be able to open.

This fictional investigation explores an authorization failure in internal search. It is not evidence that a particular organization experienced it. The scenario matters because the answer can be both factually supported and completely inappropriate to reveal.

The first debugging session focuses on relevance. Perhaps the executive agreement outranked the general policy because it contained more direct language. Someone proposes a better reranker. Someone else suggests telling the assistant to prefer official policies.

Those changes might reduce this particular occurrence. They would leave the underlying problem untouched: restricted text was available to the retrieval path. The model was being asked to exercise discretion over material the application should never have supplied.

I would take the question out of the chat interface and run it directly against the retrievers using the affected employee's identity. If the restricted chunk appears in the candidate set, we already have a concrete bug to fix. No debate about prompt quality is necessary.

Start with the caller, not the question

Resolve the caller's identity and access scope in trusted application code. Every document and chunk needs enough metadata to apply that scope: tenant, document identity, access policy, version, and source reference.

Do not let the model choose which tenant to search. Do not retrieve broadly and ask the model to omit restricted material. Once sensitive text reaches the model or a trace, you have already crossed the boundary you wanted to protect.

When permission metadata is unavailable or stale beyond your accepted window, fail closed or route to the underlying document system. A slightly less helpful answer is preferable to an unauthorized one.

Apply the same boundary to every retriever

Hybrid retrieval can help when queries mix precise identifiers with everyday language. A product code may benefit from lexical matching; a paraphrased question may benefit from semantic matching. Elastic documents reciprocal rank fusion as one way to combine ranked results.

But a permission filter on the keyword branch does not protect the vector branch. Apply equivalent restrictions across every candidate source, then rerank only authorized candidates. Check expansions, caches, and follow-up searches too.

I would test the retrieval layer directly with documents that are intentionally tempting matches but inaccessible to the caller. The expected result is their absence, not a polite refusal after retrieval.

Following the chunk through the system

In our example, the indexing pipeline copied the agreement from a document store with folder-based permissions. It extracted the text, split it into chunks, and saved the document title and source URL. It did not save the access policy.

That omission was easy to miss during development. The engineer testing the assistant had access to the whole folder. Every source link opened successfully, so citations appeared healthy. The test data never represented a user with fewer permissions.

Now imagine adding a permission field to new chunks but leaving older chunks unchanged. A filter that interprets missing metadata as public would expose the old content. A filter that rejects missing metadata would reduce available results until reindexing finishes. The second behavior is less convenient, but at least the migration has an explicit boundary.

The team also discovers a second retriever used for query expansion. The main search applies access filters, but the expansion service searches a cached collection built before those filters existed. This is why I would trace every candidate source instead of approving the design after finding one reassuring filter in the main query.

A complete fix needs an ingestion contract, a retrieval contract, and a policy for changes. How are permissions inherited? What happens when a document moves folders? How quickly does revocation reach the index? What does the product do while access metadata is uncertain?

These questions sound like document-management work because they are. RAG does not make the source system's obligations disappear. It creates additional copies and access paths that must continue honoring them.

A citation is a second access path

The citation link needs authorization as well. An application that protects its answer but hands out an unrestricted object-storage URL has moved the leak into a different feature.

Keep citation resolution tied to the current user. Verify access when someone opens the source. Where practical, show the exact section and document version used, so a reader can distinguish the policy from an old draft.

Avoid promising that a citation proves the answer. A citation proves where text came from. You still need to evaluate whether that text supports the claim being made.

Design for access changing tomorrow

Suppose someone changes teams after an answer is cached. A cache keyed only by question text can now return material from their previous permissions.

Include the relevant security scope or policy version in cache design, and invalidate appropriately when access changes. For especially sensitive sources, recheck authorization before serving a cached answer. Also consider whether conversation history itself contains material that should no longer be reused.

This is a product decision as much as an indexing decision. Define your access-change expectations before choosing a cache lifetime.

Test quality and permission together

Build an evaluation set with public policies, restricted agreements, duplicate drafts, and revoked access. Ask the same question as different users. Check retrieved IDs, generated claims, citations, logs, and cached follow-ups.

Measure answer usefulness and unauthorized retrieval separately. A single blended score can hide an unacceptable access failure behind many good answers.

Start with a small authorized corpus and a plain keyword baseline. Add hybrid retrieval when your actual questions justify it. Better ranking is valuable, but the first requirement is simpler: the right answer must also be an answer this person is entitled to receive.

A more useful launch review

Before the next rollout, the team creates two test identities with different access. They ask the same questions, inspect candidate IDs, and open citations. They then revoke one permission and repeat the sequence through a cached answer and a follow-up conversation.

That last step is important. A newly secured retriever does not automatically remove restricted text already sitting in conversation state. The product needs an explicit policy for what can be reused after access changes, and it needs tests for that policy.

In a small pilot, I would rather support fewer collections with clear ownership than ingest every drive and wiki at once. A source should have a named owner, understood permissions, and a practical way to propagate updates. Broad coverage is not useful when nobody can explain why a result was visible.

There is also a relevance benefit to this discipline. A clear distinction between an official policy, an individual agreement, and an obsolete draft helps the assistant choose the right evidence among documents the user is authorized to see.

When the employee asks about accommodation again, the answer should come from the applicable relocation policy. If that policy is ambiguous, the assistant can say so and point to HR. The product's job is not to surface the most specific sentence anywhere in the company. It is to help this employee answer this question using information they are allowed to use.

Sources & further reading

Worked scenarios are illustrative. Technical references were checked on September 10, 2026.

Working through a similar problem?

Tell me what you are building and where it gets stuck.

Let’s talk ↗