A RAG evaluation dataset needs more than questions and answers.

A team creates twenty questions for a knowledge assistant and marks whether each answer sounds right. The score is encouraging. Then an employee asks about an old policy, a lower-privileged user receives a confidential snippet and a source revocation goes unnoticed. The original evaluation never asked those questions.

The decision behind the technology

RAG is a pipeline. Search quality, source authority, permissions, answer evidence and model behavior can fail independently. A single answer-quality number hides the reason for failure and makes debugging harder.

A better way to frame the work

Create a dataset with user identity, question, authorized source passages, expected answer characteristics, forbidden sources, acceptable no-answer behavior, source version and importance. Add negative scenarios and regression cases after every real issue. Track retrieval separately from generation.

Put the idea under realistic pressure

Include a question whose only answer exists in a restricted document. The expected result for a non-privileged user is not a vague redacted summary; it is no unauthorized answer. Include two contradictory policy versions and check which one the system treats as current.

When the recommendation changes

For informal low-risk search, a smaller dataset may be adequate. For systems that support costly decisions, human review and additional testing are justified. The evaluation set should grow with deployment complexity.

What to decide before moving forward

Define expected evidence before choosing search or embedding changes. Keep source/version snapshots reproducible and verify authorization first.

References and context

Azure AI Search access controls; NIST AI RMF.