# A RAG evaluation dataset needs more than questions and answers.

> A knowledge system must retrieve evidence safely, correctly and consistently as sources change.

Canonical URL: https://marmation.com/review/rag-evaluation-datasets/

By Vikram Roy · 2026-11-12

A team creates twenty questions for a knowledge assistant and marks whether each answer sounds right. The score is encouraging. Then an employee asks about an old policy, a lower-privileged user receives a confidential snippet and a source revocation goes unnoticed. The original evaluation never asked those questions.

## The decision behind the technology

RAG is a pipeline. Search quality, source authority, permissions, answer evidence and model behavior can fail independently. A single answer-quality number hides the reason for failure and makes debugging harder.

## A better way to frame the work

Create a dataset with user identity, question, authorized source passages, expected answer characteristics, forbidden sources, acceptable no-answer behavior, source version and importance. Add negative scenarios and regression cases after every real issue. Track retrieval separately from generation.

## Put the idea under realistic pressure

Include a question whose only answer exists in a restricted document. The expected result for a non-privileged user is not a vague redacted summary; it is no unauthorized answer. Include two contradictory policy versions and check which one the system treats as current.

## When the recommendation changes

For informal low-risk search, a smaller dataset may be adequate. For systems that support costly decisions, human review and additional testing are justified. The evaluation set should grow with deployment complexity.

## What to decide before moving forward

Define expected evidence before choosing search or embedding changes. Keep source/version snapshots reproducible and verify authorization first.

### References and context

[Azure AI Search access controls](<https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview>); [NIST AI RMF](<https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence>).

## Sources

- [Azure AI Search document-level access](<https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview>) — Microsoft Learn. Enterprise search can align retrieval to source document permissions when correctly configured. Azure AI Search can support document access patterns; verify identity/preview limitations Limit: Feature/preview/connector status may change; review at build. Security depends on source permissions and configuration
- [Generative AI Profile for AI RMF](<https://www.nist.gov/publications/artificial-intelligence-risk-management-framework-generative-artificial-intelligence>) — NIST. Reference governance and risk design principles for gen AI. Risk management reference for generative AI Limit: Not legal certification, nor proof of Marmation compliance. Not certification of Marmation practices

## Related reading

- [From working prototype to dependable system](https://marmation.com/review/engineering/)
- [Design the action boundary before designing the agent.](https://marmation.com/review/designing-an-agent-with-bounded-actions/)
- [Engineering autonomy is a control design problem.](https://marmation.com/review/engineering-autonomy-is-a-control-design/)