If you cannot evaluate the behavior, you cannot responsibly scale it.
An AI system can produce a convincing response while getting a consequential detail wrong. Engineering quality must therefore include what the system should decline, how it cites evidence and what happens after a model or data source changes.
What evidence would justify releasing this behavior?
Define acceptance with product, domain and security owners. Cover serious failure modes separately and rerun the relevant evidence when a dependency changes.
A release needs more than one accuracy score.
Grounded answer
- Representative case
- A question with known supporting material
- Evidence required
- Correct answer and relevant source support
- Acceptance owners
- Domain + product
Permission boundary
- Representative case
- The same question from a lower-privileged account
- Evidence required
- No inaccessible evidence in retrieval or response
- Acceptance owners
- Security + identity
Honest refusal
- Representative case
- Missing, ambiguous or conflicting evidence
- Evidence required
- A useful no-answer, clarification or escalation
- Acceptance owners
- Domain + design
Governed action
- Representative case
- A proposed write without required approval
- Evidence required
- No unauthorized action; a recoverable review state
- Acceptance owners
- Business + engineering
Operational recovery
- Representative case
- Timeout, changed source or failed dependency
- Evidence required
- Visible failure, bounded cost and documented recovery
- Acceptance owners
- Engineering + operations
Evaluate the evaluator, too.
Automated judges can help triage, but require calibration against human review in the target domain. Leaderboards and a single pass/fail score cannot substitute for application-specific evidence.
Ask which serious failure the set still cannot detect.
- A task-specific taxonomy and agreed acceptance criteria
- Representative cases and explicit negative cases
- Regression triggers, monitoring distinctions and rollback expectations
The complete evaluation and assurance argument
For CTO, CIO, CPO, COO
The situation behind the request
A product team is ready to launch a knowledge assistant. The demo succeeded on a handful of prompts, but nobody has documented error categories or what rate of mistakes is tolerable for the use case.
Where the obvious solution falls short
A single pass/fail accuracy score rarely captures the product. Retrieval may be wrong while the answer sounds plausible. A model can produce a correct statement using data the current user has no right to access. Functional correctness and authorization are separate dimensions.
How we would approach the work
We define acceptance criteria with product, domain and security owners. Build representative datasets, negative cases, human review procedures and regression checks. Establish monitoring that can distinguish model, retrieval, API and UX failures.
What the engagement may include
- Task-specific evaluation taxonomy and measurable acceptance thresholds.
- Grounded answer, retrieval, permission and action test cases.
- Regression testing across model, prompt, source and version changes.
- Incident/rollback expectations and observability plan.
The choices worth making explicitly
Model leaderboards are not a substitute for application evaluation. Automated judges can help triage, but their reliability must itself be checked against human review in the target domain.
What we would test before calling it ready
Test access violations, hallucinations, missing sources, ambiguous requests, model upgrades, cost spikes, latency, adverse prompts and recovery. Track outcomes that product owners actually care about.
A sensible first step
Choose a small evaluation set that represents high-value work and serious failure. Improve its coverage before scaling the application.