# If you cannot evaluate the behavior, you cannot responsibly scale it.

> An AI system can produce a convincing response while getting a consequential detail wrong. Engineering quality must therefore include what the system should decline, how it cites evidence and what happens after a model or data source changes.

Canonical URL: https://marmation.com/solutions/ai-evaluation-assurance/

## The decision

What would make this feature ready to release for its intended task?

Agree the important failures with product, domain and security reviewers. Test them separately, record the limits and repeat the relevant checks when a model, source or integration changes.

## A successful demo is not yet a release decision

A product team is ready to launch a knowledge assistant. The demo succeeded on a handful of prompts, but nobody has documented error categories or what rate of mistakes is tolerable for the use case.

## One accuracy score can hide a serious product failure

A single pass/fail accuracy score rarely captures the product. Retrieval may be wrong while the answer sounds plausible. A model can produce a correct statement using data the current user has no right to access. Functional correctness and authorization are separate dimensions.

## Agree useful behavior and repeat the checks after change

Agree acceptable behavior with the people responsible for the product, the business task and security. Build a representative test set, including requests that should be refused and changes that need approval. Use human review for judgments that cannot be checked mechanically. Keep cases that can be repeated after a change to catch regressions. Monitoring should distinguish a model error from a missed source, failed API or confusing interface, so the team can act on the right cause.

## The evaluation coverage needed around a real task

- Named categories of task failures (an evaluation taxonomy) and measurable limits for accepting the release.
- Grounded answer, retrieval, permission and action test cases.
- Regression testing across model, prompt, source and version changes.
- Incident responsibilities, rollback conditions, monitoring and diagnostic records.


## Check automated judges against qualified human review

Model leaderboards are not a substitute for application evaluation. Automated judges can help triage, but their reliability must itself be checked against human review in the target domain.

## Test harmful answers, denied access, delays and recovery

Test access violations, hallucinations, missing sources, ambiguous requests, model upgrades, cost spikes, latency, adverse prompts and recovery. Track outcomes that product owners actually care about.

## Cover valuable tasks and serious failures before scaling

Choose a small evaluation set that represents high-value work and serious failure. Improve its coverage before scaling the application.

## Related reading

- [Make AI work with the systems your business already runs.](https://marmation.com/solutions/enterprise-ai-systems/)
- [Retrieval cannot be allowed to widen a user's permissions.](https://marmation.com/expertise/permission-aware-rag/)
- [Give an agent a job. Give the system boundaries.](https://marmation.com/solutions/ai-agents-and-workflows/)