Most retrieval-augmented systems are evaluated once — during the build, by the people who built them, on questions they already know the system handles. After launch, quality becomes anecdotal: someone reports a bad answer, the prompt is adjusted, and confidence is restored without evidence.
That is not a model problem. It is missing measurement infrastructure, and it is solvable with ordinary engineering.
Separate retrieval failures from generation failures
When an answer is wrong, there are two distinct causes: the relevant passage was never retrieved, or it was retrieved and the model used it badly. These require different fixes — chunking, indexing and query construction versus prompting, context ordering and model choice — so they must be measured separately.
In practice. Log the retrieved passage ids alongside every answer. Without that record, no post-hoc analysis of quality is possible — you are left debating outputs with no visibility into inputs.
Build a small, honest evaluation set
Fifty to two hundred real questions, drawn from actual users and annotated with the source that should answer them, is usually enough to detect regression. The set must include the awkward cases: questions with no answer in the corpus, questions whose answer changed, and questions that span several documents.
- Answerable from one source — the baseline case.
- Answerable only by combining sources — tests retrieval breadth.
- Not answerable at all — tests whether the system declines instead of inventing.
- Permission-restricted — tests that access boundaries hold under retrieval.
Run evaluation on every change, not every quarter
Index rebuilds, chunking changes, model upgrades and prompt edits all shift behaviour. If evaluation runs in the deployment pipeline, regressions surface as a failing check rather than as a support ticket three weeks later.
A knowledge system you cannot measure is a knowledge system you cannot maintain.
None of this requires research-grade tooling. It requires deciding, before launch, that answer quality is an operational metric with an owner — the same way latency and error rate are.
References
Method notes and evaluation-set structure used in AFC engagements are available on request.