The evaluation loop

Offline evaluation on the golden set runs on every change to the retriever, the reranker, the chunker, the embedding model, or the corpus — the same way tests run on every commit. A regression in recall at ten or in grounding rate blocks the change.

Online evaluation runs continuously on sampled production traffic: automated grounding checks, freshness compliance, abstention rate, user feedback where it exists. Divergence between offline and online numbers means the evaluation set has drifted from reality and needs refreshing.

Human review closes the loop. A rotating sample of production traces, read by someone who knows the domain, will surface failure modes that no metric anticipated. The findings feed back as new evaluation queries, new metadata fields for the pipeline to extract, new entries in the source-of-truth ranking.