The evaluation set

The order of operations matters more than any technique. If you tune the retriever before you have a way to measure it, you will tune it toward whatever the last complaint was, and you will not know whether you made anything else worse.

An evaluation set for retrieval is a collection of queries, each paired with the set of documents or chunks that a competent person would consider correct evidence, and ideally with a reference answer and its citations. It does not have to be large to be useful. A few hundred well-chosen queries with graded relevance will catch most regressions. It does have to be representative: drawn from real query logs where they exist, stratified by the kinds of questions the system actually receives, and refreshed as the corpus and the users change.

Three sources, in order of value. Real queries from production logs, judged by domain experts, are the gold standard and the most expensive. Queries written by domain experts against documents they know are the practical starting point when no logs exist yet. Synthetic queries generated by a model from the corpus are the cheapest and the most dangerous; they are fine for smoke-testing and catastrophically misleading as a sole measure, because the model that wrote the question from a chunk will find that chunk. If you use synthetic queries, verify a sample by hand and never let them be the only set.

One more thing the set needs: queries that have no answer in the corpus, and queries whose answer changed between versions. These are the cases that test grounding and freshness, and they are the ones nobody includes.