Part 2 · Topic 2 of 8
Relevance
Relevance: did we find the right material?
3 min read
Relevance is the dimension people think they are measuring when they say "accuracy." It is the classic information-retrieval problem and it has classic metrics, which I would use unchanged.
| Metric | What it tells you | When to lean on it |
|---|---|---|
| Recall@k | Of the relevant items, how many appeared in the top k | The primary metric when a downstream model reads all k results; a miss here cannot be recovered |
| Precision@k | Of the top k, how many are relevant | Matters when the context window is tight and every irrelevant chunk displaces a relevant one |
| MRR (mean reciprocal rank) | How high the first relevant result sits | Single-answer questions; user-facing search where position matters |
| nDCG@k | Rank-weighted relevance with graded judgments | Multi-document questions where some evidence is better than other evidence |
| Coverage | Share of queries with at least one relevant result in top k | The number to watch when the corpus has gaps |
For a system where a model reads the retrieved context, recall at the context size is the number I care about most. A relevant chunk that lands at position twelve when you pass ten to the model is a miss. Precision matters second, because irrelevant context is not free: it costs tokens and it distracts the model.
Now the mechanisms.
Hybrid retrieval by default. Dense vector search is good at meaning and bad at exactness. It will find "termination for convenience" when you asked about "cancelling the agreement early," which is what you want. It will also rank a paragraph about a different product's part number above the one paragraph that contains the exact part number you typed, which is not. Lexical search (BM25 or equivalent) has the opposite profile. The identifier, the code, the proper noun, the specific version string — these are the queries where vector-only retrieval fails, and they are a large share of enterprise queries. Run both, fuse the rankings — reciprocal rank fusion is simple and robust — and rerank the fused candidates with a cross-encoder that scores query and passage together. This three-stage shape (lexical + dense → fusion → rerank) is the baseline I would start every system from, and I would need evidence to remove any of the stages.
Metadata filtering before semantic search. Most enterprise queries have implicit constraints: this product line, this jurisdiction, this document type, currently effective. When those constraints are captured as metadata at ingestion, applying them as a filter before retrieval shrinks the candidate space dramatically and removes an entire class of wrong answers — the right paragraph from the wrong region — that no amount of reranking will fix. This depends on the pipeline having extracted the metadata, which is why I put the pipeline first.
Query understanding. The user's query is rarely the best retrieval query. A light query-understanding step — classify intent, extract entities and constraints, expand abbreviations, and where the query is complex decompose it into sub-queries — pays for itself. It also feeds the metadata filter. Done with a small model or rules, it costs almost nothing.
Deduplication-aware ranking. If the corpus has near-duplicate content — and every corpus does — the top ten results will often be five variants of the same passage. That is wasted context and it crowds out the second-best source. Collapse near-duplicates at ranking time (by canonical document identity where the pipeline established it, or by embedding similarity among candidates where it did not) and apply a diversity criterion such as maximal marginal relevance so that the returned set covers more of the question.
