Part 2 · Topic 5 of 8
Progressive retrieval
Progressive retrieval: accuracy and cost are not a trade-off
1 min read
There is a widespread assumption that better retrieval means more context, and more context means more tokens. I think the assumption is wrong, and that the systems with the best retrieval quality are often also the cheapest, because they spend tokens where they matter.
The pattern is progressive retrieval: start with the cheapest representation that might answer the question and escalate only when it does not.
| Stage | What is retrieved | Approximate cost | Stop if |
|---|---|---|---|
| Intent and constraints | Nothing; classify the query and extract filters | Tens of tokens | Never; always run |
| Metadata and entity filter | Candidate document set | Negligible | Candidate set is empty (abstain or broaden) |
| Summary retrieval | Document and section summaries of candidates | Hundreds of tokens | Summary answers with confidence and a citation, and the question is low-stakes |
| Chunk retrieval | Targeted chunks from top-ranked sections | Low thousands of tokens | Chunk evidence supports the answer at the required grounding threshold |
| Full-source retrieval | Entire document or long section | Many thousands of tokens | Required by the question type: audit, legal, complex reasoning across a document |
Two things make this work. Multi-granularity representations from the pipeline — you cannot retrieve summaries you did not store. And a decision rule at each stage that uses retrieval and grounding signals to decide whether to stop. That rule is where the accuracy comes from: the system escalates when confidence is low, and a system that escalates on low confidence is more accurate than one that always retrieves ten chunks and hopes.
A per-query token budget, set by use-case priority, closes the loop. A routine question gets a small budget and stops at summaries. A high-stakes question gets a large budget and goes to source. The budget is a product decision, made explicit, instead of an accident of the chunk size.