Part 2 · Topic 6 of 8

Instrumentation

Instrument everything

1 min read

None of the above is manageable without traces. Every retrieval should log the query, the understood query and extracted constraints, the filters applied, the candidates from each retriever with scores, the fused and reranked order, what was actually placed in the context and why, the generated answer with its citations, the grounding check result, and the tokens consumed at each stage. That trace is what lets you answer "why did it say that?" for any single query, and it is the raw material for every metric above when aggregated.

Here is the shape of the contract I would put between the retrieval service and its consumers. The point is that quality signals and lineage travel with the results, so a consumer — human interface or agent — can act on them.

/** A single retrieved unit, with enough lineage and signal to be cited,
 *  verified, and reasoned about by an agent. */
export interface RetrievedPassage {
  chunkId: string;
  docId: string;
  docVersion: string;
  level: "document" | "section" | "block";
  text: string;
  /** Breadcrumb and situating context captured at ingestion. */
  context: string;
  /** Where in the source this came from; enough to render a citation. */
  locator: { page?: number; sectionPath: string[]; charStart?: number; charEnd?: number };
  /** Per-retriever and fused scores, so consumers can gauge confidence. */
  scores: { lexical?: number; dense?: number; fused: number; rerank?: number };
  /** Freshness signals; consumers should not trust content without them. */
  effectiveFrom?: string;   // ISO 8601
  effectiveTo?: string;     // ISO 8601; undefined = currently effective
  supersededBy?: string;    // docId of the newer version, if any
  sourceOfTruthRank: number; // lower is more authoritative
  /** Parser/OCR confidence from ingestion; low values should be surfaced. */
  parseConfidence: number;
}

export interface RetrievalRequest {
  query: string;
  asOf?: string;            // effective-dated retrieval; defaults to now
  filters?: Record<string, string | string[]>;
  tokenBudget: number;      // hard cap on context assembled for the caller
  minGrounding?: number;    // caller's threshold for the grounding check, 0..1
  maxStage?: "summary" | "chunk" | "source"; // cap on progressive escalation
}

export interface RetrievalResponse {
  traceId: string;
  understood: { intent: string; entities: string[]; constraints: Record<string, string> };
  stageReached: "summary" | "chunk" | "source";
  passages: RetrievedPassage[];
  /** Explicit abstention beats a low-confidence answer. */
  abstained: boolean;
  abstainReason?: "no-candidates" | "low-confidence" | "conflicting-sources";
  /** Conflicts found among currently-effective sources; surface, don't hide. */
  conflicts?: Array<{ docIds: string[]; note: string }>;
  tokensUsed: { retrieval: number; context: number };
}

The consumer-facing fields that matter most are abstained, conflicts, supersededBy, and the score breakdown. A retrieval API that returns only text is asking the consumer to trust it. A retrieval API that returns text with these signals is asking the consumer to reason.