What to measure

If I had to pick a small set of pipeline metrics to put on a dashboard, it would be these, each sampled and reviewed by a person on a schedule rather than trusted blindly.

Parse fidelity on a stratified sample: does the canonical tree match the rendered page? Table recovery rate. Reading-order errors. Header/footer contamination rate. OCR confidence distribution, and the share of pages below your threshold.

Deduplication: exact and near-duplicate rates by source; review-queue volume and resolution time; false-merge incidents (there should be a process for reporting one).

Chunk quality: size distribution; share of chunks that split a table or a list; share of chunks with an empty or generic context prefix; a small human-judged sample for coherence — does this chunk make sense on its own?

Extraction: precision and recall per extractor against a labeled set, re-measured on every extractor upgrade; confidence calibration; disagreement rate where two extractors overlap.

Economics: cost per document by stage; share of pages routed to each parser family; share of model calls that were skipped because annotations already existed. That last one is the number that tells you whether "extract once" is working.