Pipeline engineering

Everything above fails if the pipeline itself is a script that someone runs. A document understanding pipeline is a data pipeline, and it needs the properties of one.

Idempotent. Processing the same document twice produces the same result and does no duplicate work. The content hash and the parser/extractor versions are the idempotency key: if the hash is unchanged and the versions are unchanged, skip. This is what lets you re-run the whole thing after a failure without paying for it twice.

Versioned. Every stage records the version of the component that produced its output. When you upgrade the chunker, you know exactly which chunks are stale. When an extractor is found to have a bug, you can find every annotation it produced. This is the difference between a targeted reprocessing job and a full rebuild.

Replayable. Raw source bytes are preserved, immutably. The canonical model can always be regenerated from them. The chunks can always be regenerated from the canonical model. Nothing derived is the only copy of anything.

Event-driven and incremental. Sources change. The pipeline should react to change events — new document, updated document, deleted document — and process only what changed, propagating through the derived representations. Batch re-ingestion of the whole corpus is a recovery mechanism, not the normal path.

Observable at every stage. Counts in, counts out, failures by stage, latency by stage, cost by stage, parse confidence distribution, chunk size distribution, extraction confidence distribution. A pipeline that only reports "ran successfully" is hiding its failures.