Part 1 · Topic 3 of 8
Deduplication and versioning
Deduplicate and version at the door
1 min read
The same document enters an enterprise many times. It is emailed, saved to a share, uploaded to a portal, attached to a ticket, exported to PDF, re-exported with a new footer. Each copy gets ingested by whichever team found it, and each team's pipeline parses, chunks, embeds, and summarizes it again. Retrieval then returns five near-identical chunks from five copies, crowding out the material that would have completed the answer.
Deduplication belongs at ingestion, before any expensive processing, and it needs three layers.
Exact duplicates are caught by the content hash on normalized text. Cheap, deterministic, catches the majority.
Near duplicates — same content, different header, different export date, one corrected typo — need a similarity fingerprint. MinHash over shingles or SimHash over tokens gives you a compact signature and a locality-sensitive index that finds candidates in sub-linear time. Threshold it conservatively and route borderline pairs to a review queue rather than auto-merging; the cost of wrongly merging two different contract versions is much higher than the cost of a duplicate.
Versions are the hard case, because they are near duplicates that must not be merged. Two revisions of a policy share ninety percent of their text and differ in the sentence that matters. The signal that distinguishes a version from a duplicate is usually in metadata — a version number, an effective date, a source-system revision id — and occasionally in structure (a new section, a changed table). The pipeline should detect the near-duplicate relationship, then ask: is there version metadata that orders these? If yes, record a supersession link and keep both, with the older marked superseded. If no, queue it for a human, because a machine cannot safely decide.
The output of this stage is a single canonical identity per distinct document, a version chain per evolving document, and a mapping from every ingested copy back to its canonical. Every downstream representation is keyed to the canonical, so the work is done once.
