Unstructured Document Management

Most enterprises hold their most consequential knowledge in the form they manage least. Contracts, policies, procedures, engineering decisions, support conversations, meeting transcripts, regulatory filings, product specifications — the reasoning behind how an organization actually works lives in documents, and documents have never been treated as an asset class. Structured data got warehouses, lineage, quality gates, ownership, and a decade of investment. Documents got a file share, a search box, and a retention policy.

That asymmetry was tolerable when the primary consumer of a document was a person who could open it, skim it, and decide whether to trust it. It is not tolerable now that the primary consumer is increasingly a language model, which will read whatever it is handed, treat a superseded policy with the same confidence as the current one, and produce a fluent answer either way.

This perspective is about closing that gap. My position is that unstructured document management is an enterprise capability, not an AI feature, and that the organizations getting real value from language models are the ones that stopped asking "which model?" and started asking "what are we feeding it, and how do we know it is right?"

The problem is upstream of the model

When a copilot gives a wrong answer, the instinct is to blame the model or the prompt. In my experience, when you trace the failure back, the model is rarely the root cause. The failure is almost always one of these:

The document was parsed badly, so the table that held the answer became a paragraph of numbers with no column headers. The chunk that was retrieved was cut mid-clause, and the qualifying sentence lived in the next chunk. Retrieval returned four near-identical copies of an old version and none of the current one. The right document existed but nobody had tagged it with the attribute that the filter needed. The answer was correct for the region the document covered, and the user was in a different region, and nothing in the system knew that mattered.

None of these are model problems. They are document management problems that surface as model problems. A better model with the same inputs produces a more confident wrong answer.

The corollary is the thing I keep coming back to: a well-engineered corpus with a modest model outperforms a poorly-engineered corpus with the most expensive model available. I have watched teams spend a quarter on prompt engineering and model evaluation while their ingestion pipeline silently dropped every second page of every scanned PDF. The leverage is upstream.

Three disciplines, one asset

I think about this as three disciplines that have to work together, each with its own engineering practice and its own measures of quality. They map to the three sub-perspectives under this theme.

Document understanding pipelines are where raw bytes become structured, reusable representations. Parsing, layout recovery, normalization into a canonical document model, deduplication, chunking, and the one-time extraction of signals — classification, entities, dates, summaries — that downstream consumers would otherwise re-derive over and over. This is the discipline where the most quality is lost and the least measurement happens. It is also the discipline that determines your token bill, because every redundant re-parse and re-summarize is paid for in model calls.

Knowledge base engineering is the discipline of turning those representations into a durable asset with an owner, a lifecycle, and a contract. A knowledge base is not a vector index. It is a governed set of representations — the raw source, its normalized form, its sections and chunks, its summaries, its extracted entities and relationships, its embeddings — held together by provenance and context, with clear rules about who owns what, what supersedes what, and how a consumer discovers what already exists. Without this discipline, every team builds its own pipeline against the same documents and the organization ends up with five inconsistent interpretations of one policy.

Retrieval quality is the discipline of making the retrieval layer a measured product surface rather than an implementation detail. Relevance: did we find the right material? Grounding: did the answer actually come from what we found? Freshness: is what we found still true? Each of these is measurable, each has a distinct set of techniques, and each degrades silently if it is not instrumented. Retrieval is where the other two disciplines are tested, and where the feedback comes from to improve them.

Principles I hold across all three

A few positions run through everything I write under this theme. I will state them here so the sub-perspectives can assume them.

Centralize the pattern, not the ownership. The teams closest to the content know what it means, when it changes, and what "correct" looks like. They should own ingestion decisions, domain vocabulary, and review workflows. What must be shared is the pattern: the document model, the metadata contract, the chunk format, the provenance schema, the retrieval API, the evaluation harness. Decentralized ownership on a shared platform is the only model I have seen scale without fragmenting. Centralized ownership becomes a bottleneck; decentralized everything becomes chaos.

Store multiple representations, serve the cheapest one that answers. Chunks and embeddings are not a knowledge base. Keep the raw document, the normalized structure, the sections, the chunks, the summaries at several granularities, the extracted entities, and the embeddings — together, linked, with lineage. Then let retrieval escalate: metadata first, then summaries, then chunks, then the full source only when the question demands it. This is how you get both accuracy and token efficiency, rather than trading one for the other.

Extract once, reuse everywhere. If a document has been classified, its entities pulled, its sections summarized, that work should be stored as annotations on the document and never redone by the next consumer. The alternative — every agent independently re-reading and re-summarizing the same material — is the single largest source of wasted model spend I encounter, and it also produces inconsistency, because each re-derivation is slightly different.

Retrieval quality is a measured, first-class concern. If you cannot state your recall at ten on a held-out query set, your grounding rate on production answers, and your freshness compliance by content class, you are not managing retrieval; you are hoping. Build the evaluation set before you tune anything. Instrument every retrieval. Treat regressions the way you would treat a failing build.

Governed by default, not by exception. Access control at retrieval time, provenance on every representation, versioning with supersession, audit logs of what was retrieved and what was generated from it. These are cheap to build in and expensive to retrofit, and in any regulated context they are not optional. Design for them on day one even if the first use case does not need them.

Build for agents, not just for chat. The consumer of the knowledge layer is increasingly a multi-step process that retrieves, reasons, retrieves again, and acts. That consumer needs structured results, citations it can carry forward, confidence it can act on, and interfaces described well enough that it can choose the right tool. A knowledge layer designed only for a human typing a question into a box will be rebuilt within a year.

What this is not

It is worth saying what I am not arguing for. I am not arguing for a single enterprise document repository — those projects fail on migration cost and political friction, and they were never necessary. I am not arguing for a specific vector database, embedding model, or framework; those choices matter less than the practices around them, and they change every six months. And I am not arguing that this is primarily a technology problem. The hardest part is the operating model: getting a domain team to accept accountability for the freshness and correctness of their content, and getting a platform team to provide contracts that make that accountability tractable.

How to read the sub-perspectives

Each of the three pieces beneath this one goes deep on its discipline: the specific positions I hold, the mechanisms I would build, the measures I would put in place, and the mistakes I have seen repeated. They are written to be read independently, but they share the vocabulary and principles above. If you only have time for one, start with retrieval quality — it is where you will discover which of the other two you need most.