Part 1 · Topic 2 of 8

Canonical document model

Normalize to a canonical document model before doing anything else

2 min read

The single most important design decision in the pipeline is that everything downstream operates on a canonical document model, not on the source format and not on flat text.

A canonical document model is a tree: document → sections → blocks (paragraphs, tables, lists, figures, captions) → spans, with structural metadata at each node (heading level, page, bounding box, table cell coordinates, list depth) and document-level metadata (source system, source identifier, version, effective dates, content hash). Every parser, whatever family it belongs to, emits this model. Every chunker, extractor, summarizer, and indexer consumes it.

This buys three things.

It decouples parsing from everything else. When a better parser appears — and one appears every few months — you swap it in behind the model and nothing downstream changes. When you add a new source format, you write one adapter.

It preserves structure through chunking. A chunker operating on the tree knows it is inside a table, or inside a section titled "Exclusions," or at list depth two. A chunker operating on flat text knows none of that and cuts wherever the token count says to cut.

It makes lineage possible. Every chunk, every extracted entity, every summary can point back to the exact nodes of the tree it came from, and the tree points back to the source bytes. When a consumer asks "where did this come from," you can answer with a page and a bounding box, not a document title.

Here is the shape of it. I keep the model deliberately small; the temptation to make it a universal schema for everything is what kills these projects.

from __future__ import annotations

import hashlib
from dataclasses import dataclass, field
from enum import Enum
from typing import Optional


class BlockType(str, Enum):
    PARAGRAPH = "paragraph"
    HEADING = "heading"
    TABLE = "table"
    LIST_ITEM = "list_item"
    FIGURE = "figure"
    CAPTION = "caption"
    CODE = "code"


@dataclass(frozen=True)
class Locator:
    """Where a node lives in the source. Enough to render a citation
    and to re-extract the region if a downstream consumer needs to verify."""
    page: Optional[int] = None
    bbox: Optional[tuple[float, float, float, float]] = None  # x0, y0, x1, y1 in page units
    char_start: Optional[int] = None  # offset in normalized text
    char_end: Optional[int] = None


@dataclass
class Block:
    block_id: str
    type: BlockType
    text: str                      # normalized text; for tables, a deterministic serialization
    locator: Locator
    level: Optional[int] = None    # heading level or list depth
    table: Optional[dict] = None   # structured cells when type == TABLE
    confidence: float = 1.0        # parser/OCR confidence; 1.0 for born-digital text


@dataclass
class Section:
    section_id: str
    title: Optional[str]
    path: list[str]                # breadcrumb of heading titles from root
    blocks: list[Block] = field(default_factory=list)
    children: list["Section"] = field(default_factory=list)


@dataclass
class CanonicalDocument:
    doc_id: str                    # stable, source-derived identity
    source_system: str
    source_ref: str                # id in the source system
    version: str                   # source version if available, else content hash
    content_hash: str              # sha256 of normalized text — the dedup key
    parser: str                    # which parser family and version produced this tree
    metadata: dict = field(default_factory=dict)
    root: Section = field(default_factory=lambda: Section("root", None, []))

    @staticmethod
    def hash_text(text: str) -> str:
        # Normalize whitespace and case before hashing so trivial
        # re-exports of the same document collapse to one identity.
        normalized = " ".join(text.split()).lower()
        return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

Two details matter more than they look. The parser field is there so that when you upgrade a parser you can find every document produced by the old one and reprocess only those. The confidence on each block is there so that retrieval can down-weight or flag content that came from a low-confidence OCR pass instead of presenting it with the same authority as clean text.