Skip to content
AP.

Part 1 · Topic 4 of 8

Chunking

Chunking is a retrieval design decision, not a preprocessing default

2 min read

Most pipelines chunk by token count with overlap because that is what the first tutorial did. Fixed-size chunking is a decision to ignore the document's structure, and the model pays for it on every query: chunks cut mid-clause, tables split from their headers, a qualifying sentence separated from the rule it qualifies.

I hold three positions on chunking.

Chunk the tree, not the text. With a canonical document model, the natural chunk boundaries are structural: a section, a paragraph group under a heading, a table with its caption, a list with its introductory sentence. Size limits still apply, but they are applied within structure — split a long section at paragraph boundaries, never at token counts — and structure is never split across chunks when it can be avoided. A table is one chunk, or if it is too large, it is split by rows with the header repeated in every piece.

Every chunk inherits its context. A chunk of text that says "the above limits do not apply to renewals" is useless on its own. The chunk record should carry the breadcrumb of headings above it, the document title and type, the effective dates, and a short prefix that states what the chunk is about. Whether you prepend that context to the text that gets embedded, store it as metadata for the retriever to use, or both, is an empirical question you should test on your own corpus. What is not optional is capturing it. The technique sometimes called contextual chunking — generating a one-sentence situating statement per chunk with a model — is one way to do this; the structural breadcrumb is a cheaper way that gets most of the benefit for many document types.

Chunk at multiple granularities. Store document-level, section-level, and paragraph-level chunks, and link them parent-to-child. A retriever can then match at the fine level and return the coarse level — find the sentence, hand the model the section — or match at the summary level and drill down. This hierarchical approach is the foundation of progressive retrieval, which I cover under retrieval quality, and it is much easier to build here than to retrofit.

from dataclasses import dataclass, field
from typing import Iterator


@dataclass
class Chunk:
    chunk_id: str
    doc_id: str
    doc_version: str
    parent_chunk_id: str | None      # section or document chunk this belongs to
    level: str                       # "document" | "section" | "block"
    text: str                        # the content to embed / return
    context_prefix: str              # breadcrumb + doc type + situating line
    block_ids: list[str]             # lineage back to the canonical tree
    metadata: dict = field(default_factory=dict)

    @property
    def embed_text(self) -> str:
        # Embedding the situated text is usually better for retrieval;
        # the un-prefixed text is what gets shown or cited.
        return f"{self.context_prefix}\n\n{self.text}"


def chunk_section(doc, section, parent_id: str | None, max_tokens: int,
                  count_tokens) -> Iterator[Chunk]:
    """Structure-first chunking: one chunk per section when it fits;
    otherwise split at block boundaries, never inside a block. Tables are
    never split from their header row."""
    breadcrumb = " > ".join(section.path) if section.path else doc.metadata.get("title", "")
    prefix = f"[{doc.metadata.get('doc_type', 'document')}] {breadcrumb}"

    section_text = "\n\n".join(b.text for b in section.blocks)
    if count_tokens(section_text) <= max_tokens and section.blocks:
        yield Chunk(
            chunk_id=f"{doc.doc_id}:{section.section_id}",
            doc_id=doc.doc_id, doc_version=doc.version,
            parent_chunk_id=parent_id, level="section",
            text=section_text, context_prefix=prefix,
            block_ids=[b.block_id for b in section.blocks],
        )
    else:
        buffer, buffer_ids, part = [], [], 0
        for block in section.blocks:
            candidate = "\n\n".join(buffer + [block.text])
            if buffer and count_tokens(candidate) > max_tokens:
                yield Chunk(
                    chunk_id=f"{doc.doc_id}:{section.section_id}:{part}",
                    doc_id=doc.doc_id, doc_version=doc.version,
                    parent_chunk_id=parent_id, level="block",
                    text="\n\n".join(buffer), context_prefix=prefix,
                    block_ids=list(buffer_ids),
                )
                buffer, buffer_ids, part = [], [], part + 1
            buffer.append(block.text)
            buffer_ids.append(block.block_id)
        if buffer:
            yield Chunk(
                chunk_id=f"{doc.doc_id}:{section.section_id}:{part}",
                doc_id=doc.doc_id, doc_version=doc.version,
                parent_chunk_id=parent_id, level="block",
                text="\n\n".join(buffer), context_prefix=prefix,
                block_ids=list(buffer_ids),
            )

    for child in section.children:
        yield from chunk_section(doc, child, f"{doc.doc_id}:{section.section_id}",
                                 max_tokens, count_tokens)

The point of showing the code is not the algorithm, which is unremarkable. It is that the chunk carries its lineage (block_ids), its position in the hierarchy (parent_chunk_id, level), and its context (context_prefix) as first-class fields. Those fields are what make everything downstream — progressive retrieval, citation, version propagation — possible.