Chunking Honestly

A RAG system can look fine for months and then quietly regress because a colleague "tidied up" the chunking. It is the stage no customer asks about, and the one that most often decides whether the system actually works.

This lesson is the one you want on the whiteboard when someone says "we can just split on paragraph."

Three strategies, each honest about a different thing

The fourth answer, and the honest production one, is almost always a mix. Structural for the heading and section boundaries, fixed-size inside sections that exceed a cap, semantic as a tiebreak when a section is long and uniform. "Which strategy" is the wrong framing; "what signal does this chunker respect" is better.

The quiet failure mode

Imagine the answer to "what is the refund window for enterprise annual contracts?" is one sentence that happens to land on a page boundary. You split on page. Chunk A ends "...and under section 7 the refund window for enterprise..." Chunk B begins "...annual contracts is 30 days, measured from the first charge..." Each half, on its own, scores poorly against the query — neither one contains both the question and the answer. Neither makes it into the top-5. Recall@5 drops without a single alarm.

This is why overlap exists. You pay a 10–20% increase in index size and retrieval latency, and you hedge against boundary-landing answers. It is a tax, not a fix: the right chunker gets the boundaries right most of the time, and overlap is for the rest.

Tables, code, lists: the content that breaks default chunkers

The thing fixed-size chunkers destroy is content whose schema lives apart from its rows. A table has a header row that gives meaning to every row below it. A code block has an import at the top that gives types to every line that uses them. A list has an introductory sentence that scopes every bullet.

The pattern that works in every case is the same: treat the schema as context and attach it to every chunk that depends on it.

chunker

1. WhyA pricing table in the customer's docs. The header row names the plans and the billing cadences — the information you need to interpret any number below it.

2. Output: | Plan        | Monthly | Annual |
   |-------------|---------|--------|
   | Starter     | $19     | $190   |
   | Growth      | $49     | $490   |
   | Enterprise  | Contact | Contact|

3. WhyA fixed-size chunker splits the markdown on the third row. The chunk that gets "Growth | $49 | $490" has no idea what $49 is the price of or what cadence it is on.

4. WhyThe row-level chunker emits one chunk per row, with the header duplicated into each: "Plan=Growth, Monthly=$49, Annual=$490". Each chunk is self-contained and findable by "what is the annual price of Growth".

5. WhyThis is not markup magic. It is deciding that the table's header is context for its rows and treating it that way in the chunker. The same pattern works on YAML, CSVs, configuration blocks, legal clauses with defined terms, and anywhere the structure assumes a reader who remembers the top.

Embed the headings

For prose documentation with a heading hierarchy, the single cheapest improvement is attaching the heading chain to every chunk from that location.

A chunk about billing frequency, taken from under Pricing > Enterprise > Annual billing, embeds not just its sentences but that path. Two things improve. The embedding is denser with locating signal, so the chunk is more retrievable for queries that mention enterprise or annual. The chunk is more answerable once retrieved, because the model sees the context it was lifted from and can say "under Enterprise annual billing..." rather than "the rate is 15%" out of nowhere.

This costs you a handful of tokens per chunk and buys several points of recall on the kinds of queries customers actually ask. It is the first thing to try before reaching for a bigger embedding model.

Versioning the chunks

Corpora change. Documents get edited, sections added, tables updated. A RAG system that re-embeds the whole corpus on every change wastes money and introduces latency for no reason; one that never re-embeds silently serves stale answers.

The pattern is chunk-level diffing. Assign every chunk a stable id derived from its structural location — document id plus heading path plus row index — rather than from its content. When a version ships, re-embed only the chunks whose text changed. Stable ids make this possible; content-hashed ids do not, because they change whenever the chunk does.

The payoff shows up in two places. The index update is proportional to the change, not the corpus. And the audit trail can show which version of which chunk was retrieved for a given answer — the citation deliverable that the grounding lesson leans on.

What to take away

Chunking is the quietest source of regression in a RAG system. Structural boundaries are the right default for documentation; fixed-size is the baseline that destroys tables and code; semantic is a tool for long uniform sections. Overlap is a tax you pay for boundary mistakes, not a replacement for a good chunker. Treat the schema of a table, a code block or a list as context attached to each row-chunk, and embed the heading chain into every chunk — the two cheapest moves with the biggest return. Give chunks stable ids so a corpus update can diff down to the chunks that actually changed.

Next: hybrid search and reranking, which is how you make the top-k the model sees actually be the top-k that answers the question.