Entry 36 · Friday

rag chunking strategy: the header lived in another chunk

6 November 2026 · By FNU Ratnavarma

A paper timetable torn in half on a wooden table, the lower half with its columns drifting away from the headed top half under a brass lamp.

Kitchen table

If you tear a train timetable in half, the bottom half still has times but no longer says which station they are for.

Whiteboard

I keep seeing one RAG failure that no retriever fix touches. A tariff document holds a rate table. The column headers sit in one chunk. The rows land in the next chunk. The retriever finds the rows because they match the query well. The model then reads numbers with no labels above them. It guesses which column is which, and it guesses with confidence. Fixed-size chunking caused this, not the embedding model. Overlap helps a little but rarely spans a whole header block. Structure-aware chunking keeps each table together with its header. Parent-document retrieval matches the small piece, then passes the larger section. The trade-off is more context cost and blurrier embeddings for big chunks. I measure one thing first. For each gold answer, does the evidence cross a chunk boundary? I score those questions as their own slice. They often hide the worst failures behind a healthy average.

Boardroom

Our AI assistants answer from documents we cut into small pieces. The cutting is a cheap, mechanical step. It often separates a table from its headings. The assistant then quotes a real number under the wrong label. In a rate schedule or a contract, that becomes a billing or compliance error. It looks correct, so reviewers rarely catch it. The fix costs little next to the rest of the retrieval budget. It needs document-aware splitting and a test aimed at split evidence. The question for this room is simple. Do we test our assistants on answers that cross a page or table break?