the retriever was fine and the pdf was not

Kitchen table

If you copy a recipe but the numbers land in the wrong columns, the cake fails even when your reading is perfect.

Whiteboard

Most retrieval failures I chase are not retrieval failures. They begin in the parser. A table in a scanned PDF becomes a flat run of numbers. Column headers detach from the cells below them. The chunk still embeds cleanly and still scores well. The answer is then wrong about which year a figure belongs to. OmniDocBench measures this layer across tables, formulas and reading order. I treat the parser as a model with its own error profile. It is not plumbing. The failure is silent, because a mangled table looks like prose to a chunker. So I measure parse fidelity before I measure recall. I sample pages by layout type rather than at random. Rotated scans and merged cells are where the loss concentrates. I also keep the page image beside the chunk. A reviewer can then check the claim against the original. Fixing the parser moved more answers than any reranker I tried.

Boardroom

We fund the models well and the document reading poorly. At large US utilities the sources are scanned permits and tariff sheets. Vendor tables arrive as page images rather than as data. If the parser drops a column header, the model still answers. It answers with confidence and leaves no trace of the error. That is a regulatory exposure, not a software defect. Better parsing costs less than a better model does. The accuracy gain is usually larger as well. It also makes each answer auditable against the original page. So the decision in front of this room is narrow. Do we fund document ingestion as a capability, with one owner and one target?

Get the week by email