theNotebook

Entry 40 · Tuesday

llm tokenization trailing space: one blank changed the answer

10 November 2026 · By

A toy train on a wooden track at dusk, its front wheel about to reach a small glowing gap between two track pieces.
Trailing space before and afterbeforeprompt + spacerare boundarydrifted answerafterspace strippedfamiliar tokensteady answer

Kitchen table

A robot finishing your sentence trips on one extra space, like a toy train wheel catching on a tiny gap in the track.

Whiteboard

I am seeing a quiet bug in prompt templates. The template ends with a trailing space before the answer slot. Most tokenizers attach the leading space to the next word. So the model usually sees " Paris" as one common token. After a bare space, it must start from a boundary it rarely saw. Output quality drops in ways that look like model weakness. The guidance library calls the fix token healing. It backs up one token and constrains the next token to match the removed text. The cheaper fix is to strip trailing whitespace before every call. The trade-off is real, because some formats need the space. The failure mode is silent. Nothing errors, and eval scores drift down a little. I measure it directly. I diff the token ids at the end of the prompt across template versions. I also run the same eval with and without the trailing space. A gap between those runs points at the template, not the model.

Boardroom

AI answers can get worse because of one invisible character in a template. Nothing breaks and no alarm fires. Quality slips a little on every request that uses that template. Teams often blame the model and pay for a larger one. That is money spent on the wrong cause. The fix costs almost nothing once someone looks. It needs a habit, not a budget. Every prompt template change should pass a token check and a short eval before release. The decision: do our prompt templates pass the same release checks as code?

Tags

  • tokenization
  • prompting
  • prompt templates
  • evals
  • debugging

All entries · 43 in all

Search the notebook