Entry 39 · Monday
llm streaming guardrails: the user read it before the filter
Kitchen table
It is like a teacher checking your essay after you already read it aloud to the whole class.
Whiteboard
I am seeing a quiet gap in streamed chat answers. The model sends tokens to the screen as they arrive. The output guardrail runs on the finished answer. So the user reads every word before the check runs. When the check fails, the interface deletes the text. The user has already seen it, and may have copied it. The deletion also shows them exactly what the filter dislikes. The fix is a buffer between the model and the screen. Hold a short window of tokens and check it before release. Check again at sentence ends, where the meaning settles. The trade-off is delay before the first visible word. A larger window catches more, but the answer feels slower. Some checks need the whole answer, such as citation or policy review. Those answers should not stream at all. Measure how often text is shown and then withdrawn. Also measure the delay added before the first word. A withdrawn answer is a leak, not a save.
Boardroom
Many AI assistants show answers word by word so they feel fast. The safety check often runs only after the full answer is done. So a customer can read a harmful or wrong answer before it is caught. Removing it afterwards does not undo what they saw. Screenshots travel fast. Fixing this costs a little speed on every answer. That is a small price next to one public incident. Ask your teams one question. Does any customer ever see text before our checks have passed it?
Tags
All entries · 43 in all