Entry 22 · Friday
the speed-up went backwards once we asked for json
Kitchen table
It is like having a friend shout out your next word while you write, except nobody told him you are filling in a form, so almost everything he shouts has to be thrown away.
Whiteboard
Someone filed a vLLM issue this week with the kind of number that makes you put the coffee down: with speculative decoding switched on and a JSON schema enforced, the draft model's guesses were thrown out by the grammar checker 84.9% of the time, and the whole thing ran marginally slower than not speculating at all. Both features work. They simply have no idea the other exists. The drafter proposes the most likely next token, the verifier only accepts tokens the schema permits, and nobody ever showed the drafter the schema, so it keeps offering plausible prose into a slot that will only take a closing brace. Masking the drafter's logits with the same grammar before it guesses made it faster than no speculation and, oddly, produced more valid JSON than before. Each optimisation was benchmarked on its own. Which pairs in your stack have never been measured together?
Boardroom
No boardroom telling for this one yet. Here is the whiteboard one.
Someone filed a vLLM issue this week with the kind of number that makes you put the coffee down: with speculative decoding switched on and a JSON schema enforced, the draft model's guesses were thrown out by the grammar checker 84.9% of the time, and the whole thing ran marginally slower than not speculating at all. Both features work. They simply have no idea the other exists. The drafter proposes the most likely next token, the verifier only accepts tokens the schema permits, and nobody ever showed the drafter the schema, so it keeps offering plausible prose into a slot that will only take a closing brace. Masking the drafter's logits with the same grammar before it guesses made it faster than no speculation and, oddly, produced more valid JSON than before. Each optimisation was benchmarked on its own. Which pairs in your stack have never been measured together?
Tags
All entries · 32 in all