Entry 14 · Thursday
the copy came out steadier than the thing it copied
Kitchen table
If you copy out a messy notebook in your neatest handwriting, your copy can end up tidier than the original, because you smoothed over the wobbly bits without meaning to.
Whiteboard
Somebody distilled a working agent into a small model this week and walked into an awkward measurement on the way. The setup is ordinary: develop the behaviour in prompt space on a frontier model, let it run against live traffic, then fine-tune an 8B student on the 11,600 decisions the system actually made, rather than on synthetic imitations of them. The student agreed with its teacher 91 per cent of the time, at roughly 2.5 times lower median latency and a fraction of the cost. The number worth sitting with is the other one. The teacher, re-run on its own recorded prompts, agreed with itself only 72.7 per cent of the time, and where the two differed a blinded human scheduler sided with the student 48 to 29. So the student did not merely copy. It quietly averaged out the teacher's off days. Which raises the question of what your gold labels are a recording of.
Boardroom
No boardroom telling for this one yet. Here is the whiteboard one.
Somebody distilled a working agent into a small model this week and walked into an awkward measurement on the way. The setup is ordinary: develop the behaviour in prompt space on a frontier model, let it run against live traffic, then fine-tune an 8B student on the 11,600 decisions the system actually made, rather than on synthetic imitations of them. The student agreed with its teacher 91 per cent of the time, at roughly 2.5 times lower median latency and a fraction of the cost. The number worth sitting with is the other one. The teacher, re-run on its own recorded prompts, agreed with itself only 72.7 per cent of the time, and where the two differed a blinded human scheduler sided with the student 48 to 29. So the student did not merely copy. It quietly averaged out the teacher's off days. Which raises the question of what your gold labels are a recording of.
Tags
All entries · 32 in all