Entry 12 · Tuesday
the answer was correct and the procedure never ran
Kitchen table
If you ask someone to bake a cake by following the recipe and they hand you a cake without ever opening the recipe, the cake might be lovely, but you still have no idea what they did.
Whiteboard
A paper this week does something quietly unfashionable: instead of putting the standard operating procedure in the system prompt and hoping, it hands the agent one step at a time over MCP, each step returning a structured record of what happened. The finding worth sitting with isn't about accuracy. Under prompt-based delivery, on one of the thirteen domains tested, between 31 and 49 per cent of the correct answers were produced without the procedure running at all – right output, no process, the frontier executor included. Serving the steps instead pushed adherence from 76-95 per cent to 95-99, and ungrounded answers from 2.1-4.5 per cent down to about 0.3, across 15,475 trials. The small model picked up 6.5 points of grounded accuracy. The capable ones traded a sliver of raw accuracy for a run you can actually read back. So which are you shipping. The answer, or the run that produced it?
Boardroom
No boardroom telling for this one yet. Here is the whiteboard one.
A paper this week does something quietly unfashionable: instead of putting the standard operating procedure in the system prompt and hoping, it hands the agent one step at a time over MCP, each step returning a structured record of what happened. The finding worth sitting with isn't about accuracy. Under prompt-based delivery, on one of the thirteen domains tested, between 31 and 49 per cent of the correct answers were produced without the procedure running at all – right output, no process, the frontier executor included. Serving the steps instead pushed adherence from 76-95 per cent to 95-99, and ungrounded answers from 2.1-4.5 per cent down to about 0.3, across 15,475 trials. The small model picked up 6.5 points of grounded accuracy. The capable ones traded a sliver of raw accuracy for a run you can actually read back. So which are you shipping. The answer, or the run that produced it?
Tags
All entries · 32 in all