Entry 1 · Friday
the pass rate went up and so did everything else
Kitchen table
If you ask a friend to fetch a book and they come back holding it, you still have no idea whether they walked straight to the shelf or wandered round the whole library twice.
Whiteboard
Somebody published traces from an agent harness this week, and the interesting part was not the failures. One revision pushed task success from 70 per cent to 81 per cent, three more tasks finished, while the mean number of model calls drifted from 3.8 to 4.9, tool calls from 2.8 to 3.9, the data dragged back from those tools roughly doubled, and the whole run took 56 per cent longer. None of that shows up on a pass/fail dashboard, because a correct answer is very good at hiding the route it took. The awkward bit is that the extra turns were not simply waste: a failed search retried, a truncated file read fetched again. Recovery and flailing look identical from the outside, and only a trace can tell them apart. Which leaves the question of what the green tick is actually certifying – that the agent was right, or that it got there eventually, at a price nobody wrote down.
Boardroom
No boardroom telling for this one yet. Here is the whiteboard one.
Somebody published traces from an agent harness this week, and the interesting part was not the failures. One revision pushed task success from 70 per cent to 81 per cent, three more tasks finished, while the mean number of model calls drifted from 3.8 to 4.9, tool calls from 2.8 to 3.9, the data dragged back from those tools roughly doubled, and the whole run took 56 per cent longer. None of that shows up on a pass/fail dashboard, because a correct answer is very good at hiding the route it took. The awkward bit is that the extra turns were not simply waste: a failed search retried, a truncated file read fetched again. Recovery and flailing look identical from the outside, and only a trace can tell them apart. Which leaves the question of what the green tick is actually certifying – that the agent was right, or that it got there eventually, at a price nobody wrote down.
Tags
All entries · 32 in all