the longer list looked better because guessing got easier

Kitchen table

A big bundle of keys is more likely to hold the right one, and a lot harder to pick from.

Whiteboard

I have been reading about tool retrieval for agents this week. Most systems show the model a fixed number of candidate tools. We tuned that number by coverage, meaning how often the right tool appeared. Coverage rises with depth, so a longer list always looked safer. That measure quietly flatters the retriever that produced it. A wider list makes inclusion easier by luck. It also makes the choice harder for the model. The paper How Many Tools Should an LLM Agent See? A Chance-Corrected Answer puts a baseline under this. It scores success at a given depth against what random selection would reach there. With that baseline in place, extra depth stops being free. Coverage grades the retriever alone, and the agent is doing another job. The failure I now watch for is a confident wrong call on an easy request. So measure selection accuracy at each depth, not coverage alone. Measure it again on requests where the right tool ranks low.

Boardroom

Our assistants are gaining tools faster than we can test them. Each new tool is sold to us as more capability. It also makes every other tool harder to choose correctly. Our reporting hides this, because the usual measure rewards a longer menu. Capability rises on paper while wrong actions rise in production. A wrong action in a regulated business is not a speed problem. It is rework, and it is an audit finding, and both cost money. The decision in front of us is narrow. Do we fund a measure of correct tool choice before the next ten integrations?

Get the week by email