nobody asks the whole question on the first turn

Kitchen table

It's like asking someone a question in little bits spread over five minutes. They either go and look up the wrong thing, or they find exactly the right book and still can't work out what you were asking.

Whiteboard

A study published this week took multi-hop questions from the usual benchmarks and broke each one into the sort of conversation people actually have – a vague opening, a follow-up, then the real question somewhere near the end – and ran a million and a half of them past ten assistants and eight retrieval systems. Accuracy fell by up to 21 per cent relative, and unreliability rose 47 per cent: less accurate and less steady at the same time. Two failure modes, and they want different fixes. Sometimes the rewritten query drifts and the right document is never fetched. Sometimes it is sitting right there and the model cannot assemble an answer out of evidence spread across three turns. The awkward part is that our retrieval evals hand the system the whole question at once, fully specified and politely phrased, which is the one thing no user has ever done. So what is a recall number worth when the query it was measured on hasn't been asked yet?

Boardroom

No boardroom telling for this one yet. Here is the whiteboard one.

A study published this week took multi-hop questions from the usual benchmarks and broke each one into the sort of conversation people actually have – a vague opening, a follow-up, then the real question somewhere near the end – and ran a million and a half of them past ten assistants and eight retrieval systems. Accuracy fell by up to 21 per cent relative, and unreliability rose 47 per cent: less accurate and less steady at the same time. Two failure modes, and they want different fixes. Sometimes the rewritten query drifts and the right document is never fetched. Sometimes it is sitting right there and the model cannot assemble an answer out of evidence spread across three turns. The awkward part is that our retrieval evals hand the system the whole question at once, fully specified and politely phrased, which is the one thing no user has ever done. So what is a recall number worth when the query it was measured on hasn't been asked yet?

Get the week by email