theNotebook

Entry 23 · Saturday

it found the needle every time and still lost count

24 October 2026

Shelf three · Language models & generative AI

Kitchen table

A friend with a brilliant memory can tell you what colour the first cup was, and still lose track of which cup the coin is under after you shuffle them forty times.

Whiteboard

A benchmark out this week separates three things we usually lump together as long-context ability: recalling a fact from far back, keeping track of a changing state, and learning a pattern from examples spread across the window. Recall held steady from 8K to 128K tokens, which is the number the needle-in-a-haystack tests have been reporting for a year and the one that goes in the launch post. State tracking and pattern learning fell by up to 80 per cent over the same stretch, with frontier models landing near or below naive baselines on tasks like following a coin under shuffled cups. The thinking variants often did worse than their base models, burning the generation budget without converging. So the window is not a bigger desk. It is a longer corridor, and what reaches the far end is the fact, not the running total. Which of your prompts ask the model to remember, and which quietly ask it to keep score?

Boardroom

No boardroom telling for this one yet. Here is the whiteboard one.

A benchmark out this week separates three things we usually lump together as long-context ability: recalling a fact from far back, keeping track of a changing state, and learning a pattern from examples spread across the window. Recall held steady from 8K to 128K tokens, which is the number the needle-in-a-haystack tests have been reporting for a year and the one that goes in the launch post. State tracking and pattern learning fell by up to 80 per cent over the same stretch, with frontier models landing near or below naive baselines on tasks like following a coin under shuffled cups. The thinking variants often did worse than their base models, burning the generation budget without converging. So the window is not a bigger desk. It is a longer corridor, and what reaches the far end is the fact, not the running total. Which of your prompts ask the model to remember, and which quietly ask it to keep score?

Tags

  • long context
  • context engineering
  • state tracking
  • evals
  • benchmarks

All entries · 32 in all

Search the notebook