we kept pruning the memories that never got a turn

Kitchen table

It's a bit like tidying your room by binning anything you haven't used lately, which is exactly how you lose the torch you only ever need when the lights go out.

Whiteboard

A paper out this week put a number on something that should probably worry anyone running a memory store behind an agent: on two long-conversation benchmarks, 54 per cent and 67 per cent of the memories a task actually required were never retrieved at all, which means their usefulness was never measurable in the first place. The pruning logic ran anyway. It scores each memory by its effect on outcomes, and a memory that never surfaced looks identical to one that surfaced and did nothing, so the housekeeping was deciding what to forget on evidence that could not exist. The proposed fix is to reserve a few context slots for randomly sampled memories, which lifts the separation between needed and not-needed from 0.54 to 0.66 – honest, if not triumphant. The sting is the last finding: a well-measured memory scored 0.78 on the question it was measured against and predicted nothing about the next one. So what is a retention policy actually ranking?

Boardroom

No boardroom telling for this one yet. Here is the whiteboard one.

A paper out this week put a number on something that should probably worry anyone running a memory store behind an agent: on two long-conversation benchmarks, 54 per cent and 67 per cent of the memories a task actually required were never retrieved at all, which means their usefulness was never measurable in the first place. The pruning logic ran anyway. It scores each memory by its effect on outcomes, and a memory that never surfaced looks identical to one that surfaced and did nothing, so the housekeeping was deciding what to forget on evidence that could not exist. The proposed fix is to reserve a few context slots for randomly sampled memories, which lifts the separation between needed and not-needed from 0.54 to 0.66 – honest, if not triumphant. The sting is the last finding: a well-measured memory scored 0.78 on the question it was measured against and predicted nothing about the next one. So what is a retention policy actually ranking?

Get the week by email