theNotebook

Entry 43 · Friday

ai agent memory: the old fact kept winning

13 November 2026 · By

Stale agent memory and the fixFact savedFact changesSearch by matchOld fact winsWrite by keyOld one retiredCurrent fact

Kitchen table

The robot remembered you liked red, but it forgot to cross that out when you told it you like blue now.

Whiteboard

I am seeing agent memory fail in a quiet way. An agent saves a fact about a customer. Months later the fact changes, and the agent saves the new one. Both versions now sit in the same store. Retrieval ranks them by similarity to the question. The old fact can score higher because it was written in fuller words. So the agent answers from the past with full confidence. Nothing errors and nothing looks wrong in the trace. The fix is to treat memory as state, not as a pile of notes. Give each fact a subject, a key and a time. A new write for the same key should retire the old one. Keep the retired fact for audit, but out of retrieval. The trade-off is extra work at write time. Every write needs a step that spots conflicts, and that step can be wrong too. Measure how often a retrieved fact has a newer version in the store. Then seed tests with a changed fact and check which version the agent uses.

Boardroom

Our AI assistants now remember customers across conversations. That memory can go stale without anyone noticing. A customer changes an address or a preference. The assistant keeps using the old one. Each wrong answer sounds confident and polite. The cost shows up as complaints, rework and compliance findings. The fix is not a better model. It is a rule for how memory gets updated and retired. That rule costs engineering time now and protects trust later. The question for the board is simple. Who owns the correctness of what our AI remembers about customers?

Tags

  • agent memory
  • agents
  • staleness
  • retrieval
  • evals

All entries · 43 in all

Search the notebook