the questions looked alike and the answers did not

Kitchen table

If a helper reuses yesterday's answer because your question sounded similar, you can get directions to the wrong house.

Whiteboard

I keep finding semantic caches tuned against the wrong metric. The mechanism is plain enough and the economics are real. You embed the incoming query and find the nearest cached one. Above a chosen similarity threshold you reuse the stored answer. Hit rate rises, cost falls, and the dashboard reads healthy. The failure sits in pairs that embed close but deserve different answers. A negation, a different account, a date, a stale policy version. Error A and error B look almost identical to an embedding model. The MeanCache paper names this the false hit problem. It fits the threshold against labelled pairs rather than guessing a number. Hit rate alone cannot say whether two requests deserved one answer. So I measure false hit rate on a labelled pair set first. I also log the similarity score behind every served hit. Hits clustered just above the line mean the cache is borrowing from correctness.

Boardroom

Caching similar questions is one of the cheapest ways to cut model spend. It also quietly creates a new class of wrong answer. A customer asks about their own account and receives a reply built for someone else. Nothing fails, nothing alerts, and the saving still shows in the monthly bill. The exposure is largest where answers depend on identity, entitlement, pricing or policy dates. Hit rate and cost savings cannot measure that risk. The decision is narrow. Ask who owns a measured false hit rate before the next cost target is signed.

Get the week by email