the cache turned out to be slower than thinking about it again

Kitchen table

It's a bit like running all the way to the shop for a sandwich you already made, when making a fresh one at home would have been quicker.

Whiteboard

Someone put a shared cache in front of a long-context serving stack, the kind meant to save you from re-reading a prompt you have already read, and then had the nerve to measure it. A cache hit took 0.88 seconds. Recomputing the same prompt from scratch took 0.31. The median latency never moved. The tail went from about four seconds to seventeen, and in the worst stretch requests queued for eighty-three seconds while the GPUs sat seven per cent busy. The arithmetic is unforgiving: there is a fixed cost per hit, roughly a third of a second, so below about seventeen thousand tokens no amount of bandwidth makes remembering cheaper than thinking again. Caching feels like thrift, which is exactly why nobody audits it. But a cache is only ever a bet that fetching beats redoing, and that bet has a break-even you can work out on paper first. Do you know which side of it your traffic sits on?

Boardroom

No boardroom telling for this one yet. Here is the whiteboard one.

Someone put a shared cache in front of a long-context serving stack, the kind meant to save you from re-reading a prompt you have already read, and then had the nerve to measure it. A cache hit took 0.88 seconds. Recomputing the same prompt from scratch took 0.31. The median latency never moved. The tail went from about four seconds to seventeen, and in the worst stretch requests queued for eighty-three seconds while the GPUs sat seven per cent busy. The arithmetic is unforgiving: there is a fixed cost per hit, roughly a third of a second, so below about seventeen thousand tokens no amount of bandwidth makes remembering cheaper than thinking again. Caching feels like thrift, which is exactly why nobody audits it. But a cache is only ever a bet that fetching beats redoing, and that bet has a break-even you can work out on paper first. Do you know which side of it your traffic sits on?

Get the week by email