prompt caching: a timestamp up top broke every hit

Kitchen table

It is like a librarian who keeps your books ready, until you swap the top book and she has to fetch them all again.

Whiteboard

Prompt caching reuses work the model already did on a repeated prefix. The match is exact, token by token, from the very first token. I have seen teams start the system prompt with the current time. It felt harmless and it kept answers aware of the date. It also changed the prefix on every single request. So every call missed the cache and paid full price. A request ID, a user name or a shuffled tool list does the same. Even reordered keys in tool schemas can break the match. The fix is layout, not tuning. Put stable content first: instructions, tool definitions, reference documents. Put anything that changes last, near the user turn. Then measure the cache hit rate per route, not the bill alone. Track time to first token beside it, because a miss shows up there first. Treat a falling hit rate as a regression and alert on it. The trade-off is small. The model still sees the date, just later in the prompt.

Boardroom

Many AI products process the same long instructions on every request. Providers charge much less for repeated text that arrives in exactly the same order. Small design choices can quietly cancel that discount. A date stamp or a user name at the top is enough. Spend rises and responses slow down, yet nothing looks broken. This is a design habit, not a vendor problem. It costs almost nothing to fix once someone owns it. So ask your teams one question. Who is alerted when the cache hit rate drops?