Shelf five · Putting it to work
Cost, Latency & ROI
Kitchen table
Thinking harder takes longer and costs more – true for people, true for machines. The big clever model is like phoning a professor: brilliant, slow, expensive. The small quick one is like asking your friend: instant, cheap, usually right. The skill is knowing which questions deserve the professor. Most do not. And if you will ask the same question a hundred times, write the answer down the first time. That is caching, the oldest homework trick in the world.
Whiteboard
Route easy traffic to small models and hard cases up the tier. Cache aggressively (semantic caching catches paraphrases). Batch what is not latency-sensitive. Stream tokens so perceived latency beats real latency. Cut context flab, because you pay per token for every word of that bloated prompt. Measure cost per resolved task, not per token. The cheap model that fails and escalates twice is the expensive model. Unit economics are a design input from day one, not a finance review after launch.
Boardroom
Know two numbers: cost per completed task, and the cost of the human path it replaces. If nobody can produce them, the pilot is not ready for a budget conversation, and this, not model quality, is where most AI initiatives die at their first renewal. Insist the unit economics improve with scale: caching and routing should make each additional task cheaper, and if the curve bends the other way you are subsidising a demo. The margin between those two numbers is the entire business case. Watch it like revenue.
Related concepts