Shelf three · Language models & generative AI
Tokens & Embeddings
Kitchen table
The machine does not read words the way you do. First the words get chopped into pieces – 'unbelievable' might become 'un', 'believ', 'able' – and each piece becomes a long list of numbers. Here is the lovely part: pieces that mean similar things get numbers that are close together, like houses on the same street. 'King' and 'queen' live near each other. 'King' and 'kettle' do not. Meaning becomes a map, and the machine finds its way around by distance.
Whiteboard
Subword tokenisation sets your context budget and your bill – token counts, not word counts, are the unit of everything. Embeddings are the workhorse nobody demos: dense vectors where cosine distance approximates semantic similarity, giving you search, clustering, deduplication and recommendation almost for free. Chunking strategy and embedding-model choice quietly decide your retrieval quality, and mixing embeddings from different models is the classic silent failure. The maps do not share a coordinate system.
Boardroom
Two practical facts hide in this book. Tokens are the unit you are billed in. Every price on an AI invoice is per-token, so verbose prompts and long documents are literally expensive, and a cost review of prompts is a real savings line. Embeddings are what make 'search that understands meaning' possible over your own documents. The unassuming technology behind most successful enterprise AI so far, and considerably cheaper than the chatbots that get the headlines.
Related concepts