The shelf that decides whether any of the above survives contact with a real business. Titles are on the spines; the notes underneath are the three tellings, in short — for a ten-year-old, for the architect, and for the boardroom.
6 concepts on this shelf · Intermediate → Advanced · about 8 min read
Kitchen table — You keep a secret stack of questions you already know the answers to, and you make the machine sit the same test every week. Not because you enjoy marking — because machines can get worse without telling you. An update meant to make it politer might quietly make it worse at maths. The test stack is how you notice. And when the machine invents an answer that sounds right but is not — that has a name, hallucination, and it is why you never mark its homework by vibes.
Whiteboard — Task-level suites, graders (exact-match where you can, LLM-as-judge where you must — with the judge itself evaluated), golden sets guarded from contamination, and regression runs wired into CI so a prompt change cannot silently ship a capability loss. Writing down what 'correct' means is the uncomfortable part and the entire point: most teams discover they never agreed. Evals are to AI what tests are to code, with one difference — the system underneath drifts even when you change nothing.
Boardroom — One question separates serious AI vendors from the rest: how do you measure that it is right, and can I see the test set? Vibes are not a control, and a demo is a performance, not a measurement. Internally, insist every AI feature ships with its eval suite and that the scores are reported over time like any other operational metric — because the failure mode is not day-one wrongness, it is quiet degradation while everyone assumes someone else is watching. The eval budget is the insurance premium.
evals, evaluation, hallucination, LLM as judge, golden set, regression testing, quality
Kitchen table — Rules about what the machine may do on its own and when it must come and ask a grown-up. It can draft the letter — but not send it. It can suggest the refund — but a person clicks the button. Like a learner driver: allowed on quiet roads with an instructor who has their own brake pedal, not on the motorway on day one. The point is not distrust. The point is that trust is earned a road at a time.
Whiteboard — Least privilege for something that improvises: scoped credentials, allow-lists over deny-lists, approval gates on irreversible actions, sandboxes for anything exploratory, input and output filters, and a blast radius you have actually calculated rather than gestured at. Layered, because any single check will be argued past eventually — by the model or by a user. The design question that organises everything: if this goes as wrong as it can go, what is the maximum damage? Engineer that number down first, capability second.
Boardroom — This is where regulators, auditors and your own risk committee will look first, and the EU AI Act era has made 'we had controls designed in' versus 'we were going to add them' the difference between a finding and a fine. Retrofitting is far more expensive than designing in — true of security for twenty years, truer here. The reframe worth carrying into strategy: strong guardrails are not the brake on AI adoption, they are what makes bolder adoption defensible. The teams that can prove control get to move faster.
guardrails, responsible AI, AI governance, permissions, least privilege, human in the loop, compliance, audit
Kitchen table — Building the machine is one job. Keeping it working every single day is a different, longer job. Machines that learned from the world go stale as the world changes — like a map from five years ago that still looks crisp but is missing the new bridge. So you watch it, test it, keep the old version ready in case the new one misbehaves, and check its answers now and then against reality. Less like finishing a Lego set; more like owning a puppy.
Whiteboard — Versioning for models, data and prompts together — a prompt change is a deploy; canary releases and rollback because model behaviour cannot be fully known before contact with real traffic; drift monitoring on inputs and outputs; tracing so any answer can be reconstructed after the fact. A model in a notebook is a hypothesis, not a system. The LLM twist: your 'model' is often someone else's endpoint that changes underneath you, so pin versions where you can and eval continuously where you cannot.
Boardroom — This is the gap between a successful pilot and a dependable service, and it is where most AI programmes actually stall — the industry's pilot graveyard is not full of bad models, it is full of models nobody could operate. Running AI costs real money after the launch party: monitoring, retraining, incident response, the second shift nobody budgeted. When a pilot is approved, approve its operating cost at the same time, or you have approved a demo. 'Who owns this model at 2 a.m.?' remains the sharpest question in the room.
MLOps, LLMOps, deployment, model monitoring, drift, observability, CI/CD, rollback
Kitchen table — Every letter, bill, text and email a company sends — there are millions of them, and behind each one is a system deciding who gets which message, filling in the right name and numbers, and keeping a copy. Boring? A bit. But a confusing bill makes thousands of people phone the help desk in one morning, and a clear one keeps them happy at breakfast. Writing so people actually understand is a superpower wearing a very sensible cardigan.
Whiteboard — Template and content management, variable data, composition engines, channel orchestration across print, email, SMS and app, plus archive and retrieval for the regulator. Now generation and personalisation are layering in — under strict review, because these documents are contractual. The architecture question is where generated content enters: drafting new templates with human sign-off is low-risk and immediately useful; per-customer free generation at send time is a different risk class needing validation, sampling and audit. Sequence them in that order.
Boardroom — Millions of regulated documents a year, and small improvements in clarity move call volume, complaint rates and compliance findings all at once — which makes this one of the most measurable AI opportunities in the building, with a baseline already sitting in your call-centre stats. It is also one of the safer entry points: AI drafting templates that humans approve captures most of the value at a fraction of the exposure of generating live customer text. Unfashionable, quantifiable, and usually underestimated.
customer communications management, CCM, correspondence, document composition, personalisation, omnichannel, notifications
Kitchen table — Imagine a filing cabinet the size of a building — every contract, form and report the company ever made. For decades you could only find things if you knew exactly which drawer to open, so mostly nobody looked. The new trick is a filing cabinet that has read its own files. Ask 'which contracts end this year?' and it does not hand you a drawer, it answers — and points at the exact pages, so you can check it is not making things up.
Whiteboard — Capture, classification, extraction and lifecycle over enterprise content — the classic ECM stack — with document intelligence turning scans into structured data and retrieval turning archives into something answerable. The order of operations matters more than the model: metadata, deduplication, permissions and retention first, because AI over an unmanaged repository answers from the obsolete draft with total confidence, and retrieval that ignores access control is a breach with a chat interface. The truth-versus-noise ratio of the corpus is the product.
Boardroom — Decades of documents are usually the most valuable and least used asset in the building — evidence of every deal, decision and obligation, sitting inert because finding anything cost more than it returned. AI finally changes that arithmetic, which quietly turns document management from a cost centre into the foundation the other AI initiatives stand on: retrieval, communications, agents all drink from this well. The governance dividend comes first, though — you cannot safely make an archive answerable until you know what is in it and who may ask.
content management, ECM, document intelligence, document extraction, OCR, records, knowledge management
Kitchen table — Thinking harder takes longer and costs more — true for people, true for machines. The big clever model is like phoning a professor: brilliant, slow, expensive. The small quick one is like asking your friend: instant, cheap, usually right. The skill is knowing which questions deserve the professor. Most do not. And if you will ask the same question a hundred times, write the answer down the first time — that is caching, the oldest homework trick in the world.
Whiteboard — Route easy traffic to small models and hard cases up the tier; cache aggressively (semantic caching catches paraphrases); batch what is not latency-sensitive; stream tokens so perceived latency beats real latency; cut context flab, because you pay per token for every word of that bloated prompt. Measure cost per resolved task, not per token — the cheap model that fails and escalates twice is the expensive model. Unit economics are a design input from day one, not a finance review after launch.
Boardroom — Know two numbers: cost per completed task, and the cost of the human path it replaces. If nobody can produce them, the pilot is not ready for a budget conversation — and this, not model quality, is where most AI initiatives die at their first renewal. Insist the unit economics improve with scale: caching and routing should make each additional task cheaper, and if the curve bends the other way you are subsidising a demo. The margin between those two numbers is the entire business case; watch it like revenue.
AI cost, token cost, latency, caching, model routing, unit economics, ROI, TCO