Shelf one · How a machine learns
Evaluation & Metrics
Kitchen table
Before you trust the machine, you give it a test, questions you already know the answers to, and count how many it gets right. But you have to be careful about which questions. If you quiz it on the same examples it studied, it just remembers the answers, like peeking at the answer sheet. So you always keep back some questions it has never seen. And one number is never enough: a doctor-machine that says 'nobody is sick' gets 99% right if almost nobody is sick, and is still completely useless.
Whiteboard
Hold-out sets and cross-validation for honesty. Precision against recall because they pull in opposite directions and the right trade depends on the cost of each mistake. Calibration because a confident wrong answer is worse than a hesitant one. Class imbalance turns accuracy into a vanity metric, and distribution shift turns last quarter's beautiful eval into fiction. The discipline is keeping a test set the training process has never touched, and treating any metric the team optimises directly as already compromised.
Boardroom
Never accept a single number. '92% accurate' means nothing until you know what guessing would have scored, what the errors cost, and who bears them – a false alarm and a missed case are rarely the same price. Ask for the metric, the baseline, and the confusion between the two kinds of wrong. And insist evaluation continues after launch: a model is measured on the day it shipped and then the world starts moving. The quiet degradation eighteen months in is the failure mode nobody budgets for.
Related concepts