Shelf one · How a machine learns
Gradient Descent
Kitchen table
You are standing on a huge hill in thick fog, and you want to reach the valley. You cannot see the bottom, you cannot see anything, but you can feel the slope under your feet. So you take one small step downhill. Then you feel again, and step again. Do that a few million times and you end up somewhere low, even though you never once saw where you were going. That is how a machine learns: it feels which way 'less wrong' is, and takes a tiny step that way, over and over.
Whiteboard
Differentiate the loss with respect to every parameter, then step against the gradient. Everything else is commentary: batch sizes trade noise for speed, momentum smooths the ride, Adam rescales each dimension's steps, and the learning-rate schedule is where theory quietly gives way to folklore. The fog matters. The loss surface is high-dimensional and non-convex, and the honest answer for why we still find good minima is that overparameterised networks make most valleys good enough. Nobody fully knows. It works anyway.
Boardroom
This is literally what 'training' means: millions of small corrections, each one nudging the system slightly less wrong. It is why training takes weeks and megawatts, why progress shows up as a curve that flattens, and why 'just train it a bit more' eventually stops buying anything. When a team asks for more compute, they are asking for more steps down this hill. The strategic question is always whether the remaining descent is worth the electricity, and increasingly, someone has already walked most of the way down for you.
Related concepts