Shelf four · Agents, tools & reinforcement
RLHF & Preference
Kitchen table
How do you teach a machine to be helpful when 'helpful' is not a thing you can score with a ruler? You show people two answers to the same question and ask 'which is better?' Thousands and thousands of times. The machine studies all those choices and slowly learns the taste of a good answer: polite, clear, actually answering the question. It is how the raw word-guesser from the shelf below got its manners. Somebody's idea of good manners, anyway.
Whiteboard
Train a reward model on pairwise human preferences, then optimise the policy against it – PPO classically, the DPO family when you would rather skip the reward model, constitutional and rubric-based variants when human labels are the bottleneck. Alignment is a data problem wearing a maths hat: whose preferences, which edge cases, how consistent the raters. Over-optimising the reward model produces sycophancy and safe blandness. The model learns to please the grader, which is not the same as being right.
Boardroom
This is the finishing step that turns a raw model into something safe enough to put in front of customers, and it is where a model's values and refusals actually come from, not physics, but choices made by whoever ran the preference process. 'Whose preferences went in?' is a fair and revealing vendor question. The residual risk to manage is agreeableness: a system tuned to please raters can drift toward telling users what they want to hear, which in your context may be a compliance issue, not a charming quirk.
Related concepts