the forgetting was in the words your data never used

A paper out this week went looking for where fine-tuning actually does its damage, and the answer is almost comic: the forgetting concentrates in the output embeddings of tokens the new corpus simply never mentions. Not the words you taught it badly. The words you didn't say. The mechanism is plainer than it sounds. Every step pushes an absent token down with a one-sided gradient, and Adam's second-moment normalisation, which exists to scale small gradients up, dutifully inflates that steady nudge into a full-sized update. The words go quiet, then drift. Raising Adam's epsilon on the output projection alone, one line, removed 39 to 68 per cent of the forgetting across eight settings, and pulled a released-head LoRA back from a twenty-three-fold surge. Editing the drifted rows afterwards recovered under five per cent. So which parts of the vocabulary has your training data gone quiet on, and when would you find out?