llm eval statistical significance: the gain was inside the noise

A prompt change raised our eval score, and the team shipped it. A week later the gain had vanished on fresh questions. I am now wary of any single eval number. An eval set is a sample drawn from all the questions users ask. Every score therefore carries a standard error, and small sets carry large ones. Evan Miller's paper Adding Error Bars to Evals makes this case plainly. It recommends reporting error bars with every score. It also recommends paired comparisons, where both models answer the same questions. Pairing removes the noise from question difficulty. Questions that share one passage are not independent, so it clusters them. Sampling several answers per question shrinks the noise from the model itself. The trade-off is cost, since tighter bars need more questions or more runs. The failure mode is quiet. Teams chase gains that sit inside the noise and call them progress. Measure the paired difference and its interval before you ship. If the interval crosses zero, you have not learned anything yet.