the steadiest judge was wrong the same way every time

A paper out this week audited six frontier models doing the job most of us quietly hand them, scoring other models' answers, and the numbers are worth reading twice. At temperature zero the verdicts still moved between identical reruns. Swap which answer is shown first and most of the hard cases flip. And the judge that never wavered turned out to be repeating the same wrong call: it agreed with ground truth about half the time, which is a coin flip with better manners. The authors suggest counting only the verdicts that are reproducible, order-invariant and correct all at once, and found that holistic rubric scoring buys more trustworthiness than any prompt tweak. Which leaves the awkward part. Nearly every pipeline now has a judge somewhere in it, and almost nobody evaluates the evaluator. What would it take to put your judge on the dashboard, right beside the thing it is grading?