llm upgrade testing: the new model passed the old tests

Kitchen table

A new puppy can pass every test you made for your old dog and still chew the shoes you never thought to hide.

Whiteboard

I am wary of model upgrades that pass the old eval suite. That suite was written against the old model's mistakes. Each case exists because something once broke. So the suite tests yesterday's failure modes, not tomorrow's. A new model fails in different places. It may refuse less and invent more. It may follow the format but drop a constraint buried mid-prompt. It may write longer answers that a downstream parser truncates. None of that shows up in cases built around old bugs. The prompts carry the same bias. They hold workarounds tuned for the old model's quirks. Some of those workarounds now confuse the new one. Before switching, I compare behaviour, not just scores. I run real traffic through both models in shadow mode. I sample the cases where their outputs disagree and read them by hand. I measure disagreement rate, length drift, refusal rate and parser failures. Then I turn the disagreements into new eval cases. The upgrade ships when the new failures are known, not when the old ones pass.

Boardroom

Switching to a newer AI model looks like a free upgrade. The test suite says the new model passes. But that suite was built to catch the old model's mistakes. It cannot see the new model's mistakes. Those surface later, in customer answers and broken downstream systems. Checking first costs little compared with one public incident. Run the new model quietly beside the old one on real work. Review the cases where they disagree before any switch. Do this on every model change, not only the major ones. So the question for the boardroom is simple. Who signs off that a model change is safe, and on what evidence?

Get the week by email