Entry 35 · Thursday

llm fallback models: the backup answered in another format

5 November 2026 · By FNU Ratnavarma

A stand-in cook plates dishes that differ from the reference plate by the pass.

Kitchen table

It is like a backup cook who knows a different recipe, so dinner still arrives on time but tastes wrong.

Whiteboard

I am wary of model fallback chains that nobody tests. The idea is sound. When the primary model times out, a second model takes the request. Uptime improves on the dashboard, and the trouble starts downstream. The backup model reads the same prompt differently. It wraps JSON in markdown fences or drops an optional field. It refuses where the primary would answer. The parser fails, or it quietly accepts a changed record. So the outage moves from the API into the data. It also arrives at the worst time, during a provider incident. Few teams run their eval suite against the fallback path. I treat each fallback as its own release. It gets its own prompt, its own schema check and its own tests. I measure schema pass rate, refusal rate, latency and task accuracy. I also force a failover in staging every week. A fallback I have never exercised is a guess, not a plan.

Boardroom

Fallback models are sold as insurance against outages. They often are not. The backup keeps the lights on but can change what the system says. Records get written in the wrong shape. Customers get refusals the main model would never give. These failures land during a vendor incident, when attention is elsewhere. The fix is cheap next to the risk. Test every backup path like a product release, and rehearse failover on a schedule. The question for the board is simple. When our main AI provider goes down, has anyone proven the backup gives correct answers?