What actually breaks when you switch LLMs

Changing the model behind an AI feature looks like the smallest change you will make all week. One string moves from gpt-x to gemini-y. The new model is cheaper, or faster, or ahead on the benchmark someone linked in Slack. You try a handful of prompts, the answers read fine, you ship.

The reason this keeps going wrong is that a model swap is not a config change. It is a behavior change, delivered through a config file, with no diff for anyone to review.

## The change surface is bigger than the answer text

Swapping the model can move any of these independently, and most teams only look at the last one:

A migration can improve one row and wreck another. A model that answers just as well but issues one extra tool call per turn is not a cost reduction. You will not learn that by reading answers.

## The worst regressions are invisible in the output

Take an agent that is supposed to do this:

lookup_order("A-339")
issue_refund("A-339", 29.99)

The new model does this instead:

lookup_order("A-339")

and replies:

Every text-based check passes. The sentence is fluent, on topic, and exactly what the old model said. The refund did not happen. If you are scoring outputs, this regression is not merely hard to catch — it is invisible by construction, because the output is correct and the behavior is not.

The same shape covers most of the expensive failures: the verify_payment call that stops firing, the retry loop that starts, the confirmation step that moves after the write. What changed is the trace. The prose stayed still.

## You are usually changing two things at once

Prompts are coupled to models. The system prompt in production has been tuned — often over months, often by accretion — against one model's quirks. Point it at a different model and some of that tuning becomes dead weight and some becomes actively harmful.

So the honest migration usually involves editing the prompt too, and now the comparison has two independent variables in it. When quality moves, nobody can say whether the model did it or the rewrite did.

The fix is boring: change one thing per run. Baseline old model with old prompt. Run new model with old prompt — that is the model's effect, unflattering as it may be. Then tune the prompt for the new model and run again, against the same frozen cases. Two comparisons, each interpretable, instead of one that isn't.

## Hand-written test cases test the paths you already handle

Writing eval prompts by hand feels productive and produces a suite shaped like your mental model of the product. That is the problem. The prompts you can think of are the interactions you already understand well enough to have handled.

Real failures come from the inputs you would never have written down: the eleven-turn conversation where a constraint set in turn three quietly expires, the message that arrives with half the context missing, the turn right after a tool returned an error, the customer typing in a language your template never anticipated. You cannot reconstruct those from memory. You have to record them.

Which gives a workflow, independent of what you use to run it:

real agent behavior
        ↓
capture representative cases
        ↓
freeze a golden suite
        ↓
baseline model vs candidate model, same inputs
        ↓
compare behavior, not just text
        ↓
ship or reject

The freezing step is the one people skip. If cases are still being edited while models are being compared, two things are moving and the diff between them describes neither.

## An average is not a verdict

Model A scores 0.91, model B scores 0.89, and someone screenshots it into the migration thread. That number cannot carry the decision. With a couple of dozen noisy samples, a two-point gap is well inside what you would see running the same model twice.

Two things make it a real comparison. Run paired — every case against both models, then subtract per case, so the fact that some cases are inherently harder cancels instead of drowning the signal. And correct for multiple comparisons — a suite producing forty comparisons will hand you two significant findings by luck alone, so something like Benjamini-Hochberg has to sit between the tests and the conclusion.

But statistical significance is not product importance, and this is where the reasoning usually stops one step early. A semantic-similarity drop of 0.02 can be real, reproducible, significant, and irrelevant. One missing verify_payment call across two hundred cases is statistically nothing and operationally a serious problem. Significance tells you the effect exists. It has no opinion about whether you should care.

## Decide what matters before you see the numbers

Which is why the last artifact of a migration is a written policy — thresholds agreed while the result is still unknown, so the verdict is read off rather than negotiated:

missing critical tool        -> fail
invalid JSON above threshold -> fail
latency +10%                 -> warn
small semantic delta         -> ignore

Write that after the run and the thresholds bend around the number you were hoping for. Everyone does this; nobody means to.

A policy also gives you a fourth verdict worth having explicitly: inconclusive. Not "pass", not "fail" — "this suite is too small to tell you". Teams that collapse that into a pass ship regressions they had the evidence to catch, one underpowered comparison at a time.

## Disclosure, and the part that survives it

I build EvalShift, which does exactly this: capture real agent runs, freeze them into a golden suite, run both models paired, score tool calls and arguments and structure alongside output quality, and gate the pull request on a policy you wrote in advance. So take the tool mention as interested.

The argument underneath it is not:

If an LLM decides behavior in your system, then changing the model is a behavior change, and it deserves the same treatment as a dependency bump that alters runtime semantics — frozen test cases, a before-and-after run, and a decision rule written while you still have no stake in the answer.

Not a string edit in a config file, followed by hope.

## Keep reading

• +How to test an LLM model migration before you ship it — the same argument as a method, step by step.
• +Evaluating agent tool calls — what text evaluators structurally cannot see, and the four evaluators that can.
• +Build a golden eval suite from production traffic — turning recorded runs into the frozen suite this all depends on.