What actually breaks when you switch LLMs
A model swap is a behavior change with no diff to review. What moves besides the answer text, and why the worst regressions read as correct.
When you switch LLMs, much more changes than the answer text: which tools an agent calls, the order and arguments of those calls, the shape of structured output, what the model refuses, latency, and cost per task. Treat the swap as a behavior change. Freeze a set of real cases, run the old and the new model on the same inputs, compare tool calls and structure as well as text, and write the pass and fail thresholds down before you see the results.
Changing the model behind an AI feature looks like the smallest change you will make all week. One
string moves from gpt-x to gemini-y. The new model is cheaper, or faster, or ahead on the
benchmark someone linked in Slack. You try a handful of prompts, the answers read fine, you ship.
The reason this keeps going wrong is that a model swap is not a config change. It is a behavior change, delivered through a config file, with no diff for anyone to review.
A disclosure: I build EvalShift, a tool for testing model changes on agents. It has a short section at the end. Everything before it works without it.
When do you have to switch models?
Often on the provider's schedule rather than yours. Models are retired on a published timeline, and some providers now reroute requests to a newer model instead of failing them.
Anthropic's deprecation policy ↗
promises "at least 60 days' notice" before a publicly released model is retired, and requests to a
retired model fail. Google's changelog ↗ describes a
quieter kind of retirement. From December 1, 2026, requests to gemini-2.5-pro will route to
gemini-3.8-flash and be billed at its rates, with breaking changes that "include function calling
and thought_signature validation".
| Event | Announced | Takes effect | What your code sees |
|---|---|---|---|
claude-sonnet-4-5-20250929 retired | September 30, 2026 | November 30, 2026 | Requests fail |
gemini-3.5-flash rerouted | October 8, 2026 | Already routed | Answers from gemini-3.6-flash |
gemini-3.7-flash rerouted | October 8, 2026 | Already routed | Answers from gemini-3.8-flash |
gemini-2.5-pro rerouted | November 2, 2026 | December 1, 2026 | Answers from gemini-3.8-flash |
gemini-2.5-flash rerouted | November 2, 2026 | December 1, 2026 | Answers from gemini-3.5-flash-lite |
A rerouted request keeps returning a success, so nothing in your error logs says the model changed. A forced swap is still a swap, and everything below applies whether you picked the new model or the provider picked it for you.
What changes when you switch LLMs?
Far more than the answer text. Swapping the model can move any of these independently, and most teams only look at the last one:
| What moves | How it usually surfaces |
|---|---|
| which tools the agent calls | a step silently stops happening |
| the order of those calls | a check runs after the action it was meant to gate |
| tool arguments | right tool, wrong amount, wrong id, wrong units |
| structured output | your parser throws, or worse, doesn't |
| refusal behavior | the model declines work it used to do |
| verbosity and format | downstream regex and UI truncation start missing |
| latency | p95 doubles, nobody attributes it to the swap |
| tokens and cost | the cheaper model turns out to be the pricier one per task |
| answer quality | the only one the playground actually shows you |
None of these rows is hypothetical. Published studies and the providers' own migration notes record each kind of change:
| What moved | Model change | What happened | Source |
|---|---|---|---|
| Output format | GPT-4, March → June 2023 | Directly executable code fell from 52.0% to 10.0% | Chen et al. ↗ |
| Verbosity | GPT-4, March → June 2023 | Code answers grew 20% longer in characters | Chen et al. ↗ |
| Tool arguments | gpt-4o vs gpt-3.5-turbo | 0.46 vs 2.08 calls per task with IDs that don't exist | τ-bench ↗ |
| Tool choice | Claude Opus 5 → Opus 5.5 | Forced tool_choice returns a 400 | Anthropic ↗ |
| Response shape | Claude Opus 5 → Opus 5.5 | Text between tool calls moves into thinking blocks | Anthropic ↗ |
| Defaults | Claude Opus 5 → Opus 5.5 | Default effort drops from high to medium | Anthropic ↗ |
| Tokens | Earlier Claude → Opus 4.7 | 1M tokens holds about 555k words, down from 750k | Anthropic ↗ |
Two of those rows invert the obvious reading. The GPT-4 code drop was largely formatting: the June model wrapped its code in markdown fences, and the authors note the "extra triple quotes render the code not executable". And Opus 5.5 costs $4 / $20 per million input and output tokens against Opus 5's $5 / $25, while Anthropic says it "tends to think more per turn" at the same effort setting.
A lower price per token says nothing about the price per task. A migration can improve one row and wreck another, and a model that answers just as well but issues one extra tool call per turn is not a cost reduction. You will not learn that by reading answers.
Why don't the worst regressions show up in the output?
Because the reply can stay correct while the behavior underneath it changes. A text check reads the reply, and the damage is in the tool calls.
Take an agent that is supposed to do this:
lookup_order("A-339")
issue_refund("A-339", 29.99)The new model does this instead:
lookup_order("A-339")and replies:
Your refund has been processed.
Every text-based check passes. The sentence is fluent, on topic, and exactly what the old model said. The refund did not happen. If you are scoring outputs, this regression is not merely hard to catch — it is invisible by construction, because the output is correct and the behavior is not.
The same shape covers most of the expensive failures: the verify_payment call that stops firing,
the retry loop that starts, the confirmation step that moves after the write. What changed is the
trace. The prose stayed still.
This is why τ-bench ↗, a benchmark for customer-service agents, grades the database state at the end of a conversation instead of what the agent said. Its authors found gpt-4o "usually makes the right type of tool call(s) but fills in one or more arguments incorrectly". Another 25% of its failures came from the wrong type of tool call, because the agent misread the domain rules.
Should you change the prompt during a model migration?
Not in the same comparison. Prompts are tuned to one model, so a run that changes both cannot tell you which change moved the score.
Prompts are coupled to models. The system prompt in production has been tuned — often over months, often by accretion — against one model's quirks. Point it at a different model and some of that tuning becomes dead weight and some becomes actively harmful.
How tight the coupling is has been measured. Sclar et al. found ↗ that meaning-preserving formatting changes, such as separators, spacing and casing, moved LLaMA-2-13B's few-shot accuracy by up to 76 points. They also found that "format performance only weakly correlates between models", so a format that suits your old model tells you little about the new one.
So the honest migration usually involves editing the prompt too, and now the comparison has two independent variables in it. When quality moves, nobody can say whether the model did it or the rewrite did.
The fix is boring: change one thing per run. Baseline old model with old prompt. Run new model with old prompt — that is the model's effect, unflattering as it may be. Then tune the prompt for the new model and run again against the same frozen cases, for two comparisons that each mean something instead of one that doesn't.
Why aren't hand-written test cases enough?
Because they test the paths you already handle. The cases worth comparing on come from recorded traffic, frozen before the comparison starts.
Writing eval prompts by hand feels productive and produces a suite shaped like your mental model of the product. That is the problem. The prompts you can think of are the interactions you already understand well enough to have handled.
Real failures come from the inputs you would never have written down: the eleven-turn conversation where a constraint set in turn three quietly expires, the message that arrives with half the context missing, the turn right after a tool returned an error. Or the customer typing in a language your template never anticipated. You cannot reconstruct those from memory, so you have to record them.
Which gives a workflow, independent of what you use to run it:
real agent behavior
↓
capture representative cases
↓
freeze a golden suite
↓
baseline model vs candidate model, same inputs
↓
compare behavior, not just text
↓
ship or rejectThe freezing step is the one people skip. If cases are still being edited while models are being compared, two things are moving and the diff between them describes neither.
Is a small score difference between two models meaningful?
Not on its own. A gap between two averages means something only once you know how far the same model moves against itself, and only when both models ran on the same cases.
Model A scores 0.91, model B scores 0.89, and someone screenshots it into the migration thread. That number cannot carry the decision. With a couple of dozen noisy samples, a two-point gap is well inside what you would see running the same model twice.
Agents make the noise worse. On τ-bench's retail tasks, gpt-4o succeeded on about 61% of tasks in a single trial, but the chance that it solved the same task in all of eight trials fell to about 25%. An agent that inconsistent can swing a small suite by several points between two runs of one model.
Two things make it a real comparison. Run paired — every case against both models, then subtract per case, so the fact that some cases are inherently harder cancels instead of drowning the signal. Miller's Adding Error Bars to Evals ↗ works an example where scores correlate at 0.5 and pairing cuts the variance of the estimated difference by a third, which he calls a "free" reduction.
The same paper warns about cases that come in related groups, such as several turns of one conversation. On two evals run with Anthropic models, clustered standard errors came out over 3X larger than naive ones, so a suite built from multi-turn traces is noisier than its case count suggests.
Then correct for multiple comparisons. A suite producing forty comparisons at a 5% threshold will hand you two significant findings by luck alone, so something like Benjamini-Hochberg ↗ has to sit between the tests and the conclusion.
But statistical significance is not product importance, and this is where the reasoning usually stops
one step early. A semantic-similarity drop of 0.02 can be real, reproducible, significant, and
irrelevant. One missing verify_payment call across two hundred cases is statistically nothing and
operationally a serious problem.
Significance tells you the effect exists. It has no opinion about whether you should care.
When should you set the pass and fail thresholds?
Before the run. Thresholds agreed while the result is still unknown let you read the verdict off rather than negotiate it.
The last artifact of a migration is a written policy, something like:
missing critical tool -> fail invalid JSON above threshold -> fail latency +10% -> warn small semantic delta -> ignore
Write that after the run and the thresholds bend around the number you were hoping for. Everyone does this; nobody means to.
A policy also gives you a fourth verdict worth having explicitly: inconclusive. Not "pass", not "fail" — "this suite is too small to tell you". Teams that collapse that into a pass ship regressions they had the evidence to catch, one underpowered comparison at a time.
Treat a model swap like a dependency bump
If an LLM decides behavior in your system, then changing the model is a behavior change. It deserves the same treatment as a dependency bump that alters runtime semantics: frozen test cases, a before-and-after run, and a decision rule written while you still have no stake in the answer.
Not a string edit in a config file, followed by hope.
Doing this with EvalShift
Disclosure again: I'm the founder, so take this section as interested. EvalShift does the workflow above for agents, and nothing in the method depends on it.
- +Capture. The capture SDK records real agent runs, with every model call, tool call and tool result, and those runs are frozen into a golden suite.
- +Paired runs. Both models run the same cases, and every score is a per-case difference.
- +Behavior, not just text. The evaluators score which tools were called, their arguments and the structure of the output alongside output quality.
- +A policy written in advance. A
migration_policysets the thresholds, the verdict can come back inconclusive, and the GitHub Action gates the pull request on it.
FAQ
What breaks when you switch LLM models? More than the answer text. A different model can change which tools an agent calls, the order it calls them in, the arguments it passes, whether structured output still validates, what it refuses, how long it takes, and how many tokens it burns. A migration can improve answer quality and regress tool behavior in the same run.
Why don't LLM regressions show up in the output? Because output and behavior are separate
surfaces. An agent that calls lookup_order but skips issue_refund can still reply that the
refund was processed — fluent, on topic, identical to what the old model said. Every text-scoring
check passes while the trace underneath has changed.
Should you change the prompt during a model migration? Eventually yes, but not in the same comparison. Prompts are tuned against one model's quirks, so a migration that edits both leaves nobody able to say whether the model or the rewrite moved the score. Run the new model against the old prompt first to isolate the model's effect, then tune and run again against the same frozen cases.
Is a 2% eval score difference between two models meaningful? Usually not on its own. With a few dozen noisy samples, that gap is inside the range you would see running the same model twice. It takes paired runs over identical cases, plus a correction for multiple comparisons, before a difference is evidence — and even then, significance says the effect is real, not that it matters to your product.
How much notice do providers give before retiring a model? Anthropic promises at least 60 days' notice before it retires a publicly released model, after which requests to it fail. Google announced on November 2, 2026 that requests to gemini-2.5-pro and gemini-2.5-flash will route to newer models from December 1, 2026, so those requests keep succeeding on a different model.
Keep reading
- +How to test an LLM model migration before you ship it — the same argument as a method, step by step.
- +Evaluating agent tool calls — what text evaluators structurally cannot see, and the four evaluators that can.
- +Build a golden dataset for LLM evals from production traffic — turning recorded runs into the frozen suite this all depends on.
