← all posts
migration·Lukas Babaliauskas·Aug 20, 2026·updated Nov 17, 2026·12 min read

What actually breaks when you switch LLMs

A model swap is a behavior change with no diff to review. What moves besides the answer text, and why the worst regressions read as correct.

When you switch LLMs, much more changes than the answer text: which tools an agent calls, the order and arguments of those calls, the shape of structured output, what the model refuses, latency, and cost per task. Treat the swap as a behavior change. Freeze a set of real cases, run the old and the new model on the same inputs, compare tool calls and structure as well as text, and write the pass and fail thresholds down before you see the results.

Changing the model behind an AI feature looks like the smallest change you will make all week. One string moves from gpt-x to gemini-y. The new model is cheaper, or faster, or ahead on the benchmark someone linked in Slack. You try a handful of prompts, the answers read fine, you ship.

The reason this keeps going wrong is that a model swap is not a config change. It is a behavior change, delivered through a config file, with no diff for anyone to review.

A disclosure: I build EvalShift, a tool for testing model changes on agents. It has a short section at the end. Everything before it works without it.

When do you have to switch models?

Often on the provider's schedule rather than yours. Models are retired on a published timeline, and some providers now reroute requests to a newer model instead of failing them.

Anthropic's deprecation policy ↗ promises "at least 60 days' notice" before a publicly released model is retired, and requests to a retired model fail. Google's changelog ↗ describes a quieter kind of retirement. From December 1, 2026, requests to gemini-2.5-pro will route to gemini-3.8-flash and be billed at its rates, with breaking changes that "include function calling and thought_signature validation".

EventAnnouncedTakes effectWhat your code sees
claude-sonnet-4-5-20250929 retiredSeptember 30, 2026November 30, 2026Requests fail
gemini-3.5-flash reroutedOctober 8, 2026Already routedAnswers from gemini-3.6-flash
gemini-3.7-flash reroutedOctober 8, 2026Already routedAnswers from gemini-3.8-flash
gemini-2.5-pro reroutedNovember 2, 2026December 1, 2026Answers from gemini-3.8-flash
gemini-2.5-flash reroutedNovember 2, 2026December 1, 2026Answers from gemini-3.5-flash-lite

A rerouted request keeps returning a success, so nothing in your error logs says the model changed. A forced swap is still a swap, and everything below applies whether you picked the new model or the provider picked it for you.

What changes when you switch LLMs?

Far more than the answer text. Swapping the model can move any of these independently, and most teams only look at the last one:

What movesHow it usually surfaces
which tools the agent callsa step silently stops happening
the order of those callsa check runs after the action it was meant to gate
tool argumentsright tool, wrong amount, wrong id, wrong units
structured outputyour parser throws, or worse, doesn't
refusal behaviorthe model declines work it used to do
verbosity and formatdownstream regex and UI truncation start missing
latencyp95 doubles, nobody attributes it to the swap
tokens and costthe cheaper model turns out to be the pricier one per task
answer qualitythe only one the playground actually shows you

None of these rows is hypothetical. Published studies and the providers' own migration notes record each kind of change:

What movedModel changeWhat happenedSource
Output formatGPT-4, March → June 2023Directly executable code fell from 52.0% to 10.0%Chen et al. ↗
VerbosityGPT-4, March → June 2023Code answers grew 20% longer in charactersChen et al. ↗
Tool argumentsgpt-4o vs gpt-3.5-turbo0.46 vs 2.08 calls per task with IDs that don't existτ-bench ↗
Tool choiceClaude Opus 5 → Opus 5.5Forced tool_choice returns a 400Anthropic ↗
Response shapeClaude Opus 5 → Opus 5.5Text between tool calls moves into thinking blocksAnthropic ↗
DefaultsClaude Opus 5 → Opus 5.5Default effort drops from high to mediumAnthropic ↗
TokensEarlier Claude → Opus 4.71M tokens holds about 555k words, down from 750kAnthropic ↗

Two of those rows invert the obvious reading. The GPT-4 code drop was largely formatting: the June model wrapped its code in markdown fences, and the authors note the "extra triple quotes render the code not executable". And Opus 5.5 costs $4 / $20 per million input and output tokens against Opus 5's $5 / $25, while Anthropic says it "tends to think more per turn" at the same effort setting.

A lower price per token says nothing about the price per task. A migration can improve one row and wreck another, and a model that answers just as well but issues one extra tool call per turn is not a cost reduction. You will not learn that by reading answers.

Why don't the worst regressions show up in the output?

Because the reply can stay correct while the behavior underneath it changes. A text check reads the reply, and the damage is in the tool calls.

Take an agent that is supposed to do this:

text
lookup_order("A-339")
issue_refund("A-339", 29.99)

The new model does this instead:

text
lookup_order("A-339")

and replies:

+

Your refund has been processed.

Every text-based check passes. The sentence is fluent, on topic, and exactly what the old model said. The refund did not happen. If you are scoring outputs, this regression is not merely hard to catch — it is invisible by construction, because the output is correct and the behavior is not.

The same shape covers most of the expensive failures: the verify_payment call that stops firing, the retry loop that starts, the confirmation step that moves after the write. What changed is the trace. The prose stayed still.

This is why τ-bench ↗, a benchmark for customer-service agents, grades the database state at the end of a conversation instead of what the agent said. Its authors found gpt-4o "usually makes the right type of tool call(s) but fills in one or more arguments incorrectly". Another 25% of its failures came from the wrong type of tool call, because the agent misread the domain rules.

Should you change the prompt during a model migration?

Not in the same comparison. Prompts are tuned to one model, so a run that changes both cannot tell you which change moved the score.

Prompts are coupled to models. The system prompt in production has been tuned — often over months, often by accretion — against one model's quirks. Point it at a different model and some of that tuning becomes dead weight and some becomes actively harmful.

How tight the coupling is has been measured. Sclar et al. found ↗ that meaning-preserving formatting changes, such as separators, spacing and casing, moved LLaMA-2-13B's few-shot accuracy by up to 76 points. They also found that "format performance only weakly correlates between models", so a format that suits your old model tells you little about the new one.

So the honest migration usually involves editing the prompt too, and now the comparison has two independent variables in it. When quality moves, nobody can say whether the model did it or the rewrite did.

The fix is boring: change one thing per run. Baseline old model with old prompt. Run new model with old prompt — that is the model's effect, unflattering as it may be. Then tune the prompt for the new model and run again against the same frozen cases, for two comparisons that each mean something instead of one that doesn't.

Why aren't hand-written test cases enough?

Because they test the paths you already handle. The cases worth comparing on come from recorded traffic, frozen before the comparison starts.

Writing eval prompts by hand feels productive and produces a suite shaped like your mental model of the product. That is the problem. The prompts you can think of are the interactions you already understand well enough to have handled.

Real failures come from the inputs you would never have written down: the eleven-turn conversation where a constraint set in turn three quietly expires, the message that arrives with half the context missing, the turn right after a tool returned an error. Or the customer typing in a language your template never anticipated. You cannot reconstruct those from memory, so you have to record them.

Which gives a workflow, independent of what you use to run it:

text
real agent behavior
        ↓
capture representative cases
        ↓
freeze a golden suite
        ↓
baseline model vs candidate model, same inputs
        ↓
compare behavior, not just text
        ↓
ship or reject

The freezing step is the one people skip. If cases are still being edited while models are being compared, two things are moving and the diff between them describes neither.

Is a small score difference between two models meaningful?

Not on its own. A gap between two averages means something only once you know how far the same model moves against itself, and only when both models ran on the same cases.

Model A scores 0.91, model B scores 0.89, and someone screenshots it into the migration thread. That number cannot carry the decision. With a couple of dozen noisy samples, a two-point gap is well inside what you would see running the same model twice.

Agents make the noise worse. On τ-bench's retail tasks, gpt-4o succeeded on about 61% of tasks in a single trial, but the chance that it solved the same task in all of eight trials fell to about 25%. An agent that inconsistent can swing a small suite by several points between two runs of one model.

Two things make it a real comparison. Run paired — every case against both models, then subtract per case, so the fact that some cases are inherently harder cancels instead of drowning the signal. Miller's Adding Error Bars to Evals ↗ works an example where scores correlate at 0.5 and pairing cuts the variance of the estimated difference by a third, which he calls a "free" reduction.

The same paper warns about cases that come in related groups, such as several turns of one conversation. On two evals run with Anthropic models, clustered standard errors came out over 3X larger than naive ones, so a suite built from multi-turn traces is noisier than its case count suggests.

Then correct for multiple comparisons. A suite producing forty comparisons at a 5% threshold will hand you two significant findings by luck alone, so something like Benjamini-Hochberg ↗ has to sit between the tests and the conclusion.

But statistical significance is not product importance, and this is where the reasoning usually stops one step early. A semantic-similarity drop of 0.02 can be real, reproducible, significant, and irrelevant. One missing verify_payment call across two hundred cases is statistically nothing and operationally a serious problem.

Significance tells you the effect exists. It has no opinion about whether you should care.

When should you set the pass and fail thresholds?

Before the run. Thresholds agreed while the result is still unknown let you read the verdict off rather than negotiate it.

The last artifact of a migration is a written policy, something like:

text
missing critical tool        -> fail
invalid JSON above threshold -> fail
latency +10%                 -> warn
small semantic delta         -> ignore

Write that after the run and the thresholds bend around the number you were hoping for. Everyone does this; nobody means to.

A policy also gives you a fourth verdict worth having explicitly: inconclusive. Not "pass", not "fail" — "this suite is too small to tell you". Teams that collapse that into a pass ship regressions they had the evidence to catch, one underpowered comparison at a time.

Treat a model swap like a dependency bump

If an LLM decides behavior in your system, then changing the model is a behavior change. It deserves the same treatment as a dependency bump that alters runtime semantics: frozen test cases, a before-and-after run, and a decision rule written while you still have no stake in the answer.

Not a string edit in a config file, followed by hope.

Doing this with EvalShift

Disclosure again: I'm the founder, so take this section as interested. EvalShift does the workflow above for agents, and nothing in the method depends on it.

  • +Capture. The capture SDK records real agent runs, with every model call, tool call and tool result, and those runs are frozen into a golden suite.
  • +Paired runs. Both models run the same cases, and every score is a per-case difference.
  • +Behavior, not just text. The evaluators score which tools were called, their arguments and the structure of the output alongside output quality.
  • +A policy written in advance. A migration_policy sets the thresholds, the verdict can come back inconclusive, and the GitHub Action gates the pull request on it.

FAQ

What breaks when you switch LLM models? More than the answer text. A different model can change which tools an agent calls, the order it calls them in, the arguments it passes, whether structured output still validates, what it refuses, how long it takes, and how many tokens it burns. A migration can improve answer quality and regress tool behavior in the same run.

Why don't LLM regressions show up in the output? Because output and behavior are separate surfaces. An agent that calls lookup_order but skips issue_refund can still reply that the refund was processed — fluent, on topic, identical to what the old model said. Every text-scoring check passes while the trace underneath has changed.

Should you change the prompt during a model migration? Eventually yes, but not in the same comparison. Prompts are tuned against one model's quirks, so a migration that edits both leaves nobody able to say whether the model or the rewrite moved the score. Run the new model against the old prompt first to isolate the model's effect, then tune and run again against the same frozen cases.

Is a 2% eval score difference between two models meaningful? Usually not on its own. With a few dozen noisy samples, that gap is inside the range you would see running the same model twice. It takes paired runs over identical cases, plus a correction for multiple comparisons, before a difference is evidence — and even then, significance says the effect is real, not that it matters to your product.

How much notice do providers give before retiring a model? Anthropic promises at least 60 days' notice before it retires a publicly released model, after which requests to it fail. Google announced on November 2, 2026 that requests to gemini-2.5-pro and gemini-2.5-flash will route to newer models from December 1, 2026, so those requests keep succeeding on a different model.

Keep reading