Can a model migration be cheaper and faster and still fail?

Yes. In this run the candidate was 58.8% cheaper and 75.2% faster with no failed or truncated calls, and the report still returned FAIL because tool-selection divergence hit 25% against a 10% limit and the equivalence rate fell to 61.4% against a required 75%. Cost and latency say nothing about whether the agent still does the same thing.

Why is the regression rate passing while the migration fails?

Because a migration is judged against several independent budgets, not one score. The overall regression-rate budget was within policy at 13.6%, but tool-selection divergence and the overall equivalence rate each breached their own limit, and any breached budget fails the migration.

What does a behavioral regression look like in a model swap?

An agent that issues a search_web call and returns no text where the previous model answered directly; an action reported as completed although the corresponding tool was never called; a request routed to a different tool; an unnecessary tool call where the source answered without one. The output can read fine while the trace underneath has changed.

← all posts

migration·Sep 12, 2026·4 min read

The new model was 59% cheaper and 75% faster. I still wouldn't ship it.

A candidate model cut cost 59% and latency 75% with zero failed calls. The migration report still said FAIL, because the agent's behavior changed.

takeaways

Cost, latency and error rate all improved — 58.8% cheaper, 75.2% faster, zero failed or truncated calls — and the migration report still returned FAIL.
The overall regression-rate budget passed. Tool-selection divergence at 25% against a 10% limit and an equivalence rate of 61.4% against a 75% floor did not.
The regressions were behavioral: an extra search_web call instead of an answer, an action reported as done with no tool call, a request routed to a different tool.
Per-evaluator results disagreed — routing arguments and conformance held while tool-selection divergence, semantic similarity and the pairwise judge regressed. One averaged score would have hidden that.
Economics alone cannot tell you whether a migration is safe. Replay the same suite against both models and read the per-example diff before the model string changes.

The migration looked like an obvious win on economics. In one EvalShift run, the candidate model was:

+58.8% cheaper
+75.2% faster
+0 failed or truncated calls

Then the migration report returned:

FAIL

Not because the model crashed. Not because the API changed. Not because the output stopped parsing.

It changed what the agent actually did.

Migration verdict card: FAIL, 5 of 7 budgets within policy, 61.4% equivalent, 25.0% improved, 13.6% regressed, cost −58.8%, latency −75.2%. — The verdict card. Five of seven budgets passed; the two that did not are the two that describe behavior.

## What failed

I replayed 16 real examples against the source and candidate models. Across the run, 13.6% of evaluated comparisons regressed.

The overall regression-rate budget actually passed.

Two other checks did not:

+Tool-selection divergence hit 25%, against a 10% limit.
+The overall equivalence rate fell to 61.4%, below the required 75%.

Report verdict and findings: the target breached max_tool_divergence at 25.0% against a 10% ceiling and fell below min_equivalence_rate at 61.4% against 75%; findings list searching instead of answering, confirming actions it never performed, and routing to the wrong tool. — The written verdict and its findings. The recommendation is not to migrate until the tool-selection and semantic regressions are fixed.

That distinction matters. Looking only at latency, cost, successful requests, or even a few manually inspected outputs would have made this migration look pretty attractive.

The behavioral diff told a different story.

## The failures weren't cosmetic

One example asked for an opinion on working late.

The source model answered the question directly.

The candidate instead issued a search_web call and returned no text.

Per-example diff for the working-late question: the source called no tools and answered in two sentences; the target called search_web with the query "is working late worth it productivity well-being" and produced no final text. Flagged as TOOL_SELECTION_DRIFT. — The diff for that example. Source trace on the left, target trace on the right, and the reason it was flagged at the top.

Other cases had the same general problem in different forms:

+an action was reported as completed even though the corresponding tool was never called;
+a request was routed to a different tool;
+an unnecessary tool call appeared where the source answered directly.

These are not necessarily signs that the candidate model is "bad." They show that changing the model changed the application.

For an agent, the model is part of the control flow.

## Aggregate quality can hide this

The evaluator breakdown made the tradeoff clearer.

Routing arguments and routing conformance were broadly equivalent in this run, while tool-selection divergence, semantic similarity, and the pairwise equivalence judge showed regressions.

Per-evaluator table over 16 examples: routing divergence regressed (score delta −0.167, likely), routing args equivalent (+0.179, unclear), routing conformance equivalent (+0.047, unclear), the LLM equivalence judge regressed critically (−0.438, certain), and semantic similarity regressed (−0.214, likely). — Overall, by evaluator. Two rows say equivalent, three say regressed, and each comes with its own effect size, confidence interval and confidence label.

A single average score would flatten all of that into one number.

For a migration, I care more about the question:

What changed, on which examples, and is that change acceptable for this application?

## This is why I built EvalShift

I kept running into model migrations that were tested roughly like this:

change model → try several prompts → outputs look fine → ship.

That works until the difference is something subtle like an extra tool call, a missing action, or different routing behavior.

EvalShift replays the same suite against the source and target model, compares the behavior, and produces the migration decision and individual diffs before the model string gets changed in production. The CLI is local-first and produces a self-contained HTML report; hosted upload is optional.

The interesting result from this run wasn't that the candidate was worse.

It was that "59% cheaper and 75% faster" wasn't enough information to decide whether the migration was safe.

← all posts