## The new model was 59% cheaper and 75% faster. I still wouldn't ship it.

URL: https://www.evalshift.dev/blog/cheaper-faster-and-still-a-fail
Published: 2026-09-12
Tag: migration
Summary: A candidate model cut cost 59% and latency 75% with zero failed calls. The migration report still said FAIL, because the agent's behavior changed.

Takeaways:
- Cost, latency and error rate all improved — 58.8% cheaper, 75.2% faster, zero failed or truncated calls — and the migration report still returned FAIL.
- The overall regression-rate budget passed. Tool-selection divergence at 25% against a 10% limit and an equivalence rate of 61.4% against a 75% floor did not.
- The regressions were behavioral: an extra search_web call instead of an answer, an action reported as done with no tool call, a request routed to a different tool.
- Per-evaluator results disagreed — routing arguments and conformance held while tool-selection divergence, semantic similarity and the pairwise judge regressed. One averaged score would have hidden that.
- Economics alone cannot tell you whether a migration is safe. Replay the same suite against both models and read the per-example diff before the model string changes.

The migration looked like an obvious win on economics. In one EvalShift run, the candidate model
was:

- 58.8% cheaper
- 75.2% faster
- 0 failed or truncated calls

![Run economics table: source and target each made 16 calls with no failures or truncations; the target cost $0.0741 against $0.1799 and averaged 773 ms against 3.1 s.](/blog-images/blog1/blog1-3.png "Run economics. Same 16 calls on both sides, no failures, and every number a finance reviewer cares about moved the right way.")

Then the migration report returned:

**FAIL**

Not because the model crashed. Not because the API changed. Not because the output stopped
parsing.

It changed what the agent actually did.

![Migration verdict card: FAIL, 5 of 7 budgets within policy, 61.4% equivalent, 25.0% improved, 13.6% regressed, cost −58.8%, latency −75.2%.](/blog-images/blog1/blog1-1.png "The verdict card. Five of seven budgets passed; the two that did not are the two that describe behavior.")

## What failed

I replayed 16 real examples against the source and candidate models. Across the run, 13.6% of
evaluated comparisons regressed.

The overall regression-rate budget actually passed.

Two other checks did not:

- Tool-selection divergence hit 25%, against a 10% limit.
- The overall equivalence rate fell to 61.4%, below the required 75%.

![Report verdict and findings: the target breached max_tool_divergence at 25.0% against a 10% ceiling and fell below min_equivalence_rate at 61.4% against 75%; findings list searching instead of answering, confirming actions it never performed, and routing to the wrong tool.](/blog-images/blog1/blog1-2.png "The written verdict and its findings. The recommendation is not to migrate until the tool-selection and semantic regressions are fixed.")

That distinction matters. Looking only at latency, cost, successful requests, or even a few
manually inspected outputs would have made this migration look pretty attractive.

The behavioral diff told a different story.

## The failures weren't cosmetic

One example asked for an opinion on working late.

The source model answered the question directly.

The candidate instead issued a `search_web` call and returned no text.

![Per-example diff for the working-late question: the source called no tools and answered in two sentences; the target called search_web with the query "is working late worth it productivity well-being" and produced no final text. Flagged as TOOL_SELECTION_DRIFT.](/blog-images/blog1/blog1-5.png "The diff for that example. Source trace on the left, target trace on the right, and the reason it was flagged at the top.")

Other cases had the same general problem in different forms:

- an action was reported as completed even though the corresponding tool was never called;
- a request was routed to a different tool;
- an unnecessary tool call appeared where the source answered directly.

These are not necessarily signs that the candidate model is "bad." They show that changing the
model changed the application.

For an agent, the model is part of the control flow.

## Aggregate quality can hide this

The evaluator breakdown made the tradeoff clearer.

Routing arguments and routing conformance were broadly equivalent in this run, while
tool-selection divergence, semantic similarity, and the pairwise equivalence judge showed
regressions.

![Per-evaluator table over 16 examples: routing divergence regressed (score delta −0.167, likely), routing args equivalent (+0.179, unclear), routing conformance equivalent (+0.047, unclear), the LLM equivalence judge regressed critically (−0.438, certain), and semantic similarity regressed (−0.214, likely).](/blog-images/blog1/blog1-4.png "Overall, by evaluator. Two rows say equivalent, three say regressed, and each comes with its own effect size, confidence interval and confidence label.")

A single average score would flatten all of that into one number.

For a migration, I care more about the question:

*What changed, on which examples, and is that change acceptable for this application?*

## This is why I built EvalShift

I kept running into model migrations that were tested roughly like this:

change model → try several prompts → outputs look fine → ship.

That works until the difference is something subtle like an extra tool call, a missing action, or
different routing behavior.

EvalShift replays the same suite against the source and target model, compares the behavior, and
produces the migration decision and individual diffs before the model string gets changed in
production. The CLI is local-first and produces a self-contained HTML report; hosted upload is
optional.

The interesting result from this run wasn't that the candidate was worse.

It was that "59% cheaper and 75% faster" wasn't enough information to decide whether the migration
was safe.