This one passed: 71% cheaper, and the agent still does the same thing

The previous post was about a migration that was cheaper, faster and still failed. This is the other outcome. It is the more common one once a suite is in decent shape, and I see it written up far less often, because a pass is boring.

It shouldn't be. A pass is only worth something if the limits were fixed before the run. So this post is the whole thing: the policy, the reasoning behind every number in it, and the EvalShift report it produced.

For anyone arriving here cold: EvalShift is a local-first CLI for testing LLM model migrations. It replays a frozen golden suite against your current model and a candidate, scores every pair of outputs with tool-call, structural, semantic and LLM-judge evaluators, runs paired statistics over the deltas, and turns a migration_policy block you wrote in evalshift.yaml into one of four verdicts: pass, conditional_pass, fail or inconclusive. It writes a single-file report.html on your machine; nothing is uploaded unless you run evalshift push. Every figure in this post is a panel from that report.

The candidate: a customer-support agent with six tools (lookup_customer, lookup_order, check_refund_policy, issue_refund, escalate_to_human, search_kb), moving from gemini-3.7-pro to gemini-3.7-flash. The suite: 120 examples recorded in production by the EvalShift capture SDK, promoted into a golden JSONL suite with evalshift capture sync, and sliced by tag according to what the conversation was about.

One evalshift compare command, real API calls on both sides, and the report opened on:

Cheaper by 71.2%, faster by 61.9%, verdict PASS. The economics look like the last post's. The difference is everything that was decided before the run.

## The numbers were written before the run

EvalShift's migration_policy is a block in evalshift.yaml with seven budgets: overall regression rate, critical regression count, equivalent-or-better rate, tool-argument drift, tool-selection divergence, cost increase and latency increase. A slices map under it lets any slice override any budget, inheriting the top-level value where it doesn't. Evaluators are configured per suite; budgets are set once and tightened per slice.

evalshift init --profile cost-reduction scaffolds a starting policy: 2% overall regression rate, zero critical regressions, 97% equivalence, 1% tool-argument drift, 5% cost increase, 30% latency increase. That is a starting point, not a decision. This is what the project actually ran with:

migration_policy:
  max_overall_regression_rate: 0.05
  max_critical_regressions: 0
  min_equivalence_rate: 0.90
  max_tool_argument_drift: 0.10
  max_tool_divergence: 0.05
  max_cost_increase: 0.0
  max_latency_increase: 0.30
  slices:
    security:
      max_overall_regression_rate: 0.0
      max_tool_divergence: 0.0
    refund:
      max_tool_argument_drift: 0.0

Three things moved between the profile and this file.

### Check every rate against the suite size

A rate over n rows can only move in steps of 1/n. This suite has 108 tool-argument rows, so one drifted row is 0.93%, and a 1% budget is a zero-tolerance budget wearing a percentage. EvalShift prints a recommendation when a budget is below the granularity of its denominator, naming the budget, the value and the row count, and this one would have triggered it.

So the rule I use: if I mean zero, I write 0.0. Where I don't, I set a number the sample can resolve. 10% argument drift on 108 rows is ten reworded search queries, which is what drift on this agent looks like in practice.

### Tolerance where wording lives, zero where money and access live

The overall regression rate went up, from 2% to 5%. The pairwise LLM judge is blocking on this project, and a judge flips on rewording; 2% of 444 comparisons is nine judge calls deciding the migration. 5% leaves room for the judge to disagree about phrasing without anything real being allowed through.

The strictness moved into the slices instead. security gets zero regressions and zero tool-selection divergence: a model that starts routing an account-access request to a different tool does not get a percentage. refund gets zero argument drift: an order id or an amount that drifts is a wrong refund, not a reworded one.

The slices that are not in the policy matter too. customer_lookup has 16 examples. EvalShift tests a comparison below 20 pairs but flags it uncertain, so that slice inherits the top-level budgets and gets no tighter ones. It is the slice I would grow before I tightened it.

### The cost budget is the reason for the migration

max_cost_increase: 0.0. The point of the exercise is to spend less. A candidate that costs more has failed before any quality number is read, and a policy should say so instead of leaving it to whoever reads the economics card. EvalShift measures cost and latency from the run's own calls, so these two budgets gate even when no quality evaluator does.

Two evaluator decisions sit alongside the budgets. Every EvalShift evaluator carries a blocking flag; advisory (blocking: false) results are reported and ranked but never change the verdict. The judge is blocking: true here; init writes false because at a dozen examples judge noise would decide the verdict, and at 120 with an audited criterion it earns its vote. semantic stays advisory. It measures wording, and wording is the one thing this migration was allowed to change.

## What the run measured

The first row is the one worth reading twice. 16 of 444 is 3.6%, and its 95% Wilson interval runs from 2.2% to 5.8%. The interval crosses the limit.

EvalShift's rule is asymmetric on purpose. A breach fails only when the interval confirms it; a breach the interval cannot confirm returns inconclusive, because the suite was too small to say. A budget the observation held is conclusive however wide its interval, because a wide interval must never downgrade a clean run. This budget held, so it passes, and the interval is printed so the reader knows how much room there was.

The three slice budgets held at exactly zero. Zero on 96 comparisons means something. Zero on 12 would not, which is why text_only has no slice budget at all.

## By evaluator

Four evaluators scored this run. EvalShift's tool_selection evaluator reads the recorded traces and scores two axes: conformance, where each side is graded against the suite's recorded tool calls, and divergence, where the target is graded against what the source did. tool_arguments scores argument values field by field against the expected call. The llm_judge is pairwise and sees the two outputs as anonymous A and B. semantic is embedding similarity between the two outputs. The first three are blocking; the last is advisory.

Four blocking rows say equivalent or improved. The one that regressed is advisory, and it measures the one thing this migration was allowed to change. Had semantic been blocking, the same run would have come back conditional_pass on a medium-severity regression in phrasing. That is not a reason to make every evaluator advisory. It is a reason to decide, per evaluator, whether what it measures is something you are willing to block a migration on.

The conformance row says the candidate matched the suite's recorded tool calls more often than the model that produced them: 11 examples improved, one regressed. Most of the eleven were refund requests where the source went straight to issue_refund.

## The diffs I still read

A pass is not permission to skip the diff. For every flagged example, the EvalShift report shows the reason it was flagged, the tool calls on each side, the argument-level diff, and the conversation context that led to it. Two examples from this run, one from each side of the ledger.

The refund went out either way, and the final text on both sides would pass any output check you care to write. The trace is where the difference lives: one side verified before acting and the other did not. This is the class of change a text evaluator cannot see in either direction, and the reason EvalShift's tool-call evaluators score the trace rather than the prose.

A reviewer would call those the same query. The evaluator scored it 0.84 against a 0.9 floor and counted it as drift, five rows out of 108 did the same, and the budget said five is fine. That is what the 10% was for. The same five rows in the refund slice would have failed the run, and that was also decided in advance.

## What EvalShift did in this run

The whole run, as a list of the product's parts, in the order they were used:

• +The capture SDK recorded the agent's real conversations in production, tool calls included, and evalshift capture sync promoted them into a frozen golden JSONL suite with the tool evaluators written from what the captures actually contained.
• +evalshift compare replayed every example against the source and the target model, paired per example, with the same inputs, tools and context on both sides.
• +The tool-call evaluators (tool_selection, tool_arguments) scored the traces, the pairwise LLM judge scored the outputs, and semantic measured drift in wording, advisory only.
• +Paired statistics turned each evaluator's deltas into an effect size, a 95% interval and a corrected confidence label, so a two-point average drop and a real regression are told apart mechanically.
• +migration_policy in evalshift.yaml held seven budgets, three of them tightened to zero on the security and refund slices, and every proportion budget was judged with a Wilson interval that can return inconclusive instead of a false fail.
• +report.html was written locally, verdict first, with a per-example diff for everything flagged. Nothing left the machine.
• +--policy-gate made the verdict an exit code, which is what the EvalShift GitHub Action uses to block a pull request on the same policy once the suite runs in CI.

## What a pass proves

Not that nothing changed. 16 comparisons regressed and 28 improved; the agent phrases search queries differently and checks the refund policy more often than it used to.

It proves that every change stayed inside limits that were written down before anyone saw a number, on a suite that was frozen before the run. That is the entire claim, and it is enough to act on, because there is nothing left to negotiate: the argument about what counts as acceptable happened in the YAML, not in the meeting after the report.

What happens next is the boring part, which is the point. The model string changes in production. The suite does not. It runs again on the next pull request through the EvalShift GitHub Action, gated on the same policy, against the new baseline.

evalshift compare --suite-name support_routing --to gemini-3.7-flash --policy-gate --open

If the previous post was the reason to run the comparison, this one is the reason to write the policy first.