## This one passed: 71% cheaper, and the agent still does the same thing URL: https://www.evalshift.dev/blog/what-a-passing-migration-proves Published: 2026-09-16 Tag: migration Summary: An EvalShift migration report that said PASS: a 71% cheaper model inside every policy budget. Where each threshold came from, and what PASS does not claim. Takeaways: - EvalShift replayed 120 captured examples against the source and candidate model and returned PASS: 71.2% cheaper, 61.9% faster, all ten policy budgets held. - EvalShift's migration_policy in evalshift.yaml holds seven budgets, and a slices map tightens any of them per slice — here security got zero regressions and zero tool divergence, refund got zero argument drift. - Check every rate budget against the suite size first: a rate over 108 rows moves in steps of 0.93%, so a 1% budget is zero tolerance in disguise, and EvalShift warns when a budget is sub-granular. - EvalShift judges proportion budgets with a 95% Wilson interval, asymmetrically: a held budget is conclusive however wide the interval, and only a breach needs the interval to confirm it. - PASS does not mean nothing changed: 16 comparisons regressed and 28 improved, and EvalShift's per-example diff shows each one. It means every change stayed inside limits fixed before anyone saw a number. The previous post was about a migration that was cheaper, faster and still failed. This is the other outcome. It is the more common one once a suite is in decent shape, and I see it written up far less often, because a pass is boring. It shouldn't be. A pass is only worth something if the limits were fixed before the run. So this post is the whole thing: the policy, the reasoning behind every number in it, and the EvalShift report it produced. For anyone arriving here cold: EvalShift is a local-first CLI for testing LLM model migrations. It replays a frozen golden suite against your current model and a candidate, scores every pair of outputs with tool-call, structural, semantic and LLM-judge evaluators, runs paired statistics over the deltas, and turns a `migration_policy` block you wrote in `evalshift.yaml` into one of four verdicts: `pass`, `conditional_pass`, `fail` or `inconclusive`. It writes a single-file `report.html` on your machine; nothing is uploaded unless you run `evalshift push`. Every figure in this post is a panel from that report. The candidate: a customer-support agent with six tools (`lookup_customer`, `lookup_order`, `check_refund_policy`, `issue_refund`, `escalate_to_human`, `search_kb`), moving from `gemini-3.7-pro` to `gemini-3.7-flash`. The suite: 120 examples recorded in production by the EvalShift capture SDK, promoted into a golden JSONL suite with `evalshift capture sync`, and sliced by tag according to what the conversation was about. | Slice | Examples | What is in it | | --- | --- | --- | | `routine` | 42 | order status, shipping, account questions | | `refund` | 26 | refund and return requests | | `security` | 24 | account access, password and payment-method changes | | `customer_lookup` | 16 | requests that need a customer record first | | `text_only` | 12 | greetings, thanks, off-topic | One `evalshift compare` command, real API calls on both sides, and the report opened on: ```report { "kind": "verdict", "verdict": "pass", "summary": "10 of 10 budgets within policy.", "rates": { "equivalent": 90.1, "improved": 6.3, "regressed": 3.6 }, "cards": [ { "eyebrow": "Advisory signal", "value": "3.6%", "unit": "regression rate", "tone": "ok", "note": "Below the max_overall_regression_rate of 5%. 16 of 444 scored comparisons, none above medium severity." }, { "eyebrow": "Economics", "value": "-71.2%", "unit": "cost", "tone": "ok", "note": "$1.9248 → $0.5544. Latency -61.9%. Both inside +0% cost / +30% latency." } ], "caption": "EvalShift's verdict card. Ten budgets, all held; the regression rate and the economics sit beside it so nobody has to scroll to find out what the pass cost." } ``` ```report { "kind": "strip", "cells": [ { "label": "Examples", "value": "120" }, { "label": "Calls", "value": "240", "note": "0 cached" }, { "label": "Failed / truncated", "value": "0 / 0", "tone": "ok" }, { "label": "Total cost", "value": "$2.4792" }, { "label": "Latency Δ", "value": "-61.9%", "tone": "ok" }, { "label": "Avg score Δ", "value": "+0.011", "tone": "ok" } ], "caption": "The run strip. Same 120 examples on both sides, nothing failed or truncated, so every comparison below is over the full suite. EvalShift excludes truncated and errored calls from the statistics, so this row is worth checking first." } ``` Cheaper by 71.2%, faster by 61.9%, verdict PASS. The economics look like the last post's. The difference is everything that was decided before the run. ## The numbers were written before the run EvalShift's `migration_policy` is a block in `evalshift.yaml` with seven budgets: overall regression rate, critical regression count, equivalent-or-better rate, tool-argument drift, tool-selection divergence, cost increase and latency increase. A `slices` map under it lets any slice override any budget, inheriting the top-level value where it doesn't. Evaluators are configured per suite; budgets are set once and tightened per slice. `evalshift init --profile cost-reduction` scaffolds a starting policy: 2% overall regression rate, zero critical regressions, 97% equivalence, 1% tool-argument drift, 5% cost increase, 30% latency increase. That is a starting point, not a decision. This is what the project actually ran with: ```yaml migration_policy: max_overall_regression_rate: 0.05 max_critical_regressions: 0 min_equivalence_rate: 0.90 max_tool_argument_drift: 0.10 max_tool_divergence: 0.05 max_cost_increase: 0.0 max_latency_increase: 0.30 slices: security: max_overall_regression_rate: 0.0 max_tool_divergence: 0.0 refund: max_tool_argument_drift: 0.0 ``` Three things moved between the profile and this file. ### Check every rate against the suite size A rate over *n* rows can only move in steps of 1/*n*. This suite has 108 tool-argument rows, so one drifted row is 0.93%, and a 1% budget is a zero-tolerance budget wearing a percentage. EvalShift prints a recommendation when a budget is below the granularity of its denominator, naming the budget, the value and the row count, and this one would have triggered it. So the rule I use: if I mean zero, I write `0.0`. Where I don't, I set a number the sample can resolve. 10% argument drift on 108 rows is ten reworded search queries, which is what drift on this agent looks like in practice. ### Tolerance where wording lives, zero where money and access live The overall regression rate went *up*, from 2% to 5%. The pairwise LLM judge is blocking on this project, and a judge flips on rewording; 2% of 444 comparisons is nine judge calls deciding the migration. 5% leaves room for the judge to disagree about phrasing without anything real being allowed through. The strictness moved into the slices instead. `security` gets zero regressions and zero tool-selection divergence: a model that starts routing an account-access request to a different tool does not get a percentage. `refund` gets zero argument drift: an order id or an amount that drifts is a wrong refund, not a reworded one. The slices that are *not* in the policy matter too. `customer_lookup` has 16 examples. EvalShift tests a comparison below 20 pairs but flags it uncertain, so that slice inherits the top-level budgets and gets no tighter ones. It is the slice I would grow before I tightened it. ### The cost budget is the reason for the migration `max_cost_increase: 0.0`. The point of the exercise is to spend less. A candidate that costs more has failed before any quality number is read, and a policy should say so instead of leaving it to whoever reads the economics card. EvalShift measures cost and latency from the run's own calls, so these two budgets gate even when no quality evaluator does. Two evaluator decisions sit alongside the budgets. Every EvalShift evaluator carries a `blocking` flag; advisory (`blocking: false`) results are reported and ranked but never change the verdict. The judge is `blocking: true` here; `init` writes `false` because at a dozen examples judge noise would decide the verdict, and at 120 with an audited criterion it earns its vote. `semantic` stays advisory. It measures wording, and wording is the one thing this migration was allowed to change. ## What the run measured ```report { "kind": "budgets", "rows": [ { "name": "Overall regression rate", "id": "max_overall_regression_rate", "scope": "overall", "observed": "3.6%", "limit": "≤ 5.0%", "tone": "ok", "note": "16 of 444 · 95% CI 2.2–5.8%" }, { "name": "Critical regressions", "id": "max_critical_regressions", "scope": "overall", "observed": "0", "limit": "≤ 0", "tone": "ok", "note": "of 444" }, { "name": "Equivalent-or-better rate", "id": "min_equivalence_rate", "scope": "overall", "observed": "96.4%", "limit": "≥ 90.0%", "tone": "ok", "note": "428 of 444" }, { "name": "Tool-argument drift", "id": "max_tool_argument_drift", "scope": "overall", "observed": "4.6%", "limit": "≤ 10.0%", "tone": "ok", "note": "5 of 108 tool-argument rows" }, { "name": "Tool-selection divergence", "id": "max_tool_divergence", "scope": "overall", "observed": "2.8%", "limit": "≤ 5.0%", "tone": "ok", "note": "3 of 108 divergence rows" }, { "name": "Cost increase", "id": "max_cost_increase", "scope": "overall", "observed": "0.0%", "limit": "≤ 0.0%", "tone": "ok", "note": "cost fell 71.2%" }, { "name": "Latency increase", "id": "max_latency_increase", "scope": "overall", "observed": "0.0%", "limit": "≤ 30.0%", "tone": "ok", "note": "latency fell 61.9%" }, { "name": "Overall regression rate", "id": "max_overall_regression_rate", "scope": "security", "observed": "0.0%", "limit": "≤ 0.0%", "tone": "ok", "note": "0 of 96" }, { "name": "Tool-selection divergence", "id": "max_tool_divergence", "scope": "security", "observed": "0.0%", "limit": "≤ 0.0%", "tone": "ok", "note": "0 of 24" }, { "name": "Tool-argument drift", "id": "max_tool_argument_drift", "scope": "refund", "observed": "0.0%", "limit": "≤ 0.0%", "tone": "ok", "note": "0 of 26" } ], "caption": "Every budget against its limit, as EvalShift reports them. Seven at the top level, three on the two slices where a regression is a wrong action rather than a reworded one." } ``` The first row is the one worth reading twice. 16 of 444 is 3.6%, and its 95% Wilson interval runs from 2.2% to 5.8%. The interval crosses the limit. EvalShift's rule is asymmetric on purpose. A breach fails only when the interval confirms it; a breach the interval cannot confirm returns `inconclusive`, because the suite was too small to say. A budget the observation held is conclusive however wide its interval, because a wide interval must never downgrade a clean run. This budget held, so it passes, and the interval is printed so the reader knows how much room there was. The three slice budgets held at exactly zero. Zero on 96 comparisons means something. Zero on 12 would not, which is why `text_only` has no slice budget at all. ## By evaluator Four evaluators scored this run. EvalShift's `tool_selection` evaluator reads the recorded traces and scores two axes: conformance, where each side is graded against the suite's recorded tool calls, and divergence, where the target is graded against what the source did. `tool_arguments` scores argument values field by field against the expected call. The `llm_judge` is pairwise and sees the two outputs as anonymous A and B. `semantic` is embedding similarity between the two outputs. The first three are blocking; the last is advisory. ```report { "kind": "evaluators", "rows": [ { "name": "Routing — conformance", "id": "routing · tool_selection.conformance", "axis": "each side graded against the suite's recorded tool calls", "n": 108, "delta": "+0.046", "effect": "0.28", "magnitude": "small", "ci": "[0.09, 0.47]", "confidence": "likely", "severity": "improved" }, { "name": "Routing — divergence", "id": "routing · tool_selection.divergence", "axis": "the target graded against what the source did", "n": 108, "delta": "-0.028", "effect": "0.17", "magnitude": "negligible", "ci": "[-0.36, 0.02]", "confidence": "unclear", "severity": "none" }, { "name": "Routing args", "id": "routing_args", "n": 108, "delta": "-0.004", "effect": "0.03", "magnitude": "negligible", "ci": "[-0.22, 0.16]", "confidence": "unclear", "severity": "none" }, { "name": "LLM judge: equivalence", "id": "llm_judge.equivalence", "n": 120, "delta": "+0.029", "effect": "0.12", "magnitude": "negligible", "ci": "[-0.06, 0.30]", "confidence": "unclear", "severity": "none" }, { "name": "Semantic similarity", "id": "semantic", "advisory": true, "n": 120, "delta": "-0.041", "effect": "0.44", "magnitude": "small", "ci": "[-0.62, -0.26]", "confidence": "likely", "severity": "medium", "blurb": "Reported, not gating: blocking is false." } ], "caption": "Overall, by evaluator. Each row is a paired test over that evaluator's deltas: effect size with a 95% interval, and a confidence label from the Benjamini-Hochberg corrected p-value. Four blocking rows say equivalent or improved; the one regression is on the advisory evaluator that measures wording." } ``` Four blocking rows say equivalent or improved. The one that regressed is advisory, and it measures the one thing this migration was allowed to change. Had `semantic` been blocking, the same run would have come back `conditional_pass` on a medium-severity regression in phrasing. That is not a reason to make every evaluator advisory. It is a reason to decide, per evaluator, whether what it measures is something you are willing to block a migration on. The conformance row says the candidate matched the suite's recorded tool calls *more often* than the model that produced them: 11 examples improved, one regressed. Most of the eleven were refund requests where the source went straight to `issue_refund`. ## The diffs I still read A pass is not permission to skip the diff. For every flagged example, the EvalShift report shows the reason it was flagged, the tool calls on each side, the argument-level diff, and the conversation context that led to it. Two examples from this run, one from each side of the ledger. ```report { "kind": "example", "id": "cap_6b1e40f2a9c34d0b8e7d2a5f31c9e804", "turn": 1, "what": "Routing — conformance", "delta": "+0.500", "tone": "ok", "why": { "scores": "source 0.50 → target 1.00 (0–1)", "label": "IMPROVED", "text": "The suite expected check_refund_policy before issue_refund. The source went straight to issue_refund; the target called both, in order." }, "tools": { "source": "issue_refund", "target": "check_refund_policy, issue_refund" }, "source": { "score": "0.500", "tone": "bad", "calls": [ { "tool": "issue_refund", "args": "{\"order_id\": \"ord_58213\", \"amount\": 42.0, \"reason\": \"damaged on arrival\"}" } ], "final": "Done — I've refunded $42.00 to your original payment method. You'll see it in 3–5 business days." }, "target": { "score": "1.000", "tone": "ok", "calls": [ { "tool": "check_refund_policy", "args": "{\"order_id\": \"ord_58213\"}", "mark": "extra" }, { "tool": "issue_refund", "args": "{\"order_id\": \"ord_58213\", \"amount\": 42.0, \"reason\": \"damaged on arrival\"}" } ], "final": "Your order is inside the 30-day window, so I've refunded $42.00 to your original payment method. Expect it in 3–5 business days." }, "diffs": [ "extra tool: Target added check_refund_policy at position 1.", "matched: issue_refund arguments are identical on both sides." ], "caption": "An improvement, as EvalShift's per-example diff shows it. The recorded ground truth expected the policy check first; the source model that produced the traffic skipped it and the candidate did not." } ``` The refund went out either way, and the final text on both sides would pass any output check you care to write. The trace is where the difference lives: one side verified before acting and the other did not. This is the class of change a text evaluator cannot see in either direction, and the reason EvalShift's tool-call evaluators score the trace rather than the prose. ```report { "kind": "example", "id": "cap_9d27c5e1b04a4f6e9a3c7d8e2f105b6c", "turn": 0, "what": "Routing args", "delta": "-0.160", "tone": "bad", "why": { "scores": "source 1.00 → target 0.84 (0–1)", "label": "ARGUMENT_VALUE_DRIFT", "text": "search_kb.query scored 0.84 against the expected value under the semantic strategy, below the 0.9 drift floor. Same tool, same intent, different wording." }, "tools": { "source": "search_kb", "target": "search_kb" }, "source": { "score": "1.000", "tone": "ok", "calls": [ { "tool": "search_kb", "args": "{\"query\": \"change shipping address after order placed\"}" } ], "final": "You can change the address until the order ships — open the order and tap “Edit address”. Once it's marked shipped, reply here and I'll reroute it with the carrier." }, "target": { "score": "0.840", "tone": "bad", "calls": [ { "tool": "search_kb", "args": "{\"query\": \"edit delivery address existing order\"}" } ], "final": "Until the order ships you can edit the address yourself from the order page. If it has already shipped, let me know and I'll set up a carrier reroute." }, "diffs": [ "argument drift: search_kb.query scored 0.84 against the expected query (drift floor 0.9).", "same tool set: no calls added or removed." ], "caption": "A regression the budget was built to tolerate. Five of 108 tool-argument rows drifted like this one; the limit was ten." } ``` A reviewer would call those the same query. The evaluator scored it 0.84 against a 0.9 floor and counted it as drift, five rows out of 108 did the same, and the budget said five is fine. That is what the 10% was for. The same five rows in the `refund` slice would have failed the run, and that was also decided in advance. ## What EvalShift did in this run The whole run, as a list of the product's parts, in the order they were used: - **The capture SDK** recorded the agent's real conversations in production, tool calls included, and `evalshift capture sync` promoted them into a frozen golden JSONL suite with the tool evaluators written from what the captures actually contained. - **`evalshift compare`** replayed every example against the source and the target model, paired per example, with the same inputs, tools and context on both sides. - **The tool-call evaluators** (`tool_selection`, `tool_arguments`) scored the traces, the **pairwise LLM judge** scored the outputs, and **`semantic`** measured drift in wording, advisory only. - **Paired statistics** turned each evaluator's deltas into an effect size, a 95% interval and a corrected confidence label, so a two-point average drop and a real regression are told apart mechanically. - **`migration_policy`** in `evalshift.yaml` held seven budgets, three of them tightened to zero on the `security` and `refund` slices, and every proportion budget was judged with a Wilson interval that can return `inconclusive` instead of a false fail. - **`report.html`** was written locally, verdict first, with a per-example diff for everything flagged. Nothing left the machine. - **`--policy-gate`** made the verdict an exit code, which is what the EvalShift GitHub Action uses to block a pull request on the same policy once the suite runs in CI. ## What a pass proves Not that nothing changed. 16 comparisons regressed and 28 improved; the agent phrases search queries differently and checks the refund policy more often than it used to. It proves that every change stayed inside limits that were written down before anyone saw a number, on a suite that was frozen before the run. That is the entire claim, and it is enough to act on, because there is nothing left to negotiate: the argument about what counts as acceptable happened in the YAML, not in the meeting after the report. What happens next is the boring part, which is the point. The model string changes in production. The suite does not. It runs again on the next pull request through the EvalShift GitHub Action, gated on the same policy, against the new baseline. ```bash evalshift compare --suite-name support_routing --to gemini-3.7-flash --policy-gate --open ``` If the previous post was the reason to run the comparison, this one is the reason to write the policy first.