How to test an LLM model migration before you ship it

Swapping the model behind a production feature is a code change, except nothing checks it. The provider ships a newer version, you edit one string in a config file, and CI stays green because CI never had an opinion about model output. Whatever broke shows up later as support tickets, by which point the deploy that caused it is twenty deploys back.

The usual substitute for evidence is the playground: paste in twenty prompts, read the answers, decide it looks fine. That fails for a structural reason, not a diligence one. The prompts you can think of are the prompts you already handle well — the eleven-turn conversation where the model drops a constraint set in turn three, the refund request where the new model calls issue_refund before lookup_order, the input in a language your template never anticipated. You cannot type those from memory. You have to have recorded them.

## What "safe" actually means here

Safe does not mean "the new model is better." It means you can state what changed, in which direction, on which cases, and with what confidence — precisely enough that a colleague who disagrees has to argue with a number instead of your intuition.

That splits into three failure classes, independent enough that each needs its own instrumentation:

• +Output quality drift: the answer is still fluent and on topic, but less correct, less complete, or no longer the shape your downstream code parses.
• +Tool-call behavior drift: the agent picks a different tool, calls tools in a different order, skips a verification step, or passes subtly different arguments. The output text can look identical while the trace underneath has changed.
• +Cost and latency drift: the same answers, slower, or at three times the tokens per turn.

A migration can pass one class and fail another. A cheaper model that answers just as well but issues one extra tool call per turn is not a cost reduction, and reading outputs will never tell you that.

## Step 1 — freeze a golden suite

The suite has to be fixed before you touch models. If you are still editing cases while you compare, you are comparing two moving things, and the diff between them tells you nothing about either.

Real traffic beats invented prompts, for the same reason the playground fails. The capture SDK records agent runs in process to .evalshift/captures/, and evalshift capture sync promotes every capture into .evalshift/suites/<suite>/golden.jsonl — one SuiteExample per conversation turn, grouped by conversation_id and ordered by turn_index. The case that broke in turn seven stays a case about turn seven.

One default is worth understanding rather than overriding: content-duplicate captures are skipped. That is not housekeeping. Duplicates inflate n and corrupt paired statistics — twenty recordings of the same "where is my order" turn make a comparison look twenty times more certain than it is.

More on suite construction in /docs/golden-suite and on the capture format in /docs/captures.

## Step 2 — run both models over the same cases

The design is paired: every (prompt × example) combination runs against the source model — what you run in production today — and against the target candidate, with identical inputs and identical context. Pairing is what makes the arithmetic honest. You subtract per example, so the fact that some cases are inherently harder than others cancels instead of swamping the signal.

evalshift all --from gemini-3.1-flash --to gemini-3.1-pro --suite-name checkout-agent

--from and --to override defaults.source_model and defaults.target_model from your config, so the file records the migration you are planning while the flags let you audition candidates without editing it. evalshift all chains the whole pipeline: doctor → run → evaluate → analyze → report.

Rehearse for free first. evalshift demo scaffolds a runnable project, and evalshift all --offline --yes --open replays canned fixtures through the same pipeline with no API keys and no spend.

## Step 3 — score with more than one lens

No single scorer catches all three drift classes, and evaluators cost little next to the model calls. Configure several.

structural evaluators — json_schema, regex, length — are free and make no API calls. Being deterministic makes them the cheapest possible alarm. If your output has any contract, encode it here: a schema that stops validating needs no judgment call.

semantic compares embeddings. The source output is pinned at 1.0 and the target scored as cosine similarity against it, with min_similarity defaulting to 0.9. That answers "did the meaning move," not "is it better" — useful as a drift alarm, misleading as a quality score.

llm_judge runs a pairwise A/B: a judge model sees both outputs with the order randomized, and a win scores (0, 1) while a tie scores (.5, .5). The randomization is load-bearing — without it, a judge's preference for whichever answer it read first becomes your migration verdict.

tool_selection and tool_arguments cover agents. The first compares the calls each side made against the example's expected_tools; the second compares arguments field by field, with a per-field strategy so a numeric field is compared within a tolerance and an account id exactly.

Every evaluator config also takes blocking: bool = true. Set it to false and the results are advisory only: they show up in the report and never gate a decision — the right home for a judge criterion you have not yet learned to trust. Full reference: /docs/evaluators.

## Step 4 — read the statistics, not the average

Deltas are computed pairwise and grouped per (prompt_id, evaluator_name, slice_name), and the first thing the analysis does is refuse questions the data cannot support. Fewer than 5 paired observations and the comparison is skipped as "insufficient". Between 5 and 20 it is tested but flagged uncertain.

The test is chosen rather than assumed: Shapiro-Wilk at α=0.05 on the deltas picks a paired t-test when they look normal and a Wilcoxon signed-rank test when they do not. Every testable comparison in the run then goes through a Benjamini-Hochberg FDR correction at α=0.05, because a suite with forty comparisons will hand you two "significant" findings by luck alone if nobody corrects for it.

Severity falls out of the corrected p-value, the effect size (Cohen's d), and the direction:

The point of the machinery is negative: a two-point average drop across twelve cases is noise, and the statistics exist so nobody has to defend that position in a meeting. The method is written up in /docs/methodology.

## Step 5 — decide with a written policy, not a meeting

Write the thresholds down before you see results. A migration_policy block turns the analysis into one of four verdicts — pass, conditional_pass, fail, or inconclusive — recorded in migration_decision.json. Only blocking evaluators gate it; advisory results are summarized separately and never flip the verdict.

The interesting verdict is inconclusive. Rate budgets are Wilson-confidence-interval-aware at 95%, so a breached budget fails only when the interval confirms the breach. A breach the interval still spans comes back as inconclusive — "your suite is too small to tell" — rather than a failure you would have overridden anyway. Count, cost and latency budgets are exact and always conclusive.

A fail means a conclusive budget failure or a blocking critical or high comparison. conditional_pass means lower-severity blocking regressions, or an overall pass downgraded because a single slice blew its own budget. See /docs/migration-policy for the config and /docs/verdicts for how each one is computed.

## What this looks like in one afternoon

1. +Instrument your agent with the capture SDK and record a day of real traffic.
2. +Run evalshift capture sync to turn those captures into a golden suite.
3. +Run the pipeline offline once, to validate the suite before it costs anything.
4. +Run it live against both models, paired.
5. +Read the report — start at the severities, not the averages.
6. +Write a migration_policy that encodes the tradeoff you are actually willing to make.
7. +Wire the same run into CI so the next model bump is a pull request check instead of an afternoon.

Steps one through six are a one-time cost. The seventh is what keeps them from recurring every quarter.

## Keep reading

• +Running LLM regression tests in CI — the paired run as a pull request check.
• +When to trust an LLM judge — what pairwise judging is good at, and where it quietly misleads you.
• +Getting started — install, scaffold, and a first offline run.