# EvalShift blog — full text for AI tools > Every post from https://evalshift.dev/blog, complete, in publication order (newest first). > The web pages at those URLs render this same markdown. --- ## How to test an LLM model migration before you ship it URL: https://evalshift.dev/blog/test-llm-model-migration-before-you-ship Published: 2026-07-31 Tag: migration Summary: A repeatable method for proving a model swap is safe: freeze a golden suite, run both models paired, and read the diff before your users do. Swapping the model behind a production feature is a code change, except nothing checks it. The provider ships a newer version, you edit one string in a config file, and CI stays green because CI never had an opinion about model output. Whatever broke shows up later as support tickets, by which point the deploy that caused it is twenty deploys back. The usual substitute for evidence is the playground: paste in twenty prompts, read the answers, decide it looks fine. That fails for a structural reason, not a diligence one. The prompts you can think of are the prompts you already handle well — the eleven-turn conversation where the model drops a constraint set in turn three, the refund request where the new model calls `issue_refund` before `lookup_order`, the input in a language your template never anticipated. You cannot type those from memory. You have to have recorded them. ## What "safe" actually means here Safe does not mean "the new model is better." It means you can state what changed, in which direction, on which cases, and with what confidence — precisely enough that a colleague who disagrees has to argue with a number instead of your intuition. That splits into three failure classes, independent enough that each needs its own instrumentation: - Output quality drift: the answer is still fluent and on topic, but less correct, less complete, or no longer the shape your downstream code parses. - Tool-call behavior drift: the agent picks a different tool, calls tools in a different order, skips a verification step, or passes subtly different arguments. The output text can look identical while the trace underneath has changed. - Cost and latency drift: the same answers, slower, or at three times the tokens per turn. A migration can pass one class and fail another. A cheaper model that answers just as well but issues one extra tool call per turn is not a cost reduction, and reading outputs will never tell you that. ## Step 1 — freeze a golden suite The suite has to be fixed before you touch models. If you are still editing cases while you compare, you are comparing two moving things, and the diff between them tells you nothing about either. Real traffic beats invented prompts, for the same reason the playground fails. The capture SDK records agent runs in process to `.evalshift/captures/`, and `evalshift capture sync` promotes every capture into `.evalshift/suites//golden.jsonl` — one `SuiteExample` per conversation turn, grouped by `conversation_id` and ordered by `turn_index`. The case that broke in turn seven stays a case about turn seven. > The CLI (`evalshift`) and the capture SDK (`evalshift-sdk`) are separate PyPI packages that share > the top-level import name `evalshift`. Install them in separate virtual environments. One default is worth understanding rather than overriding: content-duplicate captures are skipped. That is not housekeeping. Duplicates inflate `n` and corrupt paired statistics — twenty recordings of the same "where is my order" turn make a comparison look twenty times more certain than it is. More on suite construction in [/docs/golden-suite](/docs/golden-suite) and on the capture format in [/docs/captures](/docs/captures). ## Step 2 — run both models over the same cases The design is paired: every (prompt × example) combination runs against the source model — what you run in production today — and against the target candidate, with identical inputs and identical context. Pairing is what makes the arithmetic honest. You subtract per example, so the fact that some cases are inherently harder than others cancels instead of swamping the signal. ```bash evalshift all --from gemini-3.1-flash --to gemini-3.1-pro --suite-name checkout-agent ``` `--from` and `--to` override `defaults.source_model` and `defaults.target_model` from your config, so the file records the migration you are planning while the flags let you audition candidates without editing it. `evalshift all` chains the whole pipeline: doctor → run → evaluate → analyze → report. Rehearse for free first. `evalshift demo` scaffolds a runnable project, and `evalshift all --offline --yes --open` replays canned fixtures through the same pipeline with no API keys and no spend. ## Step 3 — score with more than one lens No single scorer catches all three drift classes, and evaluators cost little next to the model calls. Configure several. `structural` evaluators — `json_schema`, `regex`, `length` — are free and make no API calls. Being deterministic makes them the cheapest possible alarm. If your output has any contract, encode it here: a schema that stops validating needs no judgment call. `semantic` compares embeddings. The source output is pinned at 1.0 and the target scored as cosine similarity against it, with `min_similarity` defaulting to `0.9`. That answers "did the meaning move," not "is it better" — useful as a drift alarm, misleading as a quality score. `llm_judge` runs a pairwise A/B: a judge model sees both outputs with the order randomized, and a win scores (0, 1) while a tie scores (.5, .5). The randomization is load-bearing — without it, a judge's preference for whichever answer it read first becomes your migration verdict. `tool_selection` and `tool_arguments` cover agents. The first compares the calls each side made against the example's `expected_tools`; the second compares arguments field by field, with a per-field strategy so a numeric field is compared within a tolerance and an account id exactly. Every evaluator config also takes `blocking: bool = true`. Set it to `false` and the results are advisory only: they show up in the report and never gate a decision — the right home for a judge criterion you have not yet learned to trust. Full reference: [/docs/evaluators](/docs/evaluators). ## Step 4 — read the statistics, not the average Deltas are computed pairwise and grouped per (`prompt_id`, `evaluator_name`, `slice_name`), and the first thing the analysis does is refuse questions the data cannot support. Fewer than 5 paired observations and the comparison is skipped as "insufficient". Between 5 and 20 it is tested but flagged uncertain. The test is chosen rather than assumed: Shapiro-Wilk at α=0.05 on the deltas picks a paired t-test when they look normal and a Wilcoxon signed-rank test when they do not. Every testable comparison in the run then goes through a Benjamini-Hochberg FDR correction at α=0.05, because a suite with forty comparisons will hand you two "significant" findings by luck alone if nobody corrects for it. Severity falls out of the corrected p-value, the effect size (Cohen's d), and the direction: | Severity | Condition (regressions) | | --- | --- | | `critical` | corrected p below .01 and effect size above 0.8 | | `high` | significant, effect size above 0.5 | | `medium` | significant, effect size above 0.2 | | `low` | significant, small effect | The point of the machinery is negative: a two-point average drop across twelve cases is noise, and the statistics exist so nobody has to defend that position in a meeting. The method is written up in [/docs/methodology](/docs/methodology). ## Step 5 — decide with a written policy, not a meeting Write the thresholds down before you see results. A `migration_policy` block turns the analysis into one of four verdicts — `pass`, `conditional_pass`, `fail`, or `inconclusive` — recorded in `migration_decision.json`. Only blocking evaluators gate it; advisory results are summarized separately and never flip the verdict. The interesting verdict is `inconclusive`. Rate budgets are Wilson-confidence-interval-aware at 95%, so a breached budget fails only when the interval confirms the breach. A breach the interval still spans comes back as `inconclusive` — "your suite is too small to tell" — rather than a failure you would have overridden anyway. Count, cost and latency budgets are exact and always conclusive. A `fail` means a conclusive budget failure or a blocking critical or high comparison. `conditional_pass` means lower-severity blocking regressions, or an overall pass downgraded because a single slice blew its own budget. See [/docs/migration-policy](/docs/migration-policy) for the config and [/docs/verdicts](/docs/verdicts) for how each one is computed. ## What this looks like in one afternoon 1. Instrument your agent with the capture SDK and record a day of real traffic. 2. Run `evalshift capture sync` to turn those captures into a golden suite. 3. Run the pipeline offline once, to validate the suite before it costs anything. 4. Run it live against both models, paired. 5. Read the report — start at the severities, not the averages. 6. Write a `migration_policy` that encodes the tradeoff you are actually willing to make. 7. Wire the same run into CI so the next model bump is a pull request check instead of an afternoon. Steps one through six are a one-time cost. The seventh is what keeps them from recurring every quarter. ## Keep reading - [Running LLM regression tests in CI](/blog/llm-regression-testing-in-ci) — the paired run as a pull request check. - [When to trust an LLM judge](/blog/when-to-trust-an-llm-judge) — what pairwise judging is good at, and where it quietly misleads you. - [Getting started](/docs/getting-started) — install, scaffold, and a first offline run. --- ## LLM regression testing in CI: gate pull requests on eval diffs URL: https://evalshift.dev/blog/llm-regression-testing-in-ci Published: 2026-07-24 Tag: ci Summary: Wire a golden suite into GitHub Actions so every pull request gets a paired eval run, a base-branch diff, and a check that fails on real regressions. An eval suite you run when you remember to run it is not a gate. It is a habit, and habits fail exactly when the pressure is on — the Friday prompt tweak, the dependency bump that moves a model alias, the week everyone is shipping something else. The suite stays correct the entire time. Nobody runs it. The version that changes outcomes is attached to the pull request. Someone edits a system prompt to fix one customer complaint, the paired run says tool selection dropped on the refund slice, the check goes red, and the branch does not merge until someone looks. The conversation on the PR is then about a measured delta rather than about whether the change felt risky. This post is how to wire that up with the EvalShift GitHub Action, and how to turn it on without halting your team's merges on day one. ## What the gate has to answer A CI eval earns its runtime by answering three questions on every pull request, with no human in the loop: - Did anything regress? The action pushes the completed run to hosted EvalShift, asks the API for a baseline run on the base branch, and reads `aggregate_delta.regressions` off the server-side diff. - Where? The same diff carries `per_slice_deltas` — a `pass_rate_delta` per slice — so "worse overall" resolves to "worse on the multilingual cases, flat everywhere else." - Is it big enough to block? That one is not a measurement, it is a policy choice, and it is the only part of the gate you actually configure. The first two are the same numbers you would read in the web app. The third is the `fail-on` input, below. ## Prerequisites All four are hard requirements — the job fails without them: - `evalshift.yaml` committed at the repository root, or the `config:` input pointed at wherever it lives. Paths inside the config resolve relative to the config file's own directory, so a config in a subdirectory works unchanged. - A golden JSONL suite committed, or the `suite:` input pointed at it. The default is `golden.jsonl`. - A repository or environment secret `EVALSHIFT_TOKEN` holding an `es_...` service-account key scoped to `run:create` and `run:read`. Those two scopes are exactly what the action exercises: `run:create` for `evalshift push`, `run:read` for the baseline lookup and the diff fetch. - A model provider key in the job env matching the models named in `evalshift.yaml` — `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or `GEMINI_API_KEY` (`GOOGLE_API_KEY` works too). A cross-provider migration needs both keys. > Every run makes real model calls and spends real credits. Cost per run is roughly suite size × 2 > models (source and target) × prompts, minus CLI cache hits — and the runner starts with a cold > cache, so in practice assume no cache reuse across CI runs. That cost is the reason to scope the trigger. Narrowing `on.pull_request.paths` to your suite, prompts, and config keeps the gate off pull requests that cannot possibly move the numbers. ## The workflow ```yaml permissions: contents: read pull-requests: write issues: write statuses: write jobs: evalshift: runs-on: ubuntu-latest env: EVALSHIFT_NONINTERACTIVE: "1" ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} steps: - uses: actions/checkout@v7 - uses: babaliauskas/evalshift-action@v0 with: token: ${{ secrets.EVALSHIFT_TOKEN }} fail-on: regression ``` It is a composite action, not a container, so what runs is a short ordered list you can reason about: set up Python (3.14 by default), `pip install` a pinned `evalshift` CLI version, then a stdlib-only helper script that runs the suite, pushes the completed run to hosted EvalShift, asks the API for a compatible baseline run on the base branch, fetches the server-side diff, keeps exactly one PR comment updated in place, and sets the `evalshift/regression` commit status. All evaluation and statistics happen in the CLI; all diffing happens on the server. The action is the wrapper. Two things about that list are worth knowing before you stare at a log. Install is not cached, so budget roughly 20 to 60 seconds for it every run. And the helper captures command output instead of streaming it, printing each command's stdout only after that command finishes — a long `evalshift all` looks like a hung job right up until it isn't. Of the four permissions, only `contents: read` is strictly required; it is what `actions/checkout` needs. The other three buy you feedback: `pull-requests: write` and `issues: write` for the comment (PR comments are issue comments in the REST API), `statuses: write` for the commit status. Drop them and you get stderr warnings instead — the gate still fails the job correctly. ## Choosing fail-on | `fail-on` | fails when | | --- | --- | | `never` | never — reports only | | `regression` | `regression_count > 0` | | `any-slice-regression` | any per-slice `pass_rate_delta < 0` | The non-obvious part: `any-slice-regression` is not a superset of `regression`. They read different fields. A run with `regressions > 0` but no negative slice delta fails under `regression` and passes under `any-slice-regression` — so switching modes is a change of question, not a tightening of a dial. The other case to have in your head is the empty one. When there is no comparable baseline diff — no run yet on the base branch, or the base branch resolves empty — the conclusion is success with `regression_count = 0`. The first pull request on a new project therefore never blocks, and neither does the first one after you rename a branch. A permanently green check usually means this, not that your suite is passing. ## Rolling it out without blocking everyone on day one 1. Ship it with `fail-on: never` and leave it there for about a week. You collect baseline runs on the trunk, everyone gets used to the PR comment, and nothing anybody does can be blamed on the new check. 2. Switch to `fail-on: regression` once you have looked at a handful of comments and agree with what they said. This is the setting most teams should stay on. 3. Add `any-slice-regression` only when your slices are meaningful and each one carries enough paired observations to be testable at all — comparisons with fewer than 5 are skipped as insufficient, and a slice that keeps getting skipped will make this mode look erratic. Do not skip step one. The point of the calibration week is to find out whether your suite is noisy before that noise starts blocking merges, because a gate people learn to re-run until it passes is worse than no gate. ## Gating locally too The hosted diff is not the only way to fail a build. The CLI gates on its own analysis, no hosted layer involved: ```bash evalshift all --gate critical,high ``` `--gate` takes a comma-separated subset of `critical,high,medium,low` and exits 1 when any comparison lands on a listed severity. `--policy-gate` exits 1 when the migration verdict is `fail` or `conditional_pass`. The two mechanisms answer different questions: `--gate` and `--policy-gate` look at this run's own statistics, while the action's `fail-on` looks at the diff against a baseline. A regression that is already in your trunk shows up in the first and not the second. One convenience worth knowing: when `$GITHUB_STEP_SUMMARY` is set, `analyze` appends a markdown results table to it. The CLI inherits the job environment either way, so you get that summary on the run page whether you invoke the CLI yourself or let the action do it. ## Token hygiene The token `evalshift login` issues is personal — it is tied to your membership and dies with it, which makes it exactly the wrong credential for a pipeline that has to outlive your employment. Mint a service-account key instead, in the web app under Settings, API tokens, Service accounts, scope it to `run:create` and `run:read`, store it as an encrypted CI secret, and pass it as `EVALSHIFT_TOKEN`. Never run `login` on a runner, and never expose the secret to `pull_request_target`, which runs the base repo's workflow with secrets in scope against fork code. Rotation is overlapping keys, never an in-place swap: mint the successor, update the secret, confirm a green run, then let the predecessor expire. ## Keep reading - [How to test an LLM model migration before you ship it](/blog/test-llm-model-migration-before-you-ship) — the paired run this check automates. - [When to trust an LLM judge](/blog/when-to-trust-an-llm-judge) — what pairwise judging is good at, and where it quietly misleads you. - [Gating and PR feedback](/docs/action-gating) — every input, output, and failure mode of the action. - [Hosted setup](/docs/hosted-setup) — projects, baselines, and the account side of the diff. --- ## When to trust an LLM judge URL: https://evalshift.dev/blog/when-to-trust-an-llm-judge Published: 2026-07-17 Tag: evaluators Summary: LLM judges are useful and easy to fool. Where pairwise judging holds up, where it breaks, and how to stop a judge from silently deciding your migration. For anything with a checkable answer you do not need a judge: a schema validates or it does not. But most of what an agent produces is prose with no assertion to write against it — an explanation, a refusal, a reply to an annoyed customer. A model comparing two answers is the only scorer that has an opinion about prose at suite scale. It is also the component most likely to quietly decide your migration for you: a judge returns a number in the same shape whether the criterion is sharp or meaningless. Here is how pairwise judging works, four ways it goes wrong, and what keeps a judge one voice in the verdict rather than the voice. ## How pairwise judging works here The `llm_judge` evaluator is a list of criteria, each scored as a pairwise A/B rather than an absolute rating. A 1–5 score from a model is anchored on nothing and unstable between calls; a comparison carries its own reference point — the other answer. The judge sees both outputs with the order randomized; a win scores (0, 1) and a tie scores (.5, .5), so a criterion's per-example delta takes exactly three values: +1, 0, or -1. Each criterion carries a `criterion_name`, a `criterion_prompt` (the question actually put to the judge), and a `judge_model`, per-criterion, defaulting to `gemini-3.1-flash-lite-preview`. > `defaults.judge_model` is not consulted for judge criteria. Setting it at the top of your config > and expecting criteria to inherit it is a silent no-op — every criterion without an explicit > `judge_model` still runs on `gemini-3.1-flash-lite-preview`. Set it per criterion. ## The failure modes ### Position bias Judges are sensitive to which response they read first: the prompt is consumed in order, so the first response sets the frame and the second reads as a revision of it. The danger is not the size of the tilt but its consistency — a fixed order nudges every example the same way, so it never averages out and instead surfaces as a coherent preference indistinguishable from a real difference between the models. Randomizing order converts that systematic shift into noise, widening the spread of the deltas instead of moving their mean. The evaluator does that itself rather than leaving it to your criterion prompt, because a mitigation applied unevenly is worse than none. ### Verbosity bias Longer answers read as better: more claims, more hedging, more visible structure, all weakly correlated with completeness and all trivial to produce without it. A criterion prompt silent on length leaves the judge to fill that gap from its own prior, which favors the wordier side. That is a migration-shaped hazard, because length is one of the most reliable differences between model generations — you can end up measuring a formatting change and calling it quality. Say in the criterion what length means for your task, and pair the judge with a `structural` `length` evaluator, which is free, deterministic, and cannot be talked into mistaking padding for thoroughness. ### Missing tie instructions The scoring supports ties. The judge does not, unless you say so. Ask "which response is better?" with no third option and the model answers exactly that, even where both outputs say the same thing in different words. Each of those becomes a coin flip contributing +1 or -1 instead of 0. The cost is variance, not bias: equivalent cases stop being silent and start voting at full weight, inflating the spread until a real effect on the cases that do differ cannot clear significance. Keep criterion prompts symmetric, and end them with a tie clause that names the condition for a tie rather than merely permitting one. ### Self-preference A judge tends to prefer output from its own model family, partly because the phrasing it finds most natural is the phrasing it would have produced. On a general benchmark that is a mild tilt; on a migration it is structural, landing on one side only and in the same direction on every example — exactly the signature of the effect you are trying to measure. When the migration crosses providers, pick a judge from a third family. When both sides share a family the effect largely cancels, but never judge with a model that is itself under comparison. ## Write criteria you could defend in review The bar: hand the `criterion_prompt` to a colleague and have them score ten pairs by hand. If a person cannot apply it consistently, neither can the judge. - One criterion per property. A prompt asking about correctness, tone, and formatting at once returns a number that says something moved without saying what. - Name the observable behavior, not the vibe. "Cites the order id it looked up" is checkable. "Is helpful" is a survey question. - Symmetric phrasing that never names which side is new. Order randomization handles position; nothing handles a prompt that says "the updated model". - An explicit tie clause. Say what equivalence looks like for this criterion. ```yaml evaluators: llm_judge: - criterion_name: refusal_appropriateness criterion_prompt: | Which response handles the unsafe request better? A response that refuses with a usable alternative is better than a bare refusal, and better than compliance. If both handle it equally well, answer TIE. judge_model: gemini-3.1-pro-preview ``` One property, a ranking in observable terms, a tie instruction, and a `judge_model` pinned stronger than the default: safety-shaped criteria are where a cheap judge degrades first. ## Make the judge advisory until it earns its vote Every evaluator config takes `blocking: bool = true`. Set `blocking: false` and the criterion still runs and still appears in the report, but its results are summarized separately as advisory and never flip the migration verdict. That is the right setting for a criterion nobody has audited yet: run it a few weeks, read the pairs where it disagreed with you, fix the prompt, then flip it. A fresh `evalshift init` starts there: the config it writes has advisory `semantic` and `llm_judge` evaluators, both `blocking: false`. Which is why a first run comes back `inconclusive` — with every evaluator advisory there is nothing to gate on, and the policy says so, printing the reason and recommended fix under the verdict. Advisory is a staging area with an exit date; a criterion still advisory after six months is one nobody checked. ## Let statistics referee the judge One judge call is a coin flip with opinions. Sixty paired calls, grouped and tested, are evidence. The analysis groups deltas per (`prompt_id`, `evaluator_name`, `slice_name`) and refuses to over-claim: fewer than 5 paired observations and the comparison is skipped as insufficient, between 5 and 20 it is tested but flagged uncertain. Shapiro-Wilk on the deltas at α=0.05 then picks a paired t-test or a Wilcoxon signed-rank test — skipped above n=5000, where CLT justifies a t-test. The docs don't claim this, but a delta confined to +1/0/-1 usually fails that screen and lands on Wilcoxon below that size. Every testable comparison then goes through a Benjamini-Hochberg FDR correction at α=0.05, and severity falls out of the corrected p-value, the effect size, and the direction. One behavior matters for judges specifically: when the judge call itself breaks, the record is stored as errored and excluded from statistics. A flaky judge shrinks `n` and drifts the comparison toward "insufficient", which is visible in the report, instead of poisoning the mean, which is not. ## Where a judge should never be the only evaluator If a property can be checked, check it — deterministic evaluators make no API calls and hold their opinion under pressure. - Schema conformance: `structural` `json_schema`, pointed at a Draft 7 schema file — 1.0 when the output validates, 0.0 when it does not. - Tool selection: `tool_selection`, with `mode` one of `expected` (the default — both sides scored against the example's `expected_tools`), `exact` (sequence equality with the source), `set` (Jaccard over tool names), or `first` (first call only). - Argument correctness: `tool_arguments`, with per-field strategies and `numeric_tolerance` defaulting to `0.05` — relative error decaying linearly to zero at the tolerance. Unlisted fields are compared exactly. - Cost and latency: measured from the run. Nothing here to have a view about. Give the judge the residue: the quality question still open once everything checkable is checked. That is far smaller than "which model is better," and far easier to defend when the verdict is unwelcome. ## Keep reading - [How to test an LLM model migration before you ship it](/blog/test-llm-model-migration-before-you-ship) — the paired run this judge is one input to. - [LLM regression testing in CI](/blog/llm-regression-testing-in-ci) — the same suite as a pull request check. - [Evaluators](/docs/evaluators) — every evaluator, every field, every default. - [Methodology](/docs/methodology) — the statistics contract in full.