LLM regression testing in CI: gate pull requests on eval diffs

An eval suite you run when you remember to run it is not a gate. It is a habit, and habits fail exactly when the pressure is on — the Friday prompt tweak, the dependency bump that moves a model alias, the week everyone is shipping something else. The suite stays correct the entire time. Nobody runs it.

The version that changes outcomes is attached to the pull request. Someone edits a system prompt to fix one customer complaint, the paired run says tool selection dropped on the refund slice, the check goes red, and the branch does not merge until someone looks. The conversation on the PR is then about a measured delta rather than about whether the change felt risky. This post is how to wire that up with the EvalShift GitHub Action, and how to turn it on without halting your team's merges on day one.

## What the gate has to answer

A CI eval earns its runtime by answering three questions on every pull request, with no human in the loop:

• +Did anything regress? The action pushes the completed run to hosted EvalShift, asks the API for a baseline run on the base branch, and reads aggregate_delta.regressions off the server-side diff.
• +Where? The same diff carries per_slice_deltas — a pass_rate_delta per slice — so "worse overall" resolves to "worse on the multilingual cases, flat everywhere else."
• +Is it big enough to block? That one is not a measurement, it is a policy choice, and it is the only part of the gate you actually configure.

The first two are the same numbers you would read in the web app. The third is the fail-on input, below.

## Prerequisites

All four are hard requirements — the job fails without them:

• +evalshift.yaml committed at the repository root, or the config: input pointed at wherever it lives. Paths inside the config resolve relative to the config file's own directory, so a config in a subdirectory works unchanged.
• +A golden JSONL suite committed, or the suite: input pointed at it. The default is golden.jsonl.
• +A repository or environment secret EVALSHIFT_TOKEN holding an es_... service-account key scoped to run:create and run:read. Those two scopes are exactly what the action exercises: run:create for evalshift push, run:read for the baseline lookup and the diff fetch.
• +A model provider key in the job env matching the models named in evalshift.yaml — ANTHROPIC_API_KEY, OPENAI_API_KEY, or GEMINI_API_KEY (GOOGLE_API_KEY works too). A cross-provider migration needs both keys.

That cost is the reason to scope the trigger. Narrowing on.pull_request.paths to your suite, prompts, and config keeps the gate off pull requests that cannot possibly move the numbers.

## The workflow

permissions:
  contents: read
  pull-requests: write
  issues: write
  statuses: write
jobs:
  evalshift:
    runs-on: ubuntu-latest
    env:
      EVALSHIFT_NONINTERACTIVE: "1"
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    steps:
      - uses: actions/checkout@v7
      - uses: babaliauskas/evalshift-action@v0
        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
          fail-on: regression

It is a composite action, not a container, so what runs is a short ordered list you can reason about: set up Python (3.14 by default), pip install a pinned evalshift CLI version, then a stdlib-only helper script that runs the suite, pushes the completed run to hosted EvalShift, asks the API for a compatible baseline run on the base branch, fetches the server-side diff, keeps exactly one PR comment updated in place, and sets the evalshift/regression commit status. All evaluation and statistics happen in the CLI; all diffing happens on the server. The action is the wrapper.

Two things about that list are worth knowing before you stare at a log. Install is not cached, so budget roughly 20 to 60 seconds for it every run. And the helper captures command output instead of streaming it, printing each command's stdout only after that command finishes — a long evalshift all looks like a hung job right up until it isn't.

Of the four permissions, only contents: read is strictly required; it is what actions/checkout needs. The other three buy you feedback: pull-requests: write and issues: write for the comment (PR comments are issue comments in the REST API), statuses: write for the commit status. Drop them and you get stderr warnings instead — the gate still fails the job correctly.

## Choosing fail-on

The non-obvious part: any-slice-regression is not a superset of regression. They read different fields. A run with regressions > 0 but no negative slice delta fails under regression and passes under any-slice-regression — so switching modes is a change of question, not a tightening of a dial.

The other case to have in your head is the empty one. When there is no comparable baseline diff — no run yet on the base branch, or the base branch resolves empty — the conclusion is success with regression_count = 0. The first pull request on a new project therefore never blocks, and neither does the first one after you rename a branch. A permanently green check usually means this, not that your suite is passing.

## Rolling it out without blocking everyone on day one

1. +Ship it with fail-on: never and leave it there for about a week. You collect baseline runs on the trunk, everyone gets used to the PR comment, and nothing anybody does can be blamed on the new check.
2. +Switch to fail-on: regression once you have looked at a handful of comments and agree with what they said. This is the setting most teams should stay on.
3. +Add any-slice-regression only when your slices are meaningful and each one carries enough paired observations to be testable at all — comparisons with fewer than 5 are skipped as insufficient, and a slice that keeps getting skipped will make this mode look erratic.

Do not skip step one. The point of the calibration week is to find out whether your suite is noisy before that noise starts blocking merges, because a gate people learn to re-run until it passes is worse than no gate.

## Gating locally too

The hosted diff is not the only way to fail a build. The CLI gates on its own analysis, no hosted layer involved:

evalshift all --gate critical,high

--gate takes a comma-separated subset of critical,high,medium,low and exits 1 when any comparison lands on a listed severity. --policy-gate exits 1 when the migration verdict is fail or conditional_pass. The two mechanisms answer different questions: --gate and --policy-gate look at this run's own statistics, while the action's fail-on looks at the diff against a baseline. A regression that is already in your trunk shows up in the first and not the second.

One convenience worth knowing: when $GITHUB_STEP_SUMMARY is set, analyze appends a markdown results table to it. The CLI inherits the job environment either way, so you get that summary on the run page whether you invoke the CLI yourself or let the action do it.

## Token hygiene

The token evalshift login issues is personal — it is tied to your membership and dies with it, which makes it exactly the wrong credential for a pipeline that has to outlive your employment. Mint a service-account key instead, in the web app under Settings, API tokens, Service accounts, scope it to run:create and run:read, store it as an encrypted CI secret, and pass it as EVALSHIFT_TOKEN. Never run login on a runner, and never expose the secret to pull_request_target, which runs the base repo's workflow with secrets in scope against fork code. Rotation is overlapping keys, never an in-place swap: mint the successor, update the secret, confirm a green run, then let the predecessor expire.

## Keep reading

• +How to test an LLM model migration before you ship it — the paired run this check automates.
• +When to trust an LLM judge — what pairwise judging is good at, and where it quietly misleads you.
• +Gating and PR feedback — every input, output, and failure mode of the action.
• +Hosted setup — projects, baselines, and the account side of the diff.