## LLM regression testing in CI: gate pull requests on eval diffs

URL: https://www.evalshift.dev/blog/llm-regression-testing-in-ci
Published: 2026-07-24
Tag: ci
Summary: Wire a golden suite into GitHub Actions so every pull request gets a paired eval run, a base-branch diff, and a check that fails on real regressions.

Takeaways:
- The gate answers three questions per pull request with no human in the loop: did anything regress, where, and is it big enough to block.
- Only the third is a policy choice — it is the fail-on input, and the only part of the gate you actually configure.
- any-slice-regression is not a stricter regression: the two read different fields, so switching modes changes the question rather than tightening a dial.
- A permanently green check usually means no comparable baseline diff exists yet, not that the suite is passing.
- Ship with fail-on: never for a calibration week — a gate people learn to re-run until it passes is worse than no gate.

An eval suite you run when you remember to run it is not a gate. It is a habit, and habits fail
exactly when the pressure is on — the Friday prompt tweak, the dependency bump that moves a model
alias, the week everyone is shipping something else. The suite stays correct the entire time.
Nobody runs it.

The version that changes outcomes is attached to the pull request. Someone edits a system prompt to
fix one customer complaint, the paired run says tool selection dropped on the refund slice, the
check goes red, and the branch does not merge until someone looks. The conversation on the PR is
then about a measured delta rather than about whether the change felt risky. This post is how to
wire that up with the EvalShift GitHub Action, and how to turn it on without halting your team's
merges on day one.

## What the gate has to answer

A CI eval earns its runtime by answering three questions on every pull request, with no human in
the loop:

- Did anything regress? The action pushes the completed run to hosted EvalShift, asks the API for a
  baseline run on the base branch, and reads `aggregate_delta.regressions` off the server-side diff.
- Where? The same diff carries `per_slice_deltas` — a `pass_rate_delta` per slice — so "worse
  overall" resolves to "worse on the multilingual cases, flat everywhere else."
- Is it big enough to block? That one is not a measurement, it is a policy choice, and it is the
  only part of the gate you actually configure.

The first two are the same numbers you would read in the web app. The third is the `fail-on` input,
below.

## Prerequisites

All four are hard requirements — the job fails without them:

- `evalshift.yaml` committed at the repository root, or the `config:` input pointed at wherever it
  lives. Paths inside the config resolve relative to the config file's own directory, so a config in
  a subdirectory works unchanged.
- A golden JSONL suite committed, or the `suite:` input pointed at it. The default is
  `golden.jsonl`.
- A repository or environment secret `EVALSHIFT_TOKEN` holding an `es_...` service-account key
  scoped to `run:create` and `run:read`. Those two scopes are exactly what the action exercises:
  `run:create` for `evalshift push`, `run:read` for the baseline lookup and the diff fetch.
- A model provider key in the job env matching the models named in `evalshift.yaml` —
  `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, or `GEMINI_API_KEY` (`GOOGLE_API_KEY` works too). A
  cross-provider migration needs both keys.

> Every run makes real model calls and spends real credits. Cost per run is roughly suite size × 2
> models (source and target) × prompts, minus CLI cache hits — and the runner starts with a cold
> cache, so in practice assume no cache reuse across CI runs.

That cost is the reason to scope the trigger. Narrowing `on.pull_request.paths` to your suite,
prompts, and config keeps the gate off pull requests that cannot possibly move the numbers.

## The workflow

```yaml
permissions:
  contents: read
  pull-requests: write
  issues: write
  statuses: write
jobs:
  evalshift:
    runs-on: ubuntu-latest
    env:
      EVALSHIFT_NONINTERACTIVE: "1"
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    steps:
      - uses: actions/checkout@v7
      - uses: babaliauskas/evalshift-action@v0
        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
          fail-on: regression
```

It is a composite action, not a container, so what runs is a short ordered list you can reason
about: set up Python (3.14 by default), `pip install` a pinned `evalshift` CLI version, then a
stdlib-only helper script that runs the suite, pushes the completed run to hosted EvalShift, asks
the API for a compatible baseline run on the base branch, fetches the server-side diff, keeps
exactly one PR comment updated in place, and sets the `evalshift/regression` commit status. All
evaluation and statistics happen in the CLI; all diffing happens on the server. The action is the
wrapper.

Two things about that list are worth knowing before you stare at a log. Install is not cached, so
budget roughly 20 to 60 seconds for it every run. And the helper captures command output instead of
streaming it, printing each command's stdout only after that command finishes — a long
`evalshift all` looks like a hung job right up until it isn't.

Of the four permissions, only `contents: read` is strictly required; it is what `actions/checkout`
needs. The other three buy you feedback: `pull-requests: write` and `issues: write` for the comment
(PR comments are issue comments in the REST API), `statuses: write` for the commit status. Drop
them and you get stderr warnings instead — the gate still fails the job correctly.

## Choosing fail-on

| `fail-on` | fails when |
| --- | --- |
| `never` | never — reports only |
| `regression` | `regression_count > 0` |
| `any-slice-regression` | any per-slice `pass_rate_delta < 0` |

The non-obvious part: `any-slice-regression` is not a superset of `regression`. They read different
fields. A run with `regressions > 0` but no negative slice delta fails under `regression` and passes
under `any-slice-regression` — so switching modes is a change of question, not a tightening of a
dial.

The other case to have in your head is the empty one. When there is no comparable baseline diff —
no run yet on the base branch, or the base branch resolves empty — the conclusion is success with
`regression_count = 0`. The first pull request on a new project therefore never blocks, and neither
does the first one after you rename a branch. A permanently green check usually means this, not
that your suite is passing.

## Rolling it out without blocking everyone on day one

1. Ship it with `fail-on: never` and leave it there for about a week. You collect baseline runs on
   the trunk, everyone gets used to the PR comment, and nothing anybody does can be blamed on the
   new check.
2. Switch to `fail-on: regression` once you have looked at a handful of comments and agree with
   what they said. This is the setting most teams should stay on.
3. Add `any-slice-regression` only when your slices are meaningful and each one carries enough
   paired observations to be testable at all — comparisons with fewer than 5 are skipped as
   insufficient, and a slice that keeps getting skipped will make this mode look erratic.

Do not skip step one. The point of the calibration week is to find out whether your suite is noisy
before that noise starts blocking merges, because a gate people learn to re-run until it passes is
worse than no gate.

## Gating locally too

The hosted diff is not the only way to fail a build. The CLI gates on its own analysis, no hosted
layer involved:

```bash
evalshift all --gate critical,high
```

`--gate` takes a comma-separated subset of `critical,high,medium,low` and exits 1 when any
comparison lands on a listed severity. `--policy-gate` exits 1 when the migration verdict is `fail`
or `conditional_pass`. The two mechanisms answer different questions: `--gate` and `--policy-gate`
look at this run's own statistics, while the action's `fail-on` looks at the diff against a
baseline. A regression that is already in your trunk shows up in the first and not the second.

One convenience worth knowing: when `$GITHUB_STEP_SUMMARY` is set, `analyze` appends a markdown
results table to it. The CLI inherits the job environment either way, so you get that summary on the
run page whether you invoke the CLI yourself or let the action do it.

## Token hygiene

The token `evalshift login` issues is personal — it is tied to your membership and dies with it,
which makes it exactly the wrong credential for a pipeline that has to outlive your employment.
Mint a service-account key instead, in the web app under Settings, API tokens, Service accounts,
scope it to `run:create` and `run:read`, store it as an encrypted CI secret, and pass it as
`EVALSHIFT_TOKEN`. Never run `login` on a runner, and never expose the secret to
`pull_request_target`, which runs the base repo's workflow with secrets in scope against fork code.
Rotation is overlapping keys, never an in-place swap: mint the successor, update the secret, confirm
a green run, then let the predecessor expire.

## Keep reading

- [How to test an LLM model migration before you ship it](/blog/test-llm-model-migration-before-you-ship)
  — the paired run this check automates.
- [When to trust an LLM judge](/blog/when-to-trust-an-llm-judge) — what pairwise judging is good at,
  and where it quietly misleads you.
- [Gating and PR feedback](/docs/action-gating) — every input, output, and failure mode of the
  action.
- [Hosted setup](/docs/hosted-setup) — projects, baselines, and the account side of the diff.