The EvalShift GitHub Action turns your golden suite into a merge gate. On every pull request it runs the suite against both models, pushes the result to hosted EvalShift, compares it to the latest run on your base branch, and fails the check when the diff shows a regression. It is a composite action that installs Python and the pinned CLI (evalshift==0.8.0 by default) — nothing compiled, nothing containerised. What it adds on top of the CLI: hosted push, baseline lookup, cross-branch diff, one self-updating PR comment, a commit status, and an exit code.

## Who this is for

You already have a golden suite and you run it by hand before shipping a model or prompt change. That works right up until it doesn’t: someone edits a system prompt on a Friday, nobody re-runs the suite, and the regression ships. The Action closes that gap — it makes “did this change make the model worse?” a required check, answered by the same statistics you’d get locally, on a pull request, before anyone can merge.

If you don’t have a suite yet, start with the CLI — Getting started gives you a working project in one command. Come back here once evalshift all passes locally.

## Prerequisites

Verify locally first. If evalshift all --yes doesn’t pass on your machine, it will not pass on a runner — you’ll just pay for the model calls to find out.

## Quick start

Create .github/workflows/evalshift.yml — or let evalshift init --ci scaffold a near-identical workflow for you:

name: evalshift

on:
  pull_request:
  push:
    branches: [main]

permissions:
  contents: read
  pull-requests: write
  issues: write
  statuses: write

jobs:
  evalshift:
    runs-on: ubuntu-latest
    env:
      EVALSHIFT_NONINTERACTIVE: "1"
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
    steps:
      - uses: actions/checkout@v7
      - uses: babaliauskas/evalshift-action@v0
        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
          fail-on: regression

Keep the push: branches: [main] trigger. Pull requests need something to compare against, and that something is the most recent run on your base branch. Without trunk runs, every PR reports “no baseline” and passes unconditionally.

## Secrets and provider keys

The Action does not manage provider credentials. It passes the job environment through to the CLI unchanged, so set the key as a job-level env: entry (ANTHROPIC_API_KEY, OPENAI_API_KEY, GEMINI_API_KEY/GOOGLE_API_KEY) and the CLI picks it up. Which key you need follows from defaults.source_model and defaults.target_model in your config — comparing across two providers means both keys:

    env:
      EVALSHIFT_NONINTERACTIVE: "1"
      OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

EVALSHIFT_NONINTERACTIVE: "1" is recommended. The Action already passes --yes, which skips the CLI’s cost confirmation, but the env var covers any other prompt — and a prompt on a runner means a hung job.

## Where to next

• +Inputs & outputs — every input, output, and permission, plus version pinning.
• +Gating & PR feedback — the fail-on modes, the PR comment, and how baselines resolve.
• +Cost control & recipes — run it only when it matters.
• +Hosted setup — the account and token the Action needs.