Every run makes real model calls against both the source and target model. Suite size × 2 is your per-run cost, and the runner starts with a cold cache every time, so nothing is free on repeat runs the way it is locally.

## Cost control

Practical levers, in order of effect. Only run when it matters — most PRs don’t touch prompts or the suite:

on:
  pull_request:
    paths:
      - "eval/**"
      - "app/prompts/**"
      - "evalshift.yaml"
  push:
    branches: [main]

Keep the CI suite smaller than your full local suite. A 40-example CI suite that runs on every PR catches more regressions in practice than a 500-example suite you disable after the first invoice.

Use cheap evaluators in CI. Structural evaluators cost nothing; LLM-judge evaluators are a third model call per example. If your local config leans on judges, consider a CI-specific config pointed at with config:. See Evaluators.

Don’t gate on draft PRs:

    if: github.event.pull_request.draft == false

## Recipes

### Config in a subdirectory

      - uses: babaliauskas/evalshift-action@v0
        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
          config: eval/evalshift.yaml
          suite: eval/golden.jsonl

Paths inside the config resolve relative to the config file, so prompts.py next to eval/evalshift.yaml needs no path changes.

### Report-only while calibrating

        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
          fail-on: never

### Two suites in one repository

Give each suite its own job, and let only one of them own the PR comment:

jobs:
  eval-text:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: babaliauskas/evalshift-action@v0
        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
          suite: eval/golden-text.jsonl

  eval-agent:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: babaliauskas/evalshift-action@v0
        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
          config: eval/agent.yaml
          suite: eval/golden-agent.jsonl
          comment: "false"

The comment marker and the commit status context are both constants, so two commenting invocations on the same PR overwrite each other’s output. One owner, always.

### Keep the HTML report as a build artifact

The Action doesn’t upload artifacts. Add a step if you want the report retained on the run:

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalshift-report
          path: .evalshift/runs/**/report.html

### Make it a required check

Branch protection → require status checks → add evalshift/regression. Do this only after the suite has been running at fail-on: never long enough that you trust it.

### Post as a bot account

        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
          github-token: ${{ secrets.MY_BOT_PAT }}

The comment upsert only edits comments authored by a Bot account. A PAT belonging to a human user will create a new comment on every run instead of updating one.