// action · recipes

Cost control & recipes

run it only when it matters

Every run makes real model calls against both the source and target model. Suite size × 2 is your per-run cost, and the runner starts with a cold cache every time, so nothing is free on repeat runs the way it is locally.

Cost control

Practical levers, in order of effect. Only run when it matters — most PRs don’t touch prompts or the suite:

yaml

on:
  pull_request:
    paths:
      - "eval/**"
      - "app/prompts/**"
      - "evalshift.yaml"
  push:
    branches: [main]

Keep the CI suite smaller than your full local suite. A 40-example CI suite that runs on every PR catches more regressions in practice than a 500-example suite you disable after the first invoice.

Use cheap evaluators in CI. Structural evaluators cost nothing; LLM-judge evaluators are a third model call per example. If your local config leans on judges, consider a CI-specific config pointed at with config:. See Evaluators.

Don’t gate on draft PRs:

yaml

    if: github.event.pull_request.draft == false

−

fail-on decides the exit code, not the spend

Even at fail-on: never, the run executes, costs money, and pushes.

Recipes

Config in a subdirectory

yaml

      - uses: evalshift/evalshift-action@v0
        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
          config: eval/evalshift.yaml
          suite: eval/golden.jsonl   # or suite-name: <key>, if it is wired under suites:

Paths inside the config resolve relative to the config file, so prompts.py next to eval/evalshift.yaml needs no path changes.

Report-only while calibrating

yaml

        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
          fail-on: never

Two suites in one repository

Give each suite its own job, select each by the name it is wired under in suites:, and let only one of them own the PR comment:

yaml

jobs:
  eval-text:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: evalshift/evalshift-action@v0
        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
          suite-name: text

  eval-agent:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v7
      - uses: evalshift/evalshift-action@v0
        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
          suite-name: agent
          comment: "false"

Select a wired suite with suite-name, not a suite path: a path loads the same rows but not that suite’s own evaluators: block, so a tool-calling suite would be scored with the top-level evaluators — silently, and often with nothing scored at all. The comment marker and the commit status context are both constants, so two commenting invocations on the same PR overwrite each other’s output: one comment owner, always. evalshift init --ci scaffolds this shape for every suite under .evalshift/suites/ — a matrix job per suite, selected by name — plus an evalshift gate job that collapses them into one check.

Keep the HTML report as a build artifact

The Action doesn’t upload artifacts. Add a step if you want the report retained on the run:

yaml

      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalshift-report
          path: .evalshift/runs/**/report.html

Make it a required check

Branch protection → require status checks. With one suite, add the evalshift/regression commit status. With several, every suite’s job writes that same status and the last one to finish overwrites the rest — require a job that depends on every suite job instead (evalshift gate in the workflow evalshift init --ci scaffolds), the only check that sees every suite. Do this only after the suite has been running at fail-on: never long enough that you trust it.

Post as a bot account

yaml

        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
          github-token: ${{ secrets.MY_BOT_PAT }}

The comment upsert only edits comments authored by a Bot account. A PAT belonging to a human user will create a new comment on every run instead of updating one.