# evalshift-action (GitHub Action) — complete reference for AI tools Canonical hosted copy: https://www.evalshift.dev/ci-llms-full.txt Repo/action ref: babaliauskas/evalshift-action | version: 0.1.0 | license: MIT Kind: composite GitHub Action (not JavaScript, not Docker) Marketplace name: "EvalShift" | branding: bar-chart-2 / purple Runtime helper: scripts/evalshift_action.py (stdlib only, no third-party deps) Purpose: check the org's plan covers this job before spending credits, run an EvalShift golden suite inside CI, push the completed run to hosted EvalShift, ask the hosted API for a compatible baseline run on the base branch, fetch the hosted diff, ask the hosted governed gate to judge the run against the project's migration policy, write action outputs, keep exactly one PR comment up to date, set the `evalshift/regression` commit status, and exit non-zero when the selected `fail-on` mode says to. The action is a thin CI wrapper: all evaluation, statistics, and reporting happen in the EvalShift CLI it installs; all diffing happens server-side in hosted EvalShift. Related packages (separate repos, separate docs): - evalshift (PyPI, the CLI this action installs and shells out to) — https://www.evalshift.dev/cli-llms-full.txt - evalshift-sdk (PyPI, in-process capture SDK) — https://www.evalshift.dev/sdk-llms-full.txt Minimal usage: ```yaml permissions: contents: read pull-requests: write issues: write statuses: write jobs: evalshift: runs-on: ubuntu-latest env: EVALSHIFT_NONINTERACTIVE: "1" ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} steps: - uses: actions/checkout@v7 - uses: babaliauskas/evalshift-action@v0 with: token: ${{ secrets.EVALSHIFT_TOKEN }} fail-on: policy # the default ``` Prerequisites in the repository, all of them hard requirements: 1. `evalshift.yaml` committed (or `config:` pointed at wherever it lives). 2. A golden JSONL suite committed (or `suite:` pointed at it). 3. Repository (or environment) secret `EVALSHIFT_TOKEN` — an `es_...` service-account key, scoped to `run:create` + `run:read`. See "Token hygiene" below. (`project:read` additionally enables the plan preflight; without it the preflight is skipped, not failed.) 4. A model provider key in the job env matching the models named in `evalshift.yaml`. Every run makes real model calls and spends real credits. Cost per run ≈ suite size × 2 models (source + target) × prompts, minus CLI cache hits (the runner starts with a cold cache, so in practice assume no cache reuse across CI runs). ## Execution model Composite `runs.steps`, in order: 1. `actions/setup-python@v7` with `python-version` (default `3.12`). 2. `python -m pip install --upgrade pip` then `python -m pip install "evalshift=="`. No pip caching; expect ~20-60s of install per run. 3. `python "$GITHUB_ACTION_PATH/scripts/evalshift_action.py"` with `id: run`, all inputs passed as `INPUT_*` env vars. (Composite actions do NOT auto-populate `INPUT_*` for `run:` steps, so action.yml maps every input explicitly — an input added to `inputs:` without a matching `env:` entry is invisible to the helper.) The helper's process CWD is the workspace root (`Path.cwd()`), so `config` / `suite` paths and the `.evalshift/` output directory are all relative to the repository root by default. The helper never streams: `run_command` uses `subprocess.run(capture_output=True)` and prints stdout/stderr only after each CLI command finishes. A long `evalshift all` looks silent in the log until it completes. ## Inputs (action.yml) | Input | Required | Default | Meaning | |---|---|---|---| | token | yes | — | Hosted EvalShift API token (`es_...`). Masked via `::add-mask::`, and redacted from CLI output. Passed to the CLI as env `EVALSHIFT_TOKEN`, never in argv. | | host | no | https://api.evalshift.dev | Hosted API base URL. Trailing `/` stripped. Passed as env `EVALSHIFT_HOST`. Set only for self-hosted/staging. | | config | no | evalshift.yaml | Path to config, relative to workspace root. Paths *inside* the config resolve relative to the config file's own directory (CLI behavior), so a config in a subdirectory works unchanged. | | suite | no | golden.jsonl | Path to golden JSONL suite, relative to workspace root. | | evalshift-version | no | 0.12.1 | Exact CLI version installed from PyPI. Pinned for reproducibility. | | python-version | no | 3.12 | Python for the CLI. Must satisfy the CLI's `requires-python` (>=3.11 for 0.12.1); lowering it below that breaks install. | | fail-on | no | policy | `policy` \| `never` \| `regression` \| `any-slice-regression`. Any other value → hard error before any work. DEFAULT CHANGED: was `regression`; `policy` gates on the server's migration-policy verdict, not on the diff. | | branch | no | "" (auto) | Candidate branch recorded on the hosted run. Auto-detected; override only for non-standard naming. | | base-branch | no | "" (auto) | Branch searched for a baseline run. Auto-detected. If it resolves empty, no baseline lookup happens at all and the check passes. | | create-project | no | true | When false, appends `--no-create-project` to `evalshift push`, making a missing hosted project a hard failure. | | comment | no | true | Whether to upsert the PR comment. Commit status is set regardless. | | github-token | no | `${{ github.token }}` | Token for the PR comment and commit status. Also masked. If empty, both comment and status are skipped entirely. | | repo-private | no | `${{ github.event.repository.private }}` | Repository visibility, asserted to the CI preflight (never verified server-side). Drives the `private_repo_ci` entitlement check. | Boolean inputs are truthy for `1`, `true`, `yes`, `on` (case-insensitive); everything else is false. All inputs are `.strip()`ed. ## Outputs | Output | Value | |---|---| | run_url | Hosted run URL, parsed as the last http(s) line of `evalshift push` stdout. Empty is impossible — a push that prints no URL is a hard error. | | diff_url | Hosted `web_diff_url` for the baseline comparison; empty string when no compatible baseline. | | run_id | Local/hosted run id (e.g. `r_20260723_myssuite_ab12cd`). | | regression_count | `aggregate_delta.regressions` from the hosted diff; `0` when no baseline. | | conclusion | `success` or `failure`, after applying `fail-on`. It is a GitHub commit-status state, so both a `conditional_pass` and an undecided policy (`inconclusive` or an unknown status) read `success` here; the PR comment and the status description carry the verdict. | Written by appending `key=value` lines to `$GITHUB_OUTPUT`. Values are single-line by construction; there is no heredoc delimiter handling, so a hypothetical multi-line value would corrupt the output file. ## Commands the action shells out to ``` evalshift all --yes --config --suite evalshift push --config --suite [--no-create-project] ``` Both inherit the job environment plus `EVALSHIFT_HOST` and `EVALSHIFT_TOKEN`. A non-zero exit from either raises `ActionError` and fails the step (message: `command failed (): `). `run_id` is NOT parsed from CLI output. It is the directory under `/.evalshift/runs` with the newest mtime. Consequences: a pre-existing `.evalshift/runs` in the checkout is harmless (the fresh run is newest), but any step that touches an older run directory between `all` and the run-id read can select the wrong run. Missing directory → `run directory ... does not exist`; empty directory → `no local EvalShift runs found in ...`. `run_url` extraction scans `push` stdout bottom-up for the first non-empty line starting with `http://` or `https://`. A CLI release that stops printing a bare URL as the last line breaks this (guarded by the `cli-contract` CI job, which only checks flags, not output shape). ## Plan preflight (runs BEFORE the CLI) Purpose: refuse a job the org's plan does not cover before any model credits are spent. Sequence, all skippable, all before `evalshift all`: 1. `project_ref_from_config(Path(config))` — regex `^project:\s*["']?([a-z0-9-]+)/([a-z0-9-]+)` against the config file, one line at a time. No YAML parser (helper is dependency-free). No match (or unreadable file) → preflight skipped entirely, run proceeds. 2. `GET {host}/orgs/{org}/projects` → first item whose `slug` matches → its `id`. Needs `project:read`. No match → skipped (the project is created by the first `evalshift push`). 3. `POST {host}/projects/{id}/ci-preflight` with body `{"repo_private": , "parallelism": 1}`. `PREFLIGHT_PARALLELISM = 1` is a constant, not an input: one EvalShift run per job, and the server counts in-flight runs itself. Outcomes: - `200 {"allowed": true}` → continue. - `402` → `PreflightDenied(message, details)`, job fails immediately with exit 1, `evalshift` never runs. `details` is the server's envelope: `feature`, `tier`, `limit`, `used`, `status`, `resets_at`, `upgrade_url`. The action renders it and decides nothing itself. - Anything else (5xx, 403, 404, timeout, DNS, malformed body) → `warning: plan preflight skipped: ` on stderr, run proceeds. Fail-closed on billing, fail-open on infrastructure. Denial rendering, all three from the same markdown body (`build_preflight_body`): - stdout: `::error title=EvalShift::%0AUpgrade: ` (workflow commands are single-line; `%`/`\r`/`\n` escape to `%25`/`%0D`/`%0A`). - `$GITHUB_STEP_SUMMARY`: the body, appended. - PR comment: the same body via `upsert_pr_comment`, carrying `COMMENT_MARKER`, so it replaces the regular EvalShift comment rather than stacking. Requires `github-token` + `comment: true`. Outputs on denial: `run_url=""`, `diff_url=""`, `run_id=""`, `regression_count=0`, `conclusion=failure`. `repo_private` is client-asserted; the server cannot verify it and records the first `true` permanently (set-once), so a later `false` from the same project changes nothing. ## Hosted API contract used Auth on all calls: `Authorization: Bearer `, `Accept: application/json`, 30s timeout, stdlib `urllib`. The preflight calls above use the same client and headers. 1. `GET {host}/runs/{run_id}/baseline-compatible?branch={base_branch}` Response (object required, else hard error): ```json { "baseline_run": {"id": "..."} | null, "compatibility": "direct", "api_diff_url": "/runs//diff/", "web_diff_url": "https://app.evalshift.dev/..." } ``` Skipped entirely when `base_branch` is empty. 2. `GET api_diff_url` (absolute URLs used as-is; relative joined onto `host`). Only called when `api_diff_url` is truthy. Response fields the action reads: ```json { "aggregate_delta": {"regressions": , "pass_rate_delta": }, "per_slice_deltas": [{"slice": "", "pass_rate_delta": }] } ``` Any other key is ignored. Non-object response → hard error. 3. `GET {host}/runs/{run_id}/policy-check` — the governed gate. Called ONLY when `fail-on: policy`, always for the run just pushed, independent of whether a baseline exists. ```json { "run_id": "...", "status": "pass" | "conditional_pass" | "fail" | "inconclusive", "verdict": "" | null, "reason": "", "policy_source": "project_policy" | "default_policy", "policy": { ... }, "budgets": [{"name": "...", "observed": , "allowed": , "passed": , "scope": "...", "ci_low": , "ci_high": , "conclusive": }], "blocking_regressions": [{"prompt_id": "...", "evaluator_name": "...", "slice_name": "...", "severity": "...", "delta_avg_score": , "effect_size": }] } ``` `status` is a CLOSED set of four, and the action recognises all four: - `pass` — every budget within policy. - `conditional_pass` — every budget held and nothing critical/high regressed, but medium/low regressions and/or comparisons that scored zero pairs. A PASSING state with caveats; it deliberately does not fail the gate. Its `reason` contains "not a gate failure" and ends "Review before merging." - `fail` — a budget busted, or a blocking critical/high regression. - `inconclusive` — three distinct causes, distinguishable ONLY by `reason`: (1) no policy metrics recorded for the run; (2) nothing comparable — every comparison scored severity `insufficient`, which also returns `budgets: []`; (3) a policy-declared slice went unmeasured. The action prints `reason` verbatim and never paraphrases it, or the three collapse into one on the surface readers see. A string outside those four means a newer server; the action treats it exactly like `inconclusive` (undecided, never a pass) so a pinned version keeps working. `budgets[]` may exceed six entries and `scope` may be a slice name rather than `overall`; it is `[]` for the all-`insufficient` case. Never raises out of `fetch_policy_check` — HTTPError (any code), URLError, a wrapped 403 `ActionError`, a malformed body, or a response whose `status` is missing/blank all return `(None, reason)` and print `warning: ; falling back to fail-on: regression` to stderr. `target_url` for the commit status = `web_diff_url` when present, else `run_url`. ## Gating algorithm (exact) ``` # Diff facts, computed in every mode (the comment renders them even when they do not gate). diff is None -> regression_count=0, slice_regressions=[] regression_count = int(aggregate_delta.regressions or 0) slice_regressions = [s for s in per_slice_deltas if float(s.pass_rate_delta) < 0] sorted ascending by pass_rate_delta # most negative first top_slice_regressions = slice_regressions[:5] # comment display only fail_on == "never" -> should_fail = False fail_on == "regression" -> should_fail = regression_count > 0 fail_on == "any-slice-regression" -> should_fail = len(slice_regressions) > 0 fail_on == "policy": # the default policy-check unavailable -> should_fail = regression_count > 0 # fallback, announced status == "fail" -> should_fail = True status == "pass" -> should_fail = False status == "conditional_pass" -> should_fail = False, reported as a PASS with caveats status == "inconclusive" -> should_fail = False, reported as undecided any other status -> should_fail = False, reported as unrecognized/undecided conclusion = "failure" if should_fail else "success" ``` Non-numeric / missing deltas coerce to `0.0`, so a malformed slice entry is treated as "flat", never as a regression. `any-slice-regression` is strictly not a superset of `regression`: a run with `regressions > 0` but no negative slice delta fails under `regression` and passes under `any-slice-regression`. `policy` is not a stricter or looser version of `regression` either — it is the server's verdict on the project's migration policy and can disagree in both directions: a diff with regressions that stay inside every budget passes, and a diff with `regressions == 0` that busts a cost/latency/per-slice budget fails. Undecided is never a pass and never a failure: `should_fail=False`, `conclusion=success` (the only GitHub states are success/failure), and `GatingResult.summary` — surfaced in the commit status description and the PR comment — says the gate could not decide, naming the unrecognized status when there is one. An unavailable policy check gates on the diff for that run and is announced in the job log, the status description and the comment; it never silently goes green. `conditional_pass` is NOT undecided. `GatingResult.policy_decided` is True for it (the set is `POLICY_DECIDED_STATUSES = {"pass", "conditional_pass", "fail"}`) and `GatingResult.policy_caveated` is True only for it (`POLICY_CAVEATED_STATUSES = {"conditional_pass"}`). `summary` reads `the gate passed, with caveats: `, and the PR comment renders a "passed, with caveats" blockquote instead of the "could not decide" one. `GatingResult` fields: `conclusion`, `should_fail`, `regression_count`, `top_slice_regressions`, `mode`, `summary`, `policy_status`, `policy_verdict`, `policy_reason`, `policy_source`, `budgets`, `blocking_regressions`, `policy_unavailable_reason`, plus the derived `policy_decided` (`policy_status in {"pass","conditional_pass","fail"}`) and `policy_caveated` (`policy_status in {"conditional_pass"}`). Process exit code: `1` when `should_fail`, `1` on any `ActionError` (printed as `error: ` to stderr), else `0`. ## GitHub context detection ``` event_name = GITHUB_EVENT_NAME repository = GITHUB_REPOSITORY event = JSON at GITHUB_EVENT_PATH (unreadable/invalid -> {}) is_pull_request = event_name.startswith("pull_request") # includes pull_request_target pull_number = event.number if int else None sha = event.pull_request.head.sha or GITHUB_SHA branch = input.branch or event.pull_request.head.ref or GITHUB_HEAD_REF or GITHUB_REF_NAME base_branch = input.base-branch or event.pull_request.base.ref or GITHUB_BASE_REF or GITHUB_REF_NAME ``` On a `push` event, `base_branch` falls back to the pushed branch itself — a push to `main` therefore diffs against the previous `main` run, which is the intended "track the trunk" behavior, not a bug. ## PR comment Marker: `` (first line of the body). Upsert logic: - Skipped unless `comment: true`, `is_pull_request`, and `pull_number is not None`. - `GET /repos/{repo}/issues/{pull_number}/comments`, then the FIRST comment whose author `user.type == "Bot"` AND whose body contains the marker is PATCHed; otherwise a new comment is POSTed. A human-authored comment containing the marker is deliberately never edited. - Pagination is not handled: only the first page of comments is inspected. On a PR with enough comments to push the EvalShift comment off page 1, a duplicate is created. - The marker is a constant, so two invocations of this action on the same PR fight over one comment. Run at most one invocation with `comment: true` per PR. - HTTP 403/404 → `warning: could not upsert PR comment: HTTP ` on stderr and continue. Any other HTTP error propagates and fails the step. Body shape (no baseline): ``` ## EvalShift regression check **Conclusion:** `success` **Hosted run:** [open run]() **Regressions:** 0 No compatible baseline run was found on the base branch. ``` Body shape (with baseline) replaces the trailing line with a `**Diff:**` link (when `web_diff_url` exists), a `**Pass-rate movement:**` line, and a two-column table of up to five regressed slices, or the single row `| No regressed slices | 0 pts |`. Under `fail-on: policy` only, two extra sections sit between the header lines and the diff sections (they render with or without a baseline): - `**Policy decision:** \`\` (from \`\`)` + `**Why:** `, where `` is the server's string verbatim (whitespace-collapsed only). On `conditional_pass`, a blockquote states the gate "passed, with caveats" and points at the reason. On `inconclusive` or an unrecognized status, a blockquote states that the gate could not decide and that this is not a pass. When the check was unavailable, the whole pair is replaced by a blockquote naming the reason and the `fail-on: regression` fallback. - `### Policy budgets` — `| Budget | Scope | Observed | Allowed | Result |`, failing rows first, capped at `MAX_BUDGET_ROWS = 12`, then ` more budgets not shown — see the hosted run.`, or ` more budgets not shown, of them failing — see the hosted run.` when the hidden rows include failures (per-slice budgets can produce more failures than fit). `Scope` is `overall` or the slice name. Result is `pass`/`fail`/`unknown`, with ` (not confident)` appended when `conclusive` is `false`. Numbers format as `f"{v:.4g}"`; a non-numeric or absent value reads `n/a`. Omitted entirely when `budgets` is `[]` — the all-`insufficient` case renders no table at all rather than an empty one. - `### Blocking regressions` — `| Prompt | Evaluator | Slice | Severity | Score delta |`, capped at `MAX_BLOCKING_ROWS = 10`, then ` more blocking regressions not shown — see the hosted run.` Server-supplied strings in these tables are collapsed to one line and `|` is escaped, so a budget or slice name cannot break the table. Percent formatting is `round(value * 100)` with a `+` prefix only when positive, suffixed ` pts` — e.g. `-20 pts`, `+5 pts`, `0 pts`. Rounding is display-only; gating uses raw floats. ## Commit status `POST /repos/{repo}/statuses/{sha}` with: - `context`: `evalshift/regression` (constant — parallel invocations overwrite each other) - `state`: `success` | `failure` (mirrors `conclusion`) - `target_url`: `web_diff_url` or `run_url` - `description`: `EvalShift : `, truncated to 140 chars. `` is the policy sentence under `fail-on: policy` (verdict, source, reason, or the fallback notice); ` regression(s)` in every other mode. Set on every event where `github-token` is non-empty, including `push`. 403/404 → warning `warning: could not set commit status: HTTP ` and continue; other HTTP errors fail. ## Secrets handling - `mask_secret` prints `::add-mask::` for `token` and `github-token` before any other work. - `redact_text` replaces every env value whose KEY (uppercased) contains `TOKEN` or `SECRET`, or ends with `API_KEY`, with `` in printed CLI stdout/stderr. Values shorter than 4 chars are skipped. Redaction applies to what is PRINTED; `CommandResult.stdout` retains the raw text for URL parsing. - Token and host reach the CLI only through env, never argv (asserted in tests). - Provider API keys are the caller's responsibility: the action passes the job environment through unchanged and never sets, reads, or forwards them to the hosted API. ## Token hygiene (service-account keys) Storage — encrypted GitHub secrets only: - Repository secret, environment secret (preferred for production: adds required reviewers and branch restrictions to the credential itself), or org secret. Nothing else is supported. - Never a committed file, never a literal `env:`/`with:` value in workflow YAML (readable by anyone who can read the repo, and retained in git history after deletion). - Never reachable from `pull_request_target`: that trigger runs the base repo's workflow with secrets in scope against fork code, so a fork PR can exfiltrate the key. Use `pull_request`. - Masking/redaction (see "Secrets handling") protects the job's logs only; it is not storage. Identity — a service account, never a personal token: - Mint at EvalShift web app → Settings → API tokens → Service accounts (`/app//settings/tokens`). Org-owned machine identity; survives the employee who created CI leaving. A personal token dies with its owner's membership and takes the pipeline with it. - Service-account roles are `member` or `viewer` only — never owner-equivalent. Use `member`; `viewer` holds `run:read` but not `run:create`. Scopes — the permission keys the action actually needs (the web-app scope picker uses the same vocabulary; scopes are an intersection, so naming a key the role lacks grants nothing): | Scope | Needed by | |---|---| | run:create | `evalshift push`: `POST /runs` + `POST /runs/{id}/finalize` | | run:read | `GET /runs/{id}/baseline-compatible` + `GET /runs/{a}/diff/{b}` | Out of reach for a scoped key, by design: - Auto-creating the hosted project (`project:create` is owner-only). Create the project in the web app and set `create-project: false`. - Rewriting gating thresholds (`policy:configure` is owner-only). `evalshift push` sends the `thresholds:` block from `evalshift.yaml` whenever one is present, so keep thresholds canonical in the web app and out of the CI config, or push fails `Project owner role required`. Rotation — overlapping keys, never an in-place secret swap: 1. Rotate in the web app (the old key keeps working for a 24-hour grace window). 2. Update the GitHub secret. 3. Confirm a green run, then let the old key expire. ## GitHub permissions | Permission | Needed for | |---|---| | contents: read | `actions/checkout` | | pull-requests: write | PR comment | | issues: write | PR comments are issue comments in the REST API | | statuses: write | `evalshift/regression` commit status | Only `contents: read` is strictly required. Missing comment/status permissions degrade to stderr warnings; the gate still fails the job correctly. ## Provider key matrix | Provider | Env var | |---|---| | Anthropic | ANTHROPIC_API_KEY | | OpenAI | OPENAI_API_KEY | | Google | GEMINI_API_KEY or GOOGLE_API_KEY | Which key is required follows from `defaults.source_model` / `defaults.target_model` in `evalshift.yaml`. A cross-provider migration needs both keys. `EVALSHIFT_NONINTERACTIVE: "1"` is recommended in the job env: `evalshift all --yes` already skips the >$10 confirmation, but the env var covers any other prompt a CLI release adds. ## Behavior rules (invariants) - Missing `token` → `input 'token' is required`, exit 1. Validation happens in the helper, i.e. AFTER the composite has already installed Python and the CLI — a misconfigured workflow still burns install time, but never model credits. - Invalid `fail-on` → `input 'fail-on' must be one of: never, regression, any-slice-regression, policy`. - A denied preflight fails the job regardless of `fail-on`, including `never`: `fail-on` governs regressions, not whether the plan permits the run at all. - The action never decides what a plan covers. It renders the server's 402 message, details and `upgrade_url`, and exits non-zero. No entitlement is inferred, cached, or hard-coded. - A preflight that cannot get an answer never blocks a run; the server re-checks every limit at `POST /runs` and at finalize, so skipping it bypasses nothing. - No baseline (empty `base_branch`, null `api_diff_url`, or first run on a branch) → check passes, `regression_count=0`, `diff_url=""`, comment says so explicitly. First PR against a repo with no history therefore always goes green. - The action never uploads report artifacts to GitHub. `report.html` and all run artifacts stay in the workspace at `.evalshift/runs//`; add your own `actions/upload-artifact` step if you want them retained. - The action never writes to the repository and never pushes commits. - A hosted 403 is self-diagnosing, never a traceback: `HostedClient._get` converts it to an `ActionError` carrying the server's message plus the fix, and `run_command` appends the same guidance when failed CLI output contains `Permission denied: `. Non-403 `HTTPError`s from hosted calls propagate unchanged, and GitHub-side 403/404s stay warnings. - Outputs are written before the comment/status calls, so a permissions failure still leaves outputs consumable by later steps. - `evalshift push` is idempotent on run id (CLI behavior); a re-run of the job creates a new run id, not a duplicate push. - The helper has zero third-party dependencies; `pyproject.toml` declares `dependencies = []` and dev-only `pytest` / `ruff` / `pip-audit`. - `requires-python = ">=3.12"` for the helper itself; the `python-version` input governs the CLI, which is looser (>=3.11), so the helper's own floor is the binding one. ## Repo CI (what guards this action) - `.github/workflows/ci.yml` job `test`: `uv run pytest`, `ruff check .`, `pip-audit`. - `.github/workflows/ci.yml` job `cli-contract`: reads `evalshift-version` and `python-version` defaults straight out of `action.yml` via awk, installs that exact CLI, strips ANSI, and asserts `evalshift all --help` still offers `--yes --config --suite` and `evalshift push --help` still offers `--no-create-project --config --suite`. Free — no API keys, no credits. This is the drift alarm for CLI flag renames. - `.github/workflows/dogfood.yml`: `workflow_dispatch` only (spends real credits), runs the action against `examples/dogfood/` with `fail-on: never`, `comment: "false"`, then asserts `run_id` non-empty, `run_url` starts with `https://`, `conclusion` ∈ {success, failure}. Warns and skips when `EVALSHIFT_TOKEN` / `GEMINI_API_KEY` secrets are absent. ## Fixture project (examples/dogfood/) ```yaml # examples/dogfood/evalshift.yaml — deliberately tiny (4 examples, one evaluator) version: 1 prompts: - id: greet detection: python_string path: prompts.py variable: GREET_PROMPT variables: [name, tone] defaults: source_model: gemini-2.5-flash target_model: gemini-2.5-pro concurrency: 4 cache: true evaluators: structural: - type: length min_chars: 5 max_chars: 200 slices: - name: formal filter: formal - name: casual filter: casual ``` ```jsonl {"id": "ex01", "inputs": {"name": "Alex", "tone": "formal"}, "tags": ["formal"]} {"id": "ex02", "inputs": {"name": "Sam", "tone": "casual"}, "tags": ["casual"]} ``` ## Recipes Config in a subdirectory: ```yaml - uses: babaliauskas/evalshift-action@v0 with: token: ${{ secrets.EVALSHIFT_TOKEN }} config: eval/evalshift.yaml suite: eval/golden.jsonl ``` Consume outputs downstream: ```yaml - uses: babaliauskas/evalshift-action@v0 id: evalshift with: token: ${{ secrets.EVALSHIFT_TOKEN }} - run: echo "diff ${{ steps.evalshift.outputs.diff_url }} (${{ steps.evalshift.outputs.regression_count }} regressions)" ``` Report only, never block (calibration phase): ```yaml with: token: ${{ secrets.EVALSHIFT_TOKEN }} fail-on: never ``` Multiple suites in one repo — give each its own job and disable the comment on all but one, or the constant marker and constant status context make them overwrite each other: ```yaml - uses: babaliauskas/evalshift-action@v0 with: token: ${{ secrets.EVALSHIFT_TOKEN }} suite: eval/golden-agent.jsonl comment: "false" ``` Only run when the suite or prompts changed (cost control): ```yaml on: pull_request: paths: ["eval/**", "app/prompts/**", "evalshift.yaml"] ``` ## Troubleshooting checklist `input 'token' is required` → the `token:` input is unset or an empty secret; secrets are not available to workflows triggered by `pull_request` from a fork. `command failed (1): evalshift all ...` → a CLI-level failure (bad config, missing provider key, model error). The CLI's own stderr is printed directly above, redacted. `no local EvalShift runs found in .../.evalshift/runs` → `evalshift all` exited 0 without writing a run; almost always a config pointing at a different `--runs-base` or a CWD mismatch. `evalshift push did not print a hosted run URL` → CLI output shape changed, or push succeeded silently; re-run `evalshift push ` locally to see what it prints. HTTP 401 from the hosted API → bad, revoked, or expired `EVALSHIFT_TOKEN`, or wrong `host`. `The EvalShift token is missing the '' permission.` → hosted 403. Printed by the helper for a 403 on its own requests and for a `Permission denied: ` line in failed CLI output; names the exact permission key and points at minting a scoped service-account key. Exit code stays non-zero. `cannot auto-create project: this token must have owner access to the org` → a service-account key cannot create projects; pre-create it and set `create-project: false`. `Project owner role required` → push tried to rewrite gating thresholds (`policy:configure`, owner-only); drop `thresholds:` from the CI config. Job failed before the CLI ran, with a plan message and an upgrade link → CI preflight got a 402; the org's plan does not cover this run (usually the monthly run quota or the parallelism cap; private-repo CI is included on every plan). Nothing ran, nothing was charged. `warning: plan preflight skipped: ` → preflight got no answer (project not hosted yet, token without `project:read`, hosted API unreachable). Intended: the run continues and the server enforces the limit at upload. `warning: could not upsert PR comment: HTTP 403` → missing `pull-requests: write` / `issues: write`, or a fork PR with a read-only `GITHUB_TOKEN`. Gate still works. Check always green → the project's migration policy is permissive enough that no budget was busted (default `fail-on: policy`; the comment shows the budget arithmetic), or no compatible baseline on the base branch yet (expected on the first PR), or `fail-on: never`, or `base-branch` resolved empty. `warning: hosted policy check ...; falling back to fail-on: regression` → the policy-check endpoint errored, 404'd, or holds no decision for the run; the job gated on the diff instead and the PR comment says so. A persistent 404 usually means the project has no migration policy. Two comments on one PR → two action invocations, or the marker comment fell off page 1 of the comments API. Job is silent for minutes → output is captured and printed per command, not streamed. Install fails on `evalshift==0.12.1` → `python-version` below the CLI's `requires-python` (3.11). Costs higher than expected → the runner has a cold CLI cache every run; narrow the trigger with `on.pull_request.paths`, or shrink the suite. ## Versioning `@v0` tracks the latest v0.x. `@v0.1.0` pins exactly. `evalshift-version` pins the CLI independently of the action tag — pin both for a fully reproducible workflow. License: MIT (the EvalShift CLI it installs is licensed separately, AGPL-3.0-or-later).