## Security model
- +Secrets are masked and redacted. The Cloud token and GitHub token are registered with GitHub’s log masking before anything else runs. On top of that, the Action redacts any environment value whose name contains
TOKENorSECRET, or ends inAPI_KEY, out of the CLI’s stdout and stderr before printing it — so a CLI that echoes a key in an error message doesn’t leak it into your logs. - +Tokens never appear in argv. The Cloud token and host URL reach the CLI through the environment only, so they can’t show up in a process listing or a
command failed:message. - +Provider keys never leave the job. The Action passes them to the CLI and nowhere else. They are not uploaded to EvalShift Cloud, not written to the bundle, not sent to GitHub.
- +The Action never writes to your repository. No commits, no pushes, no file mutations outside
.evalshift/in the workspace. - +Dependencies. The runtime helper is stdlib-only, and
pip-auditruns in the Action repo’s own CI.
pull_request from a fork, so the Action will fail on token being empty. That’s GitHub’s design, and working around it with pull_request_target means running untrusted code with your secrets in scope — don’t, unless you fully understand the exposure.## Limits and known edges
- +The PR comment lookup reads only the first page of comments. On a very long PR thread the EvalShift comment can fall off page one, and a second comment gets created instead of the first being updated.
- +The comment marker and status context are global constants. Parallel invocations on the same PR overwrite each other. One commenting invocation per PR.
- +The local run id is the newest directory under
.evalshift/runs. If a step between the run and the push touches an older run directory’s mtime, the wrong run gets pushed. In a normal workflow this never happens. - +The Cloud run URL is parsed from CLI stdout. A CLI release that changes how the push result is printed would break this; the Action repo’s
cli-contractCI job guards flag renames but not output shape. - +No retries on Cloud API calls. A 30-second timeout, one attempt. A transient Cloud outage fails the step rather than silently passing — deliberate, but it means a flaky network reads as a failed job.
- +
fail-ondecides the exit code, not whether the run happened. Even atnever, the run executes, costs money, and pushes.
## Troubleshooting
The log lines below quote evalshift all because that is what the Action invokes. all is the CLI’s permanent alias for compare and the one spelling every pinned CLI version answers to, including releases that predate the rename — so the Action stays on it deliberately. What you type locally is compare.
| Symptom | Cause and fix |
|---|---|
| input 'token' is required | The token: input is empty. Either the secret isn't set, or this is a fork PR where secrets aren't exposed. |
| command failed (1): evalshift all --yes ... | The CLI itself failed — bad config, missing provider key, model API error. The CLI's own (redacted) stderr is printed directly above this line. Reproduce with the same command locally. |
| no local EvalShift runs found in .../.evalshift/runs | evalshift all exited successfully but wrote nothing where the Action looks. Usually a config that redirects run artifacts elsewhere, or a working-directory mismatch. |
| evalshift push did not print a Cloud run URL | The push didn't emit a URL on its last output line. Run evalshift push <run-id> locally against the same host and see what it prints. Also check for a CLI version mismatch. |
| HTTP 401 or 403 from the Cloud API | Bad or expired EVALSHIFT_TOKEN, wrong host, or a project-scoped token trying to auto-create a project. Verify with evalshift whoami locally using the same token. |
| warning: could not upsert PR comment: HTTP 403 | Missing pull-requests: write / issues: write, or a fork PR with a read-only token. The gating still works — only the comment is lost. |
| The check is always green | In order of likelihood: no baseline run exists on the base branch yet (add the push trigger to main and merge once), fail-on is never, or base-branch resolved to an empty string. |
| Two EvalShift comments on one PR | Either two Action invocations are commenting, or the original comment fell off the first page of the comments API on a long thread. |
| The job hangs with no output | It doesn't — output is buffered per command and printed when each finishes. A slow suite is silent while it runs. |
| pip install evalshift==0.12.1 fails | python-version is below the CLI's minimum. EvalShift 0.12.1 needs Python 3.11+. |
| Costs are higher than expected | The runner cache is cold every run. Narrow the trigger with on.pull_request.paths, shrink the CI suite, or swap LLM-judge evaluators for structural ones in a CI-specific config. |
## FAQ
### Does this replace the CLI?
No. It runs the CLI. Everything you can inspect locally — report.html, analysis.json, the raw model outputs — is still produced, in the runner’s workspace under .evalshift/runs/.
### Can I use it without EvalShift Cloud?
Not currently. The baseline lookup and the diff are server-side; without a Cloud token there’s nothing to compare against. If you want local-only CI gating, use evalshift compare directly plus a migration policy in your config, and skip the Action. See Command reference.
### Does it upload my model outputs?
It pushes the run bundle — manifest, examples, outputs, scores, analysis, and the HTML report — to EvalShift Cloud. It never uploads provider API keys. If your suite contains sensitive production data, that’s the thing to weigh.
### Why did the check pass when the report clearly shows a regression?
Two common reasons: fail-on: never is set, or there was no compatible baseline so nothing was compared. The comment states which.
### Can I run it on a schedule instead of on PRs?
Yes — it works on any trigger. On non-PR events you get the commit status and the outputs but no comment. A nightly run against main is a reasonable way to catch provider-side model drift.
### Does it work on self-hosted runners?
Yes, provided the runner can install Python and reach PyPI, your model provider, and the Cloud API.
### How long does a run take?
Install is 20–60 seconds. After that it’s however long your suite takes at your configured concurrency, times two models. A 40-example suite is typically a few minutes.