## Security model
- +Secrets are masked and redacted. The hosted token and GitHub token are registered with GitHub’s log masking before anything else runs. On top of that, the Action redacts any environment value whose name contains
TOKENorSECRET, or ends inAPI_KEY, out of the CLI’s stdout and stderr before printing it — so a CLI that echoes a key in an error message doesn’t leak it into your logs. - +Tokens never appear in argv. The hosted token and host URL reach the CLI through the environment only, so they can’t show up in a process listing or a
command failed:message. - +Provider keys never leave the job. The Action passes them to the CLI and nowhere else. They are not uploaded to hosted EvalShift, not written to the bundle, not sent to GitHub.
- +The Action never writes to your repository. No commits, no pushes, no file mutations outside
.evalshift/in the workspace. - +Dependencies. The runtime helper is stdlib-only, and
pip-auditruns in the Action repo’s own CI.
pull_request from a fork, so the Action will fail on token being empty. That’s GitHub’s design, and working around it with pull_request_target means running untrusted code with your secrets in scope — don’t, unless you fully understand the exposure.## Limits and known edges
- +The PR comment lookup reads only the first page of comments. On a very long PR thread the EvalShift comment can fall off page one, and a second comment gets created instead of the first being updated.
- +The comment marker and status context are global constants. Parallel invocations on the same PR overwrite each other. One commenting invocation per PR.
- +The run id is the newest directory under
.evalshift/runs. If a step between the run and the push touches an older run directory’s mtime, the wrong run gets pushed. In a normal workflow this never happens. - +The hosted run URL is parsed from CLI stdout. A CLI release that changes how the push result is printed would break this; the Action repo’s
cli-contractCI job guards flag renames but not output shape. - +No retries on hosted API calls. A 30-second timeout, one attempt. A transient hosted outage fails the step rather than silently passing — deliberate, but it means a flaky network reads as a failed job.
- +
fail-ondecides the exit code, not whether the run happened. Even atnever, the run executes, costs money, and pushes.
## Troubleshooting
| Symptom | Cause and fix |
|---|---|
| input 'token' is required | The token: input is empty. Either the secret isn't set, or this is a fork PR where secrets aren't exposed. |
| command failed (1): evalshift all --yes ... | The CLI itself failed — bad config, missing provider key, model API error. The CLI's own (redacted) stderr is printed directly above this line. Reproduce with the same command locally. |
| no local EvalShift runs found in .../.evalshift/runs | evalshift all exited successfully but wrote nothing where the Action looks. Usually a config that redirects run artifacts elsewhere, or a working-directory mismatch. |
| evalshift push did not print a hosted run URL | The push didn't emit a URL on its last output line. Run evalshift push <run-id> locally against the same host and see what it prints. Also check for a CLI version mismatch. |
| HTTP 401 or 403 from the hosted API | Bad or expired EVALSHIFT_TOKEN, wrong host, or a project-scoped token trying to auto-create a project. Verify with evalshift whoami locally using the same token. |
| warning: could not upsert PR comment: HTTP 403 | Missing pull-requests: write / issues: write, or a fork PR with a read-only token. The gating still works — only the comment is lost. |
| The check is always green | In order of likelihood: no baseline run exists on the base branch yet (add the push trigger to main and merge once), fail-on is never, or base-branch resolved to an empty string. |
| Two EvalShift comments on one PR | Either two Action invocations are commenting, or the original comment fell off the first page of the comments API on a long thread. |
| The job hangs with no output | It doesn't — output is buffered per command and printed when each finishes. A slow suite is silent while it runs. |
| pip install evalshift==0.8.0 fails | python-version is below the CLI's minimum. EvalShift 0.8.0 needs Python 3.14+. |
| Costs are higher than expected | The runner cache is cold every run. Narrow the trigger with on.pull_request.paths, shrink the CI suite, or swap LLM-judge evaluators for structural ones in a CI-specific config. |
## FAQ
### Does this replace the CLI?
No. It runs the CLI. Everything you can inspect locally — report.html, analysis.json, the raw model outputs — is still produced, in the runner’s workspace under .evalshift/runs/.
### Can I use it without hosted EvalShift?
Not currently. The baseline lookup and the diff are server-side; without a hosted token there’s nothing to compare against. If you want local-only CI gating, use evalshift all directly plus a migration policy in your config, and skip the Action. See Command reference.
### Does it upload my model outputs?
It pushes the run bundle — manifest, examples, outputs, scores, analysis, and the HTML report — to hosted EvalShift. It never uploads provider API keys. If your suite contains sensitive production data, that’s the thing to weigh.
### Why did the check pass when the report clearly shows a regression?
Two common reasons: fail-on: never is set, or there was no compatible baseline so nothing was compared. The comment states which.
### Can I run it on a schedule instead of on PRs?
Yes — it works on any trigger. On non-PR events you get the commit status and the outputs but no comment. A nightly run against main is a reasonable way to catch provider-side model drift.
### Does it work on self-hosted runners?
Yes, provided the runner can install Python and reach PyPI, your model provider, and the hosted API.
### How long does a run take?
Install is 20–60 seconds. After that it’s however long your suite takes at your configured concurrency, times two models. A 40-example suite is typically a few minutes.