Inputs & outputs — EvalShift docs

// action · reference

Inputs & outputs

the full surface

Everything babaliauskas/evalshift-action@v0 accepts and produces. Boolean inputs accept 1, true, yes, on (case-insensitive); anything else is false.

## Inputs

Input	Default	What it does
token	— (required)	Hosted EvalShift API token, an es_... value. Masked in logs and redacted from CLI output.
host	https://api.evalshift.dev	Hosted API base URL. Set only for a self-hosted or staging deployment.
config	evalshift.yaml	Path to your config, relative to the repository root. Paths inside the config resolve relative to the config file's own directory, so a config in a subdirectory works.
suite	golden.jsonl	Path to the golden JSONL suite, relative to the repository root.
evalshift-version	0.8.0	Exact CLI version installed from PyPI. Pin this for run-to-run reproducibility across CLI releases.
python-version	3.14	Python used to install and run the CLI. Must satisfy the CLI's minimum (3.14 for 0.8.0).
fail-on	regression	Gating mode: never, regression, or any-slice-regression.
branch	auto	Candidate branch name recorded on the hosted run. Auto-detected from the PR head ref, else the pushed ref.
base-branch	auto	Branch to look for a baseline run on. Auto-detected from the PR base ref, else the current ref. Resolving to empty means no baseline is fetched and the check always passes.
create-project	true	Whether evalshift push may auto-create the hosted project when it doesn't exist. Set false to make a missing project a hard failure.
comment	true	Whether to create or update the PR comment. Set false to keep the commit status but stay out of the conversation.
github-token	github.token	Token used for the PR comment and the commit status. Override only to have a bot account post instead of github-actions.

## Outputs

Output	Value
run_url	Hosted run URL for this run.
diff_url	Hosted diff URL comparing this run to the baseline. Empty string when no compatible baseline was found.
run_id	EvalShift run id, usable with evalshift CLI commands locally.
regression_count	Number of regressed examples in the hosted diff. 0 when there is no baseline.
conclusion	success or failure, reflecting fail-on.

Consume them from a later step:

yaml

      - uses: babaliauskas/evalshift-action@v0
        id: evalshift
        with:
          token: ${{ secrets.EVALSHIFT_TOKEN }}
      - run: echo "Hosted diff ${{ steps.evalshift.outputs.diff_url }}"

Outputs are written before the comment and status calls, so they are still available even if the job lacks permission to comment.

## Permissions

Permission	Why
contents: read	Checking out the repository.
pull-requests: write	Posting the PR comment.
issues: write	PR comments are issue comments in the GitHub API.
statuses: write	Setting the evalshift/regression commit status.

Only contents: read is strictly required. If the comment or status permissions are missing, the Action logs a warning and carries on rather than failing the run — the gate still works.

## Versioning and stability

Pin to @v0 to track the latest v0.x, or to an exact tag such as @v0.1.0 for a fully reproducible workflow. The evalshift-version input pins the CLI separately — pin both if you want a workflow that behaves identically six months from now.

wrapper, not reimplementation

All evaluation and statistics happen in the CLI; all cross-branch diffing happens server-side. To understand what regression_count actually means, read the statistical methodology — it is paired statistics with Benjamini–Hochberg correction, not a threshold on an average.