The hosted service stores pushed runs, diffs them across branches, and comments on PRs — useful once a team wants shared reports, baselines, and PR gating. It is strictly opt-in: the CLI works fully without an account, model calls always run locally with your own provider keys, and nothing leaves your machine unless you run push (or all --push). Provider API keys are never uploaded.

## 1. Sign up and create a token

Open the hosted app, create an account, then create an API token from your org settings. For CI, prefer a project-scoped token when the project already exists; use an org-scoped token if the Action should auto-create the project.

## 2. Authenticate the CLI

evalshift login                       # device-code browser flow
evalshift login --token es_...        # or paste a token (verified via GET /me)
evalshift whoami

Credentials live in ~/.evalshift/credentials (owner-only permissions). Precedence: CLI flags (--host/--token) > env (EVALSHIFT_HOST/EVALSHIFT_TOKEN) > credentials file. --no-browser prints the approval URL for remote shells.

## 3. Point config at a project

Projects are org/project slugs — from --project, or the project: key in config. Missing projects are auto-created when permissions allow. Add the project path and the regression budget you want the hosted diff to enforce:

version: 1
project: acme/customer-router
migration_policy:
  max_overall_regression_rate: 0.03
  max_critical_regressions: 0
  min_equivalence_rate: 0.95

## 4. Push the first hosted run

all --push runs the local pipeline, builds run_bundle.json.gz, uploads it, finalizes it, and prints the hosted run URL. Pushes are idempotent on run id, and evalshift bundle builds the artefact without uploading if you want to inspect it first.

evalshift all --yes --push

# or push an existing local run
evalshift push <run-id>

## 5. Install the GitHub Action

evalshift init --ci writes this shape of workflow. The Action pushes the candidate run, finds a compatible baseline on the PR base branch, posts or updates one PR comment, and sets the evalshift/regression commit status. Add EVALSHIFT_TOKEN plus your provider key (e.g. GEMINI_API_KEY) as GitHub repository secrets. Full inputs, gating modes, and recipes live in the GitHub Action section.

permissions:
  contents: read
  pull-requests: write
  issues: write
  statuses: write

steps:
  - uses: actions/checkout@v4
  - uses: babaliauskas/evalshift-action@v0
    with:
      token: ${{ secrets.EVALSHIFT_TOKEN }}
      fail-on: regression

## 6. Review diffs and thresholds

Open the run URL to inspect the report. When a compatible baseline exists, the PR comment links to a hosted diff with aggregate, slice, and per-example deltas. fail-on: regression fails only when the hosted diff reports regressed examples.

## Troubleshooting

• +401 from the CLI or Action means the token is missing, revoked, or scoped to the wrong project.
• +No diff link means no compatible baseline exists yet on the base branch. Push one run on main, then open another PR.
• +Browser run viewing needs storage CORS to allow the web origin.