The evalshift CLI is a five-stage pipeline. You point it at a golden suite of examples; it runs the same prompts on two models — your current production model and the candidate — scores both sides with structural, semantic, LLM-as-judge, and tool-call evaluators, then runs paired statistics over the deltas and tells you, with confidence intervals and multiple-comparison correction, what actually regressed.

It is local-first: every stage runs on your machine and writes its artefact under .evalshift/runs/<run-id>/. The only network calls are the model API calls you asked for — and, if you opt in, pushes to the EvalShift Cloud.

The CLI is open source under Apache-2.0 — the code lives at babaliauskas/evalshift-cli ↗.

pip install evalshift installs the CLI (import package evalshift_cli; python -m evalshift_cli is the module form of the evalshift binary) and, since 0.14.0, the capture SDK it depends on (package evalshift-sdk, import name evalshift) — both live in one environment. Scripts that imported CLI internals from evalshift.* must import from evalshift_cli.

## The pipeline

init        →  doctor  →  run        →  evaluate      →  analyze          →  report
(scaffold)     (checks)   raw.jsonl     scores.jsonl     analysis.json        report.html
                          state.json                     migration_decision   report.json
                                                         .json (if policy)

• +doctor — validates config and shows which provider keys are visible. Missing keys are soft warnings. It also reports the toolset each configured suite carries, and flags a suite whose examples carry more than one distinct toolset — legal, but also the shape a wiring mistake takes. An evalshift-sdk row reports which package the evalshift import name resolves to (warn when the SDK is missing or shadowed by an older CLI’s leftover files; never a failure), a ci pin row warns when a workflow pins an older, newer, or no evalshift-version, and a judge family row warns when an llm_judge judge shares a provider with the source or target model (self-preference bias; advisory, validate prints the same line).
• +run — validates every example against every prompt, estimates cost, then dispatches (prompt × example × {source, target}) calls through an async orchestrator. Progress is checkpointed every 50 completions.
• +evaluate — scores each (source, target) pair with the configured evaluators.
• +analyze — paired statistics per (prompt, evaluator, slice), Benjamini–Hochberg FDR correction, severity classification, and the migration-policy verdict when configured. See Methodology.
• +report — renders a single-file HTML report (no external assets; works offline and attaches cleanly to a PR), topped by a plain-language narrative of the run: one summary each for the verdict, the advisory signal and the economics, plus behavioural findings taken from the worst regressions. The prose is machine-written; every figure in it is computed first, handed to the model pre-rendered, and validated on the way out, so the narrative cannot contain a derived or invented number. One extra model call, cached — skip it with --no-insights. The header shows an N samples per example pill when defaults.samples_per_example is above 1, and up to three banners above the verdict: models that no longer honour temperature (on a single-sample run the banner suggests samples_per_example), Constraints not honoured for generation parameters an arm could not accept, and a judge that shares a family with an arm. report.json carries samples_per_example, dropped_params and judge_family_overlap.
• +compare — chains everything end to end over one suite and adds --gate, --policy-gate, --push, --open. Formerly all; that name is hidden now but still works.

Every stage is independently re-runnable: each writes one artefact and the next stage reads it.

## Project setup: init

evalshift init is the entry point. It writes only a minimal, capture-first evalshift.yaml: a passthrough replay prompt, advisory semantic + LLM-judge evaluators, an empty managed suites: block for capture sync to fill, and a migration policy. The intended flow: instrument your agent with the capture SDK, record captures, promote them with evalshift capture sync, run against the promoted suite. Options: --provider, --profile (pre-tuned policy budgets: model-upgrade, cost-reduction, local-model, quantization, provider-switch), --ci (scaffold the GitHub Action), --wire-agents (write EVALSHIFT.md and point AI-agent files at it).

## Response cache

Live responses are cached in SQLite at ~/.evalshift/cache.db, keyed by a hash of (model, prompt, inputs, temperature, max_tokens, history), plus sample_index and round_index when set — a single-sample, single-round run keeps its existing keys — with a 7-day TTL. Re-running an identical evaluation is nearly free. Disable per-project with defaults.cache: false; wipe with evalshift cache clear.

## Checkpointing and resume

state.json records a hash over the canonicalised config plus the suite path. run --resume picks up the most recent in-progress run, verifies the hash still matches (aborts if config or suite changed — a changed config would corrupt the pairing), and skips every call already present in raw.jsonl. Errored calls are counted as done in a resumed run — start a fresh run to retry them.

## Run retention

Run history is pruned automatically after every completed run, per suite: keep the newest retention.max_runs_per_suite (default 20), optionally evict runs older than retention.run_ttl_days. In-progress runs and the run just finished are never pruned. evalshift runs clean prunes on demand.

## Which models can I use?

Anything LiteLLM supports. A small built-in registry maps aliases to canonical provider/model ids, and unknown ids pass through with the provider inferred from the prefix (gemini-* → Google, claude-* → Anthropic, gpt-*/o1-*/o3-* → OpenAI). The registry never gates. Verify a model with evalshift test-call -m <id>.