The evalshift CLI is a five-stage pipeline. You point it at a golden suite of examples; it runs the same prompts on two models — your current production model and the candidate — scores both sides with structural, semantic, LLM-as-judge, and tool-call evaluators, then runs paired statistics over the deltas and tells you, with confidence intervals and multiple-comparison correction, what actually regressed.

It is local-first: every stage runs on your machine and writes its artefact under .evalshift/runs/<run-id>/. The only network calls are the model API calls you asked for — and, if you opt in, pushes to the hosted service.

## The pipeline

init / demo  →  doctor  →  run        →  evaluate      →  analyze          →  report
(scaffold)      (checks)    raw.jsonl     scores.jsonl     analysis.json        report.html
                            state.json                     migration_decision   report.json
                                                           .json (if policy)

• +doctor — validates config and shows which provider keys are visible. Missing keys are soft warnings.
• +run — validates every example against every prompt, estimates cost, then dispatches (prompt × example × {source, target}) calls through an async orchestrator. Progress is checkpointed every 50 completions.
• +evaluate — scores each (source, target) pair with the configured evaluators.
• +analyze — paired statistics per (prompt, evaluator, slice), Benjamini–Hochberg FDR correction, severity classification, and the migration-policy verdict when configured. See Methodology.
• +report — renders a single-file HTML report (no external assets; works offline and attaches cleanly to a PR).
• +all — chains everything end to end and adds --gate, --policy-gate, --push, --open.

Every stage is independently re-runnable: each writes one artefact and the next stage reads it.

## init vs demo

Two scaffolds with different jobs:

• +evalshift init — the real-project entry point. Writes only a minimal, capture-first evalshift.yaml: a passthrough replay prompt, advisory semantic + LLM-judge evaluators, an empty managed suites: block for capture sync to fill, and a migration policy. Options: --provider, --profile (pre-tuned policy budgets: model-upgrade, cost-reduction, local-model, quantization, provider-switch), --ci (scaffold the GitHub Action), --wire-agents (write EVALSHIFT.md and point AI-agent files at it).
• +evalshift demo — a runnable worked example: full 5-file customer-support agent project including fixtures for offline replay. --pack selects the scenario pack.

## Response cache

Live responses are cached in SQLite at ~/.evalshift/cache.db, keyed by a hash of (model, prompt, inputs, temperature, max_tokens, history), with a 7-day TTL. Re-running an identical evaluation is nearly free. Disable per-project with defaults.cache: false; wipe with evalshift cache clear.

## Checkpointing and resume

state.json records a hash over the canonicalised config plus the suite path. run --resume picks up the most recent in-progress run, verifies the hash still matches (aborts if config or suite changed — a changed config would corrupt the pairing), and skips every call already present in raw.jsonl. Errored calls are counted as done in a resumed run — start a fresh run to retry them.

## Run retention

Run history is pruned automatically after every completed run, per suite: keep the newest retention.max_runs_per_suite (default 20), optionally evict runs older than retention.run_ttl_days. In-progress runs and the run just finished are never pruned. evalshift runs clean prunes on demand.

## Which models can I use?

Anything LiteLLM supports. A small built-in registry maps aliases to canonical provider/model ids, and unknown ids pass through with the provider inferred from the prefix (gemini-* → Google, claude-* → Anthropic, gpt-*/o1-*/o3-* → OpenAI). The registry never gates. Verify a model with evalshift test-call -m <id>.