CLI overview

The evalshift CLI is a five-stage pipeline. You point it at a golden suite of examples; it runs the same prompts on two models — your current production model and the candidate — scores both sides with structural, semantic, LLM-as-judge, tool-call and trace-rule evaluators, then runs paired statistics over the deltas and tells you, with confidence intervals and multiple-comparison correction, what actually regressed.

It is local-first: every stage runs on your machine and writes its artefact under .evalshift/runs/<run-id>/. The only network calls are the model API calls you asked for — and, if you opt in, pushes to the EvalShift Cloud.

The CLI is open source under Apache-2.0 — the code lives at evalshift/evalshift-cli ↗.

pip install evalshift installs the CLI (import package evalshift_cli; python -m evalshift_cli is the module form of the evalshift binary) and, since 0.14.0, the capture SDK it depends on (package evalshift-sdk, import name evalshift) — both live in one environment. Scripts that imported CLI internals from evalshift.* must import from evalshift_cli.

The pipeline

init        →  doctor  →  run        →  evaluate      →  analyze          →  report
(scaffold)     (checks)   raw.jsonl     scores.jsonl     analysis.json        report.html
                          state.json                     migration_decision   report.json
                                                         .json (if policy)

• +doctor — validates config and shows which provider keys are visible. Missing provider keys are soft warnings. It also reports the toolset each configured suite carries, and flags a suite whose examples carry more than one distinct toolset — legal, but also the shape a wiring mistake takes. An evalshift-sdk row reports which package the evalshift import name resolves to (warn when the SDK is missing or shadowed by an older CLI’s leftover files; never a failure), a ci pin row warns when a workflow pins an older, newer, or no evalshift-version, a judge family row warns when an llm_judge judge shares a provider with the source or target model (self-preference bias; advisory, validate prints the same line), a captures.store row, when the config names a store, warns when its client library is missing (naming the package to pip install) or the bucket cannot be listed — never a failure, since a run on a committed suite never reads the bucket — and an evaluator keys row names each semantic or llm_judge model with no API key: a warning when the evaluator is advisory, a failure (exit 1) when it is blocking.
• +run — checks the API keys of both arms and of every judge and embedding model (a blocking one without a key exits 1 before any call; an advisory one is skipped with a warning), validates every example against every prompt, estimates cost, then dispatches (prompt × example × {source, target}) calls through an async orchestrator. Progress is checkpointed every 50 completions.
• +evaluate — scores each (source, target) pair with the configured evaluators. An advisory semantic or llm_judge whose model has no API key is skipped and recorded as skipped_evaluators; a blocking one stops the stage (exit 1).
• +analyze — paired statistics per (prompt, evaluator, slice), Benjamini–Hochberg FDR correction, severity classification, and the migration-policy verdict when configured. See Methodology.
• +report — renders a single-file HTML report (no external assets; works offline and attaches cleanly to a PR), topped by a plain-language narrative of the run: one summary each for the verdict, the advisory signal and the economics, plus behavioural findings taken from the worst regressions. The prose is machine-written; every figure in it is computed first, handed to the model pre-rendered, and validated on the way out, so the narrative cannot contain a derived or invented number. The model is told which budgets were breached and which held, and a FAIL summary that blames a budget that held is rejected and retried, then replaced by templated prose. One extra model call, cached — skip it with --no-insights. The cache is keyed on the figures the narrative describes, so re-running evaluate or analyze after an evalshift.yaml edit regenerates it on the next report, while a plain re-render reuses it. Top regression causes and the narrative’s “most common failure” count each failure label once per example, not per scored row. On an interactive terminal, report and compare end with a one-line request to star the project on GitHub; piped and CI output never carry it. The header shows an N samples per example pill when defaults.samples_per_example is above 1, and up to three banners above the verdict: models that no longer honour temperature (on a single-sample run the banner suggests samples_per_example), Constraints not honoured for generation parameters an arm could not accept, and a judge that shares a family with an arm. report.json carries samples_per_example, dropped_params, judge_family_overlap and a top-level invariant_violations list, one row per broken trace rule: {prompt_id, example_id, evaluator_name, owner, rule_id, rule_type, tool, round_index, detail, source_also, blocking, sample}.
• +compare — chains everything end to end over one suite and adds --gate, --policy-gate, --push, --open. Formerly all; that name is hidden now but still works.

Every stage is independently re-runnable: each writes one artefact and the next stage reads it.

Project setup: init

evalshift init is the entry point. It writes only a minimal, capture-first evalshift.yaml: a passthrough replay prompt, advisory semantic + LLM-judge evaluators (both active for every provider — Anthropic and DeepSeek have no embeddings endpoint, so semantic borrows OpenAI’s or Gemini’s, whichever key you already have), an empty managed suites: block for capture sync to fill, and a migration policy. The intended flow: instrument your agent with the capture SDK, record captures, promote them with evalshift capture sync, run against the promoted suite. Options: --provider, --profile (pre-tuned policy budgets: model-upgrade, cost-reduction, local-model, quantization, provider-switch), --ci (scaffold the GitHub Action; with a borrowed embedder it wires that key as a second secret), --wire-agents (write EVALSHIFT.md and point AI-agent files at it). Since CLI 1.3.0 every --profile policy block includes max_invariant_violations: 0; older CLIs reject that key, so bump the GitHub Action’s evalshift-version pin to 1.3.0 or later before committing such a config.

Response cache

Live responses are cached in SQLite at ~/.evalshift/cache.db, keyed by a hash of (model, prompt, inputs, temperature, max_tokens, history), plus the generation_config, the tools array as sent, round_index and sample_index when set — a single-sample, single-round run keeps its existing keys — with a 7-day TTL. Re-running an identical evaluation makes no run-stage calls, agent suites included, and the evaluate stage’s embedding and judge caches apply too. Disable per-project with defaults.cache: false; wipe with evalshift cache clear.

An example with a non-empty toolset (toolset_ref or inline tools) gets one entry per replayed round, keyed on the canonical model, the prompt and inputs, the exact message list that round sends (history, current turn, and the recorded rounds and fixture results teacher forcing feeds back), the tool list exactly as sent, in order (tool names, descriptions, schemas and strict — reordering the tools is a miss), generation_config (so tool_choice and parallel_tool_calls), the effective temperature and max_tokens, the round index and the sample index. Editing a fixture re-sends only the rounds after it. A hit restores the parsed tool trace (call ids, arguments, final text, refusal), tokens, cost, latency and finish reason, so the raw.jsonl row matches the live one apart from cached, which is true only when every round was a hit, and cached_rounds. A row with any round served from the cache carries latency measured on an earlier run, so it stays out of the live latency figures and its latency delta is marked not comparable. Errors are never cached: the next run re-sends a failed round and serves the rounds before it from the cache. Truncated responses are cached and stay flagged. A cache.db written by an earlier version is upgraded in place and keeps its entries, and several evalshift processes may open it at once.

Checkpointing and resume

state.json records a hash over the canonicalised config plus the suite path. run --resume picks up the most recent in-progress run, verifies the hash still matches (aborts if config or suite changed — a changed config would corrupt the pairing), and skips every call already present in raw.jsonl. Errored calls are counted as done in a resumed run — start a fresh run to retry them.

Run retention

Run history is pruned automatically after every completed run, per suite: keep the newest retention.max_runs_per_suite (default 20), optionally evict runs older than retention.run_ttl_days. In-progress runs and the run just finished are never pruned. evalshift runs clean prunes on demand.

Which models can I use?

Anything LiteLLM supports. A small built-in registry maps aliases to canonical provider/model ids, and unknown ids pass through with the provider inferred from the prefix (gemini-* → Google, claude-* → Anthropic, gpt-*/o1-*/o3-* → OpenAI, deepseek-* → DeepSeek, text-embedding-* → OpenAI). The registry never gates. Verify a model with evalshift test-call -m <id>.