// cli · reference
Command reference
every command & flag
Common conventions: -c/--config defaults to ./evalshift.yaml; run artefacts live under .evalshift/runs/; exit code 1 on handled errors, 2 on most usage errors (an unknown flag, a malformed flag value). evalshift --version prints the installed version.
Pipeline
| Command | What it does | Flags |
|---|---|---|
| evalshift init | Scaffold a minimal capture-first evalshift.yaml with advisory semantic + llm_judge active for every provider (anthropic/deepseek borrow OpenAI's or Gemini's embeddings, and --ci wires that key as a second secret). Without --ci, warns after writing when an existing workflow under .github/workflows/ pins an evalshift-version older or newer than this CLI, or none (CI pin drift; advisory). | -f/--force · -d/--directory · --ci · --wire-agents/--no-wire-agents · --provider gemini|openai|anthropic|deepseek · --profile |
| evalshift doctor | Environment/config check: Python, the evalshift-sdk import, provider keys, each suite’s toolset, plus ci pin, judge family, captures.store (warns when the client library is missing or the bucket can’t be listed) and evaluator keys rows. Exit 1 only on an invalid existing config or a blocking evaluator whose model has no key (rows explained in the CLI overview). | — |
| evalshift run | Paired evaluation run (costs money — calls real models). Before any call, exits 1 when an arm’s API key or a blocking semantic/llm_judge model’s key is missing, or when every evaluator would be skipped; an advisory evaluator without a key is skipped with a warning. | -f/--from · -t/--to · -c/--config · -s/--suite · --suite-name · --resume · -y/--yes |
| evalshift evaluate <run-id> | Score all pairs → scores.jsonl. Prints a red “broken eval harness” row when the source model fails the recorded tool ground truth on half or more of the conformance rows. Warns when a trace_invariants applies_to glob matches no prompt in the run; fails when an entry sets traces: imported and the run has no traces.jsonl. Exits 1 when a blocking semantic/llm_judge model has no API key or every evaluator would be skipped; an advisory one without a key is skipped and recorded in state.json as skipped_evaluators. | -c/--config |
| evalshift analyze <run-id> | Paired stats → analysis.json (+ migration_decision.json). | -c/--config · --gate <severities> · --policy-gate |
| evalshift report <run-id> | Render report.html + report.json (+ insights.json — the machine-written run narrative). | -c/--config · --open · --insights/--no-insights |
| evalshift compare | Full pipeline under one live display, over one suite: two models, one comparison. Auto-selects the suite when exactly one is wired; with several, names them and prints a ready-to-run command per suite. Runs the same key checks as run before any call (blocking vs advisory). Formerly `all`, which stays registered as a hidden alias and keeps working. | every run flag, plus --gate · --policy-gate · --open · --push · --insights/--no-insights |
Cloud (opt-in)
Nothing leaves your machine unless you run these. See Cloud setup.
| Command | What it does | Flags |
|---|---|---|
| evalshift login | Device-code browser flow, or paste a token. Keeps a stored token that still works; one whose account is suspended exits 1 with the server's message. | --token es_... · --host · --no-browser · --timeout |
| evalshift logout | Remove stored credentials. | — |
| evalshift whoami | Show the authenticated identity. | --host · --token |
| evalshift bundle <run-id> | Build the upload artefact (run_bundle.json.gz) without uploading. | -c/--config · -s/--suite · --suite-name · -o/--output · --project |
| evalshift push [run-id] | Upload a run bundle to EvalShift Cloud. Idempotent per project on the local run id; the Cloud run gets its own server-minted id. | --bundle · --project · --host · --token · --create-project/--no-create-project · -c/--config · -s/--suite · --suite-name |
Captures
| Command | What it does | Flags |
|---|---|---|
| evalshift capture list [suite] | Table of recorded captures. With captures.store set, fetches new captures from the bucket first (--offline skips it); --json keeps stdout pure JSON, the fetch line goes to stderr. | --json · -c/--config · --offline |
| evalshift capture fetch | Mirror new captures and toolset sidecars from captures.store into .evalshift/, and nothing else: only objects missing locally are downloaded, captures already promoted are skipped, every file is written atomically. Needs captures.store; a missing client library (the hint names the package to pip install), or a failed list or download, exits 1 with a hint. | --suite · --since <30m|24h|7d|ISO> · -c/--config |
| evalshift capture promote <id> | Promote one capture into a golden case. Exits 1, naming the reason, on a capture that can’t be ground truth — an errored turn (--allow-errored overrides), content a text replay can’t carry, no usable toolset, or no valid case (details). | --as · --suite · --input-var · --tag · --strict-args · --names-only · --tool-count · --rounds first|all · --allow-errored · -f/--force |
| evalshift capture sync | Promote ALL captures → suites + wire the managed suites: block, keeping hand-written trace_invariants. Fetches from captures.store first when set (a failed fetch exits 1). Skips duplicates and captures promote would refuse; --repair rebuilds cases an earlier sync stringified; warns on CI pin drift (details). | --suite · --input-var · --tag · --strict-args · --names-only · --tool-count · --rounds first|all · --allow-errored · -c/--config · -f/--force · --write/--print · --keep-duplicates · --repair · --since · --offline |
| evalshift capture clean [suite] | Delete promoted capture files + sweep orphaned toolset sidecars. A sidecar a promoted suite still uses is never touched. | --promoted (default) · --all · -y/--yes |
| evalshift capture diff <a> <b> | Compare two capture tool traces. | — |
Traces, debugging
| Command | What it does | Flags |
|---|---|---|
| evalshift traces import <run-id> | Attach external agent timelines to a completed run, for the agent_trace evaluator and for trace_invariants rule sets with traces: imported. Imported traces stay local — they are not part of the pushed bundle. | --source (required) · --target (required) · --strict |
| evalshift inspect <run-id> | Inspect a run. | --failed |
| evalshift inspect case <run-id> <example> | Inspect a single example of a run. | — |
| evalshift diff case <run-id> <example> | Trace diff when imported traces exist; otherwise prints the source and target outputs one after the other. | — |
| evalshift replay case <run-id> <example> | Print the recorded source or target output for one example — no live model call. --trace prints the imported agent trace instead. | --model source|target · --trace |
Housekeeping & hidden
| Command | What it does | Flags |
|---|---|---|
| evalshift runs clean | Prune run history on demand. | --keep · --older-than · --suite · --dry-run · -y/--yes · --config |
| evalshift cache clear | Wipe the response cache. | — |
| evalshift validate | Load config + suite + prompts, cross-check compatibility (hidden). After the success line prints the CI pin drift, stringified-case (run capture sync --repair) and judge family warnings, if any (advisory, exit code unchanged). | -s/--suite · -c/--config |
| evalshift test-call | One live smoke-test call (hidden). | -m/--model (required) · -p/--prompt · -t/--temperature · --max-tokens · --tools |
Environment variables
| Variable | Meaning |
|---|---|
| GEMINI_API_KEY / GOOGLE_API_KEY | Google auth (either works) |
| OPENAI_API_KEY | OpenAI auth (also the default semantic embedding model, and the one Anthropic/DeepSeek scaffolds borrow when it is set; a judge or embedding key is checked before the run) |
| ANTHROPIC_API_KEY | Anthropic auth |
| DEEPSEEK_API_KEY | DeepSeek auth |
| EVALSHIFT_NONINTERACTIVE | Non-empty → skip the cost-confirmation prompt (implied --yes); set in scaffolded CI |
| EVALSHIFT_MAX_RUNS | Override retention.max_runs_per_suite; 0/none/unlimited/off disables count pruning |
| EVALSHIFT_DIR | Base dir for SDK captures the capture commands read (default .evalshift) |
| EVALSHIFT_HOST | Cloud API base URL (default https://api.evalshift.dev) |
| EVALSHIFT_TOKEN | Cloud token (beats the credentials file, loses to --token) |
| EVALSHIFT_CREDENTIALS_PATH | Credentials file override (default ~/.evalshift/credentials) |
| GITHUB_STEP_SUMMARY | When set, analyze appends a markdown results table |
| GITHUB_SHA / GITHUB_HEAD_REF / GITHUB_REF_NAME / GITHUB_REF / GITHUB_EVENT_PATH | Read by bundle and push for the run's git metadata: the commit SHA (else git rev-parse HEAD), the branch (head ref, then ref name, else the current git branch), and the PR number (from GITHUB_REF, else the event payload) |
Keys are consumed by LiteLLM at call time; EvalShift itself never stores or transmits them.