// cli · reference
Command reference
every command & flag
Common conventions: -c/--config defaults to ./evalshift.yaml; run artefacts live under .evalshift/runs/; exit code 1 on handled errors.
## Pipeline
| Command | What it does | Flags |
|---|---|---|
| evalshift init | Scaffold a minimal capture-first evalshift.yaml. | -f/--force · -d/--directory · --ci · --wire-agents/--no-wire-agents · --provider gemini|openai|anthropic · --profile |
| evalshift demo | Scaffold the runnable 5-file demo project. | -f/--force · -d/--directory · --ci · --profile · --pack |
| evalshift doctor | Environment/config check. Exit 1 only on an invalid existing config. | — |
| evalshift run | Paired evaluation run (costs money unless --offline). | -f/--from · -t/--to · -c/--config · -s/--suite · --suite-name · --resume · -y/--yes · --offline · --fixtures |
| evalshift evaluate <run-id> | Score all pairs → scores.jsonl. | -c/--config |
| evalshift analyze <run-id> | Paired stats → analysis.json (+ migration_decision.json). | -c/--config · --gate <severities> · --policy-gate |
| evalshift report <run-id> | Render report.html + report.json. | -c/--config · --open |
| evalshift all | Full pipeline under one live display. | all run flags, plus --gate · --policy-gate · --open · --push |
## Hosted (opt-in)
Nothing leaves your machine unless you run these. See Hosted setup.
| Command | What it does | Flags |
|---|---|---|
| evalshift login | Device-code browser flow, or paste a token. | --token es_... · --host · --no-browser · --timeout |
| evalshift logout | Remove stored credentials. | — |
| evalshift whoami | Show the authenticated identity. | --host · --token |
| evalshift bundle <run-id> | Build the upload artefact (run_bundle.json.gz) without uploading. | -c/--config · -s/--suite · --suite-name · -o/--output · --project |
| evalshift push [run-id] | Upload a run bundle to the hosted service. Idempotent on run id. | --bundle · --project · --host · --token · --create-project/--no-create-project · -c/--config · -s/--suite · --suite-name |
## Captures
| Command | What it does | Flags |
|---|---|---|
| evalshift capture list [suite] | Table of recorded captures. | --json |
| evalshift capture promote <id> | Promote one capture into a golden case. | --as · --suite · --input-var · --tag · --strict-args · --names-only · --tool-count · -f/--force |
| evalshift capture sync | Promote ALL captures → suites + wire the managed suites: block in config. | --suite · --input-var · --tag · --strict-args · --names-only · --tool-count · -c/--config · -f/--force · --write/--print · --keep-duplicates |
| evalshift capture clean [suite] | Delete promoted capture files. | --promoted (default) · --all · -y/--yes |
| evalshift capture diff <a> <b> | Compare two capture tool traces. | — |
## Tools, traces, debugging
| Command | What it does | Flags |
|---|---|---|
| evalshift tools sync <defs.py>... | Extract tool definitions from Python source (AST-parsed, never imported) into .evalshift/tools.json and wire prompts. | -V/--variable · -o/--out · -p/--prompt · --config · --dry-run |
| evalshift traces import <run-id> | Attach external agent timelines to a run. | --source (required) · --target (required) · --strict |
| evalshift inspect <run-id> [case <example>] | Inspect a run or a single example. | --failed |
| evalshift diff case <run-id> <example> | Side-by-side trace diff when traces exist, text diff otherwise. | — |
| evalshift replay case <run-id> <example> | Re-run one example live. | --model source|target · --trace |
## Housekeeping & hidden
| Command | What it does | Flags |
|---|---|---|
| evalshift runs clean | Prune run history on demand. | --keep · --older-than · --suite · --dry-run · -y/--yes · --config |
| evalshift cache clear | Wipe the response cache. | — |
| evalshift validate | Load config + suite + prompts, cross-check compatibility (hidden). | -s/--suite · -c/--config |
| evalshift test-call | One live smoke-test call (hidden). | -m/--model (required) · -p/--prompt · -t/--temperature · --max-tokens · --tools |
## Environment variables
| Variable | Meaning |
|---|---|
| GEMINI_API_KEY / GOOGLE_API_KEY | Google auth (either works) |
| OPENAI_API_KEY | OpenAI auth (also the default semantic embedding model) |
| ANTHROPIC_API_KEY | Anthropic auth |
| EVALSHIFT_NONINTERACTIVE | Non-empty → skip the cost-confirmation prompt (implied --yes); set in scaffolded CI |
| EVALSHIFT_MAX_RUNS | Override retention.max_runs_per_suite; 0/none/unlimited/off disables count pruning |
| EVALSHIFT_DIR | Base dir for SDK captures the capture commands read (default .evalshift) |
| EVALSHIFT_HOST | Hosted API base URL (default https://api.evalshift.dev) |
| EVALSHIFT_TOKEN | Hosted token (beats the credentials file, loses to --token) |
| EVALSHIFT_CREDENTIALS_PATH | Credentials file override (default ~/.evalshift/credentials) |
| GITHUB_STEP_SUMMARY | When set, analyze appends a markdown results table |
Keys are consumed by LiteLLM at call time; EvalShift itself never stores or transmits them.