EvalShift
Login
// docs

EvalShift documentation

docs · cli 1.6.2

Run your prompts on two LLMs and find out, with statistical confidence, what regressed.

// video EvalShift getting started: catch model migration regressions in 4 minutes

EvalShift is a local-first CLI that helps engineering teams migrate safely between LLM versions. Point it at your prompts and a golden suite of inputs; it runs both models, scores the outputs with structural / semantic / LLM-as-judge / tool-call / trace-rule evaluators, and produces a single-file HTML report with defensible statistics: paired tests, Cohen’s d, 95% CIs, and Benjamini–Hochberg correction across every (prompt × evaluator × slice) comparison.

the short version

bash
pip install evalshift
evalshift init                        # capture-first config
evalshift capture sync                # recorded captures -> golden suite
evalshift compare --suite-name <suite> --to <candidate-model> --open

compare chains the pipeline — doctor → run → evaluate → analyze → report — over one suite. Each stage writes its artefact under .evalshift/runs/<run-id>/:

StageArtefact
runraw.jsonl
evaluatescores.jsonl
analyzeanalysis.json
reportreport.html + report.json

two ways to use it

Local, free, no account — the default. Runs, scoring, statistics, and reports all happen on your machine under .evalshift/; the only network traffic is the model API calls you asked for, with your own provider keys. Start at Getting started, and pair the CLI with the capture SDK for the recommended workflow: record real agent behaviour in production, promote it into a golden suite, evaluate candidates against it.

Cloud, opt-in — for teams. Push immutable run bundles so everyone can view timelines, reports, diffs, and PR comments in one place. Model calls still run locally or in your CI; nothing is uploaded unless you run push. See Cloud setup.

open source

The three components you install are open source — read the code, file issues, or send patches:

where to next

+Getting started
install, init, capture real traffic, first paired run.
+Recommended workflow
capture with the SDK, evaluate with the CLI — the full loop.
+CLI overview
the five-stage pipeline, run artefacts, cache, and resume.
+Configuration
every evalshift.yaml field, type, and default.
+Golden suite
the JSONL example format — inputs, tags, expected tools, history.
+Evaluators
structural, semantic, llm-judge, tool-call, hand-written trace rules — and when to use each.
+Agent migrations
compare which tools fired, what args, and how they sequenced.
+Command reference
every evalshift command and its flags, in one place.
+Capture SDK
record real agent behavior in-process with the evalshift-sdk library.
+Instrumenting agents
@capture.agent, tools, model calls — sync, async, and streaming.
+Framework adapters
provider client wrappers (OpenAI, Anthropic, Gemini) and the LangChain callback handler — record model calls without per-call code.
+Redaction & data boundary
mask PII in-process before any byte hits disk.
+Config, sinks & hygiene
env gate, sinks (local disk or an object store), sampling, dedup, and capture GC.
+Captures → golden cases
evalshift capture sync / list / fetch / promote / clean / diff — the promote loop.
+GitHub Action
optional: turn the golden suite into a merge gate on every PR.
+Inputs & outputs
every input, output, and permission — plus version pinning.
+Gating & PR feedback
fail-on modes, the PR comment, commit status, baseline resolution.
+Cost control & recipes
path filters, CI-sized suites, two-suite repos, required checks.
+Security & troubleshooting
secret redaction, known edges, and what each failure message means.
+Cloud setup
opt-in: account, token, first push, and the GitHub Action.
+What gets uploaded
the push data contract — every field that uploads, and what never leaves your machine.
+Verdicts & gating
PASS / CONDITIONAL / FAIL / INCONCLUSIVE — what each means and the gate rule.
+Baselines & diffs
pin a baseline run; every new run auto-diffs against it.
+Trends & slices
verdict, regression, and cost/latency over time + the slice heatmap.
+Migration policy & PR gating
the ten budgets, the per-run snapshot, and per-PR enforcement.
+Share links
public, expiring, revocable links to a run report.
+Account & organizations
renaming, transferring ownership, and what each deletion removes.
+Webhooks
signed run and baseline events posted to your HTTPS endpoint.
+Methodology
paired tests, Cohen's d, BH-FDR correction, severity tiers.
+FAQ
the questions that come up after first run.
+Changelog
release notes — mirrored from GitHub.

Or jump straight to the landing page for the elevator pitch and a live diff.