Run your prompts on two LLMs and find out, with statistical confidence, what regressed.

EvalShift is a local-first CLI that helps engineering teams migrate safely between LLM versions. Point it at your prompts and a golden suite of inputs; it runs both models, scores the outputs with structural / semantic / LLM-as-judge evaluators, and produces a single-file HTML report with defensible statistics: paired tests, Cohen’s d, 95% CIs, and Benjamini–Hochberg correction across every (prompt × evaluator × slice) comparison.

## the short version

pip install evalshift
evalshift init                        # capture-first config
evalshift capture sync                # recorded captures -> golden suite
evalshift compare --suite-name <suite> --to <candidate-model> --open

compare chains the pipeline — doctor → run → evaluate → analyze → report — over one suite. Each stage writes its artefact under .evalshift/runs/<run-id>/:

## two ways to use it

Local, free, no account — the default. Runs, scoring, statistics, and reports all happen on your machine under .evalshift/; the only network traffic is the model API calls you asked for, with your own provider keys. Start at Getting started, and pair the CLI with the capture SDK for the recommended workflow: record real agent behaviour in production, promote it into a golden suite, evaluate candidates against it.

Cloud, opt-in — for teams. Push immutable run bundles so everyone can view timelines, reports, diffs, and PR comments in one place. Model calls still run locally or in your CI; nothing is uploaded unless you run push. See Cloud setup.

## open source

The three components you install are open source — read the code, file issues, or send patches:

• +evalshift-cli ↗ — the pipeline itself (Apache-2.0).
• +evalshift-sdk ↗ — the capture SDK (MIT).
• +evalshift-action ↗ — the GitHub Action (MIT).

## where to next

Or jump straight to the landing page for the elevator pitch and a live diff.