Run your prompts on two LLMs and find out, with statistical confidence, what regressed.

EvalShift is a local-first CLI that helps engineering teams migrate safely between LLM versions. Point it at your prompts and a golden suite of inputs; it runs both models, scores the outputs with structural / semantic / LLM-as-judge evaluators, and produces a single-file HTML report with defensible statistics: paired tests, Cohen’s d, 95% CIs, and Benjamini–Hochberg correction across every (prompt × evaluator × slice) comparison.

## the 60-second start

pip install evalshift
evalshift demo                        # scaffold a runnable demo project
evalshift all --offline --yes --open  # full pipeline — free, no keys, no account

all chains the pipeline — doctor → run → evaluate → analyze → report. Each stage writes its artefact under .evalshift/runs/<run-id>/:

## two ways to use it

Local, free, no account — the default. Runs, scoring, statistics, and reports all happen on your machine under .evalshift/; the only network traffic is the model API calls you asked for, with your own provider keys. Start at Getting started, and pair the CLI with the capture SDK for the recommended workflow: record real agent behaviour in production, promote it into a golden suite, evaluate candidates against it.

Hosted, opt-in — for teams. Push immutable run bundles so everyone can view timelines, reports, diffs, and PR comments in one place. Model calls still run locally or in your CI; nothing is uploaded unless you run push. See Hosted setup.

## where to next

Or jump straight to the landing page for the elevator pitch and a live diff.