$ evalshift diff baseline.. candidate
− Don't ship a model upgrade on vibes.
+ Read the diff. Ship with proof.
EvalShift runs both models against the same golden case, then shows you exactly what changed — outputs, tool calls, scores — line by line, with paired statistics that say whether the change is real. Then it fails the pull request when the answer is no.
−baseline · gemini-2.5-flash
case 17 of 40@@ case 17 — "refund a duplicate charge" @@tool: lookup_order(order_id="A-339")tool: process_refund(order_id="A-339",− amount=29.99, # full charge− reason="customer_request")reply: "Refund issued for $29.99."✗ structural score 0.62 missed duplicate-detect✗ semantic score 0.71 tone: terse✗ tool-call score 0.50 skipped find_duplicate()
+candidate · gemini-3.1-flash-lite-preview
case 17 of 40@@ case 17 — "refund a duplicate charge" @@tool: lookup_order(order_id="A-339")+ tool: find_duplicate(order_id="A-339") # NEWtool: process_refund(order_id="A-339",+ amount=29.99, # only the dup+ reason="duplicate_charge")reply: "Looks like you were charged twice — refund+ of $29.99 is on the way. Apologies for that."✓ structural score 0.91 +0.29✓ semantic score 0.86 +0.15✓ tool-call score 1.00 +0.50
✓ candidate is significantly better—Δ +0.064 · d 0.41 [0.18, 0.64] · p 0.003 · q 0.011 (BH-FDR)
40 / 40 cases scored
// 01
the pipeline
eight stages · tool-call trace style
└─init()scaffold project, prompts, and a 40-row golden suite0%
└─doctor()validate config and provider keys before spending a token
└─run()dispatch both models in parallel · sqlite cache · checkpoints
└─evaluate()structural · semantic · llm-judge · tool-call scoring
└─analyze()paired-t · cohen's d w/ 95% CI · BH-FDR correction
└─report()single-file, offline-capable HTML report
└─push()upload an immutable bundle · history, trends, share links
└─gate()the PR check fails when the candidate regresses
every stage re-runs on its own · $ evalshift run --resumereport() is the last stage that stays on your machine
// 02
three bad options
what every model migration looks like today
−
upgrade blindly
flip the model string and pray your canary catches it before customers do.
cost: silent failures
−
eyeball a notebook
spot-check a few outputs in jupyter, declare it "fine," ship on a Friday.
cost: confirmation bias
−
roll your own
spend three weeks building an eval harness nobody on the team trusts.
cost: weeks of eng time
+
run evalshift
a paired, statistically-honest answer in minutes — on your laptop, with your prompts and your golden suite.
cost: cents
// 03
real run
40 cases · 12.4s · $0.41
~/projects/support-agent$● live
$ evalshift run --suite golden/customer-support.jsonl◇ doctor ............................................. ok◇ estimated cost ..................................... $0.42▸ run ▰▰▰▰▰▰▰▰▰▰ 80/80 · cache 32/40▸ evaluate structural · semantic · judge · tool-call▸ analyze paired-t · cohen's d · BH-FDR✓ candidate is significantly betterΔ +0.064 · d 0.41 [0.18, 0.64] · p 0.003 · q 0.011⚠ 1 sub-metric regressed (tool-call · args; n.s. q=0.41)◇ report → ./.evalshift/reports/2026-05-09-1421.html◇ push → evalshift.dev/app/acme/support-agent/runs/r_8f2a✓ gate PR #481 check passed
// 04
the gate
the run that never reaches main
pull request #482 · swap the reasoning model● checks failed
✓buildSuccessful in 1m 12s
✗EvalShift / diffRegression on 2 evaluators
evalshift[bot] commented on #482✗ candidate regressed — merge blockedclaude-sonnet-4-5 → claude-sonnet-5semantic Δ −0.031 d −0.28 [−0.49, −0.07] q 0.004tool_selection Δ −0.112 d −0.71 [−0.95, −0.47] q 0.001exact_match Δ 0.000 no change3 / 40 cases regressed · 0 improvedview the full diff ↗
+ two lines of workflow
The Action runs your golden suite on every pull request, pushes the run, keeps one comment updated, and fails the check when the candidate regresses. Nobody has to remember to look.
- uses: babaliauskas/evalshift-action@v1
with:
token: ${{ secrets.EVALSHIFT_TOKEN }}
fail-on: regression· never · regression · any-slice-regression
· baseline resolves from the base branch
· every run is an immutable bundle, kept and trended
The same verdict the CLI printed on your laptop, enforced where it counts. Nothing is uploaded until you run evalshift push.
// 05
eight commitments
the things that matter when your prod depends on it
⌂
+ local by default
the run, the scoring and the report never leave your machine. nothing is uploaded until you run evalshift push — which sends the example inputs and both models' outputs.
docs →σ
◉
+ captured, not hand-written
the SDK records real model and tool calls in your app · promote them into a golden suite instead of inventing fixtures.
docs →⌘
⎇
+ gates your pull requests
the Action runs the suite on every PR, keeps one comment updated, and fails the check on a regression.
docs →∞
$
+ cost-aware
estimates spend before the run · prompts above $10 unless --yes.
↻
+ resumable
checkpoints every 50 completions · sqlite cache · crashes are cheap.
// 06
start
local in one command · hosted in two
+ run it on your laptop
The CLI, the capture SDK and the HTML report are open source and free. Scaffold a project, run both models against your golden suite, and read the diff — without an account, and without sending us anything.
$ uv pip install evalshift$ evalshift init --ci
Python 3.14+mypy --strictruff-formattedapache 2.0uv / pip install
+ keep it after the run
Push a run and it stops being a file on one machine: history your team can open, trends across releases, share links for the people who do not run the CLI, and the PR gate. Free while it is just you.
- +1 seat
- +100 hosted runs / month
- +7 days of run history
- +1 run at a time
no card · what the paid plans add →