EvalShift
docspricinglogin
−+ LLM migration testing for AI agents

LLM migration testing for AI agents. Don't ship a model upgrade on vibes.

Read the diff. Ship with proof.

Know what breaks before you switch models. EvalShift runs your existing and candidate models against the same golden suite, compares outputs, tool calls, structured behavior, cost, and latency, then tells you whether the change is a real regression.

Push the result to EvalShift Cloud for shared history, baseline comparison, and CI gating.

−baseline · gemini-2.5-flash
@@ case 17 — "refund a duplicate charge" @@
tool: lookup_order(order_id="A-339")
tool: process_refund(order_id="A-339",
−       amount=29.99,                          # full charge
−       reason="customer_request")
reply: "Refund issued for $29.99."
 
✗ structural    score 0.62   missed duplicate-detect
✗ semantic      score 0.71   tone: terse
✗ tool-call     score 0.50   skipped find_duplicate()
+candidate · gemini-3.1-flash-lite-preview
@@ case 17 — "refund a duplicate charge" @@
tool: lookup_order(order_id="A-339")
+ tool: find_duplicate(order_id="A-339")    # NEW
tool: process_refund(order_id="A-339",
+       amount=29.99,                          # only the dup
+       reason="duplicate_charge")
reply: "Looks like you were charged twice — refund
+        of $29.99 is on the way. Apologies for that."
✓ structural    score 0.91   +0.29
✓ semantic      score 0.86   +0.15
✓ tool-call     score 1.00   +0.50
✓ candidate is significantly betterΔ +0.064 · d 0.41 [0.18, 0.64] · p 0.003 · q 0.011 (BH-FDR)
40 / 40 cases scored
$ uv pip install evalshift   &&   evalshift init --ci
START FREE →SEE A FULL DIFF ↗
+ capture real agent behavior
the SDK records model and tool calls from your running app — no hand-written fixtures.
+ compare old vs new
semantic, structural and tool-call scoring, with paired statistics on every slice.
+ block regressions
the PR check fails when the candidate is worse. nobody has to remember to look.
// 01
the pipeline
eight stages · tool-call trace style
└─init()0%
└─doctor()
└─run()
└─evaluate()
└─analyze()
└─report()
└─push()
└─gate()
every stage re-runs on its own · $ evalshift run --resume
// 02
capture
what your agent already does, replayed as tests

+ Test what your agent does, not just what it says.

The SDK records model calls, tool calls, arguments, results, errors and the execution relationships between them, straight out of your own process. Representative agent behavior becomes the migration regression suite — no hand-written fixtures, no guessing at what production looks like.

behavior → regression suite
└─your running agentdev, staging or production — whatever it already does all day
└─evalshift sdkin-process, stdlib-only, no-op unless EVALSHIFT_CAPTURE=1
└─captured tracesone JSON file per invocation in .evalshift/captures/
└─golden suitepromote the cases worth defending · the rest stay on disk
└─baseline vs candidatethe same captured cases replayed against both models
└─regression analysisper-case verdicts, paired statistics, tool-call diffs
support_agent/app.py● capture on
$ uv add evalshift-sdk
 
from evalshift import capture
 
@capture.agent(suite="support_agent",
               redact=True, tools=[])
def handle_ticket(query):
    ...
 
$ EVALSHIFT_CAPTURE=1 python -m app
  wrote 37 captures → .evalshift/captures/
capture is off until you ask for it · EVALSHIFT_CAPTURE=1instrument an agent →
// 03
three bad options
what every model migration looks like today
−
upgrade blindly
flip the model string and pray your canary catches it before customers do.
cost: silent failures
−
eyeball a notebook
spot-check a few outputs in jupyter, declare it "fine," ship on a Friday.
cost: confirmation bias
−
roll your own
spend three weeks building an eval harness nobody on the team trusts.
cost: weeks of eng time
+
run evalshift
a paired, statistically-honest answer in minutes — on your laptop, with your prompts and your golden suite.
cost: cents
// 04
real run
40 cases · 12.4s · $0.41
~/projects/support-agent$● live
$ evalshift run --suite golden/customer-support.jsonl
 
◇ doctor ............................................. ok
◇ estimated cost ..................................... $0.42
▸ run         ▰▰▰▰▰▰▰▰▰▰  80/80   · cache 32/40
▸ evaluate    structural · semantic · judge · tool-call
▸ analyze     paired-t · cohen's d · BH-FDR
 
✓ candidate is significantly better
   Δ +0.064 · d 0.41 [0.18, 0.64] · p 0.003 · q 0.011
⚠ 1 sub-metric regressed (tool-call · args; n.s. q=0.41)
 
◇ report → ./.evalshift/reports/2026-05-09-1421.html
◇ push   → evalshift.dev/app/acme/support-agent/runs/r_8f2a
✓ gate     PR #481 check passed
// 05
the gate
the run that never reaches main
pull request #482 · swap the reasoning model
✓buildSuccessful in 1m 12s
✗EvalShift / diffRegression on 2 evaluators
evalshift[bot] commented on #482
 
✗ candidate regressed — merge blocked
   claude-sonnet-4-5 → claude-sonnet-5
 
   semantic       Δ −0.031  d −0.28 [−0.49, −0.07]  q 0.004
   tool_selection Δ −0.112  d −0.71 [−0.95, −0.47]  q 0.001
   exact_match    Δ  0.000  no change
 
   3 / 40 cases regressed · 0 improved
   view the full diff ↗

+ two lines of workflow

The Action runs your golden suite on every pull request, pushes the run, keeps one comment updated, and fails the check when the candidate regresses. Nobody has to remember to look.

- uses: babaliauskas/evalshift-action@v1
  with:
    token: ${{ secrets.EVALSHIFT_TOKEN }}
    fail-on: regression
· never · regression · any-slice-regression
· baseline resolves from the base branch
· every run is an immutable bundle, kept and trended
how gating decides →

The same verdict the CLI printed on your laptop, enforced where it counts. Nothing is uploaded until you run evalshift push.

// 06
eight commitments
the things that matter when your prod depends on it
⌂
+ local by default
the run, the scoring and the report never leave your machine. nothing is uploaded until you run evalshift push — which sends the example inputs and both models' outputs.
docs →
σ
+ statistically honest
paired-t / Wilcoxon, Cohen's d w/ 95% CIs, Benjamini–Hochberg FDR.
docs →
◉
+ captured, not hand-written
the SDK records real model and tool calls in your app · promote them into a golden suite instead of inventing fixtures.
docs →
⌘
+ agent-aware
scores tool selection, argument correctness, parallel calls, ordering.
docs →
⎇
+ gates your pull requests
the Action runs the suite on every PR, keeps one comment updated, and fails the check on a regression.
docs →
∞
+ provider-agnostic
one config across Anthropic, OpenAI, Google. tools defined once.
docs →
$
+ cost-aware
estimates spend before the run · prompts above $10 unless --yes.
↻
+ resumable
checkpoints every 50 completions · sqlite cache · crashes are cheap.
// 07
start
local in one command · Cloud in two

+ run it on your laptop

The CLI, the capture SDK and the HTML report are open source and free. Scaffold a project, run both models against your golden suite, and read the diff — without an account, and without sending us anything.

$ uv pip install evalshift
$ evalshift init --ci
Python 3.14+mypy --strictruff-formattedagpl-3.0uv / pip install

+ keep it after the run

Push a run and it stops being a file on one machine: history your team can open, trends across releases, share links for the people who do not run the CLI, and the PR gate. Free while it is just you.

  • +1 seat
  • +100 Cloud runs / month
  • +7 days of run history
  • +1 run at a time
START FREE →

no card · what the paid plans add →