LLM migration testing for AI agents. Don't ship a model upgrade on vibes.
Read the diff. Ship with proof.
Compare your current and candidate models on real production behavior. Catch regressions before they reach users. Upgrade, downgrade, or switch LLM providers without guessing whether your agent will break. EvalShift runs both models against the same golden suite of captured agent traces, compares outputs, tool calls, structured behavior, cost, and latency, then tells you whether the change is a real regression.
Push the result to EvalShift Cloud for shared history, baseline comparison, and CI gating.
The full report is a copy of the report the CLI writes to your machine — reports stay local and are never uploaded, even when you push.
Your eval data stays on your machine unless you explicitly push a run to EvalShift Cloud.
Works with Anthropic, OpenAI and Google models.
How it works
Your agent's real behavior becomes the test suite. Two models run it. You read one verdict.
Capture
One decorator. The SDK records what your agent really does — prompts, tool calls, arguments — while it runs.
from evalshift import capture @capture.agent(suite="main_chat", tools=TOOLS) def handle(message): ...
Compare
Replay those cases on the model you have and the model you want. Same inputs, both models, scored side by side.
Decide
A verdict — pass, conditional, or fail — with the reasons, the numbers and the cost difference.
It tells you what broke, in plain English.
The same run as above. Every figure is measured; the prose is generated from the figures, never the other way round.
Score change per example
16 examples · target minus sourceScores run 0–1, so −1.00 means the target scored a full point lower than the source on that example's worst-hit evaluator.
Then read the diff, one example at a time.
Every finding points at a real conversation: what the user asked, what each model did, and why it was flagged. This is the example behind “Searches instead of answering”.
“Cool. What's your take on whether working late is worth it?”
The source called no tools; the target called search_web.
final text: It's occasionally necessary during crunch times or when you're in a great flow state, but making it a habit usually costs more in focus and recovery than it delivers in output. Sustainable momentum during your core hours almost always wins out long term.
1. search_web ({"query": "is working late worth it productivity well-being"})
final text: (none — the model returned no text)
extra tool: Target added search_web at position 1.
Runs on your laptop. Blocks the merge in CI.
The Action runs your golden suite on every pull request, keeps one comment updated, and fails the check when the candidate regresses. Nobody has to remember to look.
- buildSuccessful
- EvalShift / migration2 budgets breached
- uses: babaliauskas/evalshift-action@v0
with:
token: ${{ secrets.EVALSHIFT_TOKEN }}
fail-on: regressionWhen it's safe, it says so.
A pass is not silence. The report still lists what moved, so a non-blocking drop in similarity is a note to read before promoting, not a surprise later.
Safe to migrate under the configured policy. 0 blocking regressions, 0 critical regressions.
Semantic similarity fell on every prompt (d = −2.51, p < 0.0001). Non-blocking, but worth a look before promoting.
Start in one command.
The CLI, the capture SDK and the HTML report are open source and free. Run locally without an account; push when you want history, trends, share links and the PR gate.
$ uv pip install evalshift $ evalshift init --ci
Python 3.11+ · Apache-2.0 · reports stay on your machine
- 1 seat
- 100 Cloud runs / month
- 7 days of run history
- 1 run at a time
no card · what the paid plans add →
Read before you migrate.
Field notes from real model swaps: what breaks, how many cases you need, and when to trust the judge.
An EvalShift migration report that said PASS: a 71% cheaper model inside every policy budget. Where each threshold came from, and what PASS does not claim.
A candidate model cut cost 59% and latency 75% with zero failed calls. The migration report still said FAIL, because the agent's behavior changed.
A model swap is a behavior change with no diff to review. What moves besides the answer text, and why the worst regressions read as correct.