Do I still need human review if I use EvalShift?

Yes, in two places: deciding which examples belong in the golden suite and what each evaluator should check, and then reading the cases a run flags. What changes is the volume — reviewers look at flagged cases and moved slices rather than at every output.

Is manual testing enough for an LLM model migration?

It is enough to find problems and not enough to show their absence. Reading a sample cannot separate a real regression from sampling variance, which is what paired tests and confidence intervals are for, and a manual pass leaves no artifact CI can re-run on the next change.

How many examples do I need before automated evals beat reading by hand?

Count per comparison, not per suite: deltas are grouped by prompt, evaluator and slice, so a 200-example suite across 3 prompts, 4 evaluators and 3 slices is up to 36 separate tests. Below 5 paired observations a comparison is skipped as insufficient and between 5 and 20 it is flagged uncertain, so target about 20 in each comparison you would act on.

Can I tell whether an agent regressed just by reading its answers?

Often not. Agent regressions live in the trace — a dropped tool call, a drifted argument, a refusal where work used to get done — and all of those produce fluent, well-formed final text. EvalShift scores tool selection, arguments and trace structure separately from the prose.

Does using EvalShift mean sending my prompts and outputs to a vendor?

The run, the scoring and the HTML report all happen on your machine under .evalshift/, and captures and the report are never uploaded. Nothing leaves until you run evalshift push, which sends the finalized run — example inputs, both models' outputs, scores and the analysis — to EvalShift Cloud.

compareevalshift-vs-manual-testing

EvalShift vs manual testing

Reading outputs by hand is how most model swaps actually get checked, and it is still the only method that can tell you a passing score is measuring the wrong thing. It is also the one method that does not scale with the suite, does not repeat, and leaves nothing for the next migration to start from.

claims checked 2026-08-19

// 01

where each one sits

the category ladder, broad to narrow

reading model output
general LLM evaluation
LLM migration testing — one suite, two models, paired per example
migration testing for agents — which tools fired, with what arguments, in what order

// 02

what each is built for

two jobs, not one job done twice

+ what EvalShift is built to do

Run one frozen golden suite against both models, paired per example, so differences in case difficulty cancel instead of swamping the signal.
Score every pair with structural, semantic, LLM-judge and tool-call evaluators, and keep the config that produced each score.
Test the deltas rather than eyeball them: a Shapiro-Wilk screen picks a paired t-test or Wilcoxon, with Cohen's d, a 95% CI, and Benjamini-Hochberg correction across every comparison in the run.
Score what the agent did — tool selection, argument values, call count, parallelism, refusals — not only the text it ended on.
Reduce the run to one verdict against budgets written down in advance, and let the GitHub Action enforce it on pull requests.
Keep the result: run history, pinned baselines, automatic diffs and trends, so the next migration starts from the last one.

· what hand review is genuinely best at

A person catches the failure nobody wrote a check for — the one that was never on the list because nobody had imagined it yet.
Tone, register and taste have no assertion to write against them; judging those is human work at every suite size.
A reviewer is the only thing that can tell you an evaluator is passing while measuring the wrong property.
It needs no config, no suite and no provider budget — you can start in the ten minutes after a model announcement.
On a handful of cases it is both faster and more informative than anything you could set up first.

// 03

side by side

every row checked against the sources below

dimension

evalshift

manual testing

Coverage

evalshiftEvery prompt × example × evaluator scored on both models, every run.

manual testingHowever many outputs one person can read attentively in the time available.

Repeatability

evalshiftSame suite, same evaluators, same recorded config — a rerun is comparable by construction.

manual testingTwo reviewers score the same output differently, and one reviewer disagrees with themselves a week later.

Regression vs. noise

evalshiftPaired t-test or Wilcoxon on the deltas, Cohen's d with a 95% CI, BH-corrected across comparisons.

manual testingA read of a sample, with no way to size the difference against its own variance.

Agent behavior

evalshifttool_selection, tool_arguments and tool_trace_structure score the trace; the report diffs traces side by side.

manual testingReadable call by call, but a dropped tool or a drifted argument leaves final text that still reads fine.

What “good” means

evalshiftWritten down as evaluator config and policy budgets, versioned alongside the code.

manual testingHeld in the reviewers' heads. Nothing records the standard that was applied.

Enforcement

evalshiftThe Action runs the suite on every PR, keeps one comment current, and fails the check when the migration policy says no.

manual testingA review someone has to ask for, from a person who has to be free.

What survives the migration

evalshiftRuns, baselines, diffs and trends persist per project; the next change is diffed against the last known-good run.

manual testingWhatever ended up in the pull request thread.

Cost of doing it again

evalshiftOne command. The response cache absorbs pairs whose model, prompt and inputs did not change.

manual testingThe same reading time again, in full, after every prompt tweak.

Unanticipated failures

evalshiftOnly measures what an evaluator was configured to check — the suite is a floor, not a ceiling.

manual testingA reviewer notices the thing no check exists for. This is the reason to keep people in the loop.

Setup before a first answer

evalshiftA config, a golden suite and provider keys.

manual testingNone. Open the outputs and read them.

// 04

which one to reach for

honest routing — both answers are real

reach for evalshift when

You are swapping a model under production traffic and need evidence a skeptical reviewer can re-derive, not a summary of one afternoon's reading.
The behavior you care about is in the trace — which tools fired, with what arguments, in what order — where a fluent final answer hides the regression.
The check has to run unattended on every pull request and block the merge when a budget is busted.
You will do this again. Baselines, diffs and trends mean the next migration starts from the last one instead of from zero.

reach for manual testing when

The prototype has no stable spec yet. Writing the bar down now would freeze a guess, and you learn more from reading the outputs than from scoring them.
The suite is five examples. Below five paired observations EvalShift reports insufficient and declines to test — reading five outputs is the faster and more honest answer.
The quality bar is subjective and nobody has articulated it. Judge twenty pairs by hand first; the criterion you could then hand to a colleague is the one worth automating.
You need an answer this afternoon, on a model that was announced this morning, with no config and no budget approved.

Automated scoring does not replace human judgement; it routes it. Judgement is what decides which recorded interactions belong in the golden suite, which properties deserve an evaluator, and how a judge criterion is worded — the bar being that a colleague could apply that criterion consistently across ten pairs by hand. After the run, the same judgement goes where the analysis points: the failing cases, the per-case trace diffs in the report, the one slice that moved while the overall number held. A new criterion runs non-blocking until someone has read the pairs it got wrong and agreed with it. Reviewers end up reading fewer outputs, and the ones they read are the ones actually in question.

// 05

questions

the ones people ask

Do I still need human review if I use EvalShift?: Yes, in two places: deciding which examples belong in the golden suite and what each evaluator should check, and then reading the cases a run flags. What changes is the volume — reviewers look at flagged cases and moved slices rather than at every output.
Is manual testing enough for an LLM model migration?: It is enough to find problems and not enough to show their absence. Reading a sample cannot separate a real regression from sampling variance, which is what paired tests and confidence intervals are for, and a manual pass leaves no artifact CI can re-run on the next change.
How many examples do I need before automated evals beat reading by hand?: Count per comparison, not per suite: deltas are grouped by prompt, evaluator and slice, so a 200-example suite across 3 prompts, 4 evaluators and 3 slices is up to 36 separate tests. Below 5 paired observations a comparison is skipped as insufficient and between 5 and 20 it is flagged uncertain, so target about 20 in each comparison you would act on.
Can I tell whether an agent regressed just by reading its answers?: Often not. Agent regressions live in the trace — a dropped tool call, a drifted argument, a refusal where work used to get done — and all of those produce fluent, well-formed final text. EvalShift scores tool selection, arguments and trace structure separately from the prose.
Does using EvalShift mean sending my prompts and outputs to a vendor?: The run, the scoring and the HTML report all happen on your machine under .evalshift/, and captures and the report are never uploaded. Nothing leaves until you run evalshift push, which sends the finalized run — example inputs, both models' outputs, scores and the analysis — to EvalShift Cloud.

sources

run your own migration diff

the CLI is open source and runs offline — no account, no upload, nothing leaves your machine until you push a run.

get started →how scoring works