compareevalshift-vs-deepeval
EvalShift vs DeepEval
DeepEval is a metric library for testing whether an LLM app is good enough. EvalShift is a paired run for deciding whether a new model behaves like the one already in production.
claims checked 2026-08-19DeepEval ↗
// 01
where each one sits
the category ladder, broad to narrow
- general LLM evaluation — dozens of frameworks
- LLM observability — dozens more
- LLM migration testing — a narrow category
- LLM migration testing for agents — what EvalShift is built for
// 02
what each is built for
two jobs, not one job done twice
+ What EvalShift is built around
- The capture SDK records real model and tool calls inside your agent; capture sync promotes them into a golden suite.
- Every example runs on the current model and the candidate in one paired run, so the per-example delta is what gets tested.
- Tool selection, argument values, call structure and refusals are scored on both sides of that pair.
- Deltas go through a paired t-test or Wilcoxon, Cohen's d with a 95% CI, and Benjamini-Hochberg FDR correction.
- A migration policy turns the numbers into one verdict: pass, conditional_pass, fail or inconclusive.
- The GitHub Action pushes the run, keeps one PR comment updated, and fails the check on that verdict.
· What DeepEval is for
- 50+ ready-to-use metrics spanning RAG, agents, conversation, safety and multimodal output.
- G-Eval and DAG define a custom judge criterion without writing a scorer by hand.
- Metrics run as assertions under pytest via deepeval test run, and under Vitest in TypeScript.
- End-to-end, component-level and trajectory evals score spans, tools and planners, not only the final answer.
- A Synthesizer generates goldens from docs, contexts or scratch when you have no collected examples yet.
- Safety metrics cover bias, toxicity and PII leakage, and the docs carry a red-teaming guide over 40+ vulnerabilities.
// 03
side by side
every row checked against the sources below
dimension
evalshift
DeepEval
Baseline vs candidate
evalshiftEvery example runs on both models inside one run; the delta between them is the unit of analysis.
DeepEvalA test run scores one system; Confident AI compares two runs side by side and marks what improved or regressed.
Statistical significance
evalshiftShapiro-Wilk screen, paired t-test or Wilcoxon, Cohen's d with a 95% CI, Benjamini-Hochberg FDR at 0.05.
DeepEvalEach metric returns a 0-1 score with a reason, and passes or fails against its threshold.
Ship / no-ship budgets
evalshiftMigration policy sets budgets for regression rate, equivalence, tool-argument drift, cost and latency.
DeepEvalThresholds are set per metric; a run passes when its test cases clear them.
Where the suite comes from
evalshiftThe SDK records production model and tool calls; capture sync promotes them into golden.jsonl.
DeepEvalGoldens you author or pull from Confident AI, plus a Synthesizer that generates them from docs or from scratch.
Tool-call scoring
evalshiftTool selection, argument values, call count, parallelism and refusals, scored on both sides.
DeepEvalTool Correctness and Argument Correctness at component level, plus trajectory metrics like Plan Adherence.
CI gating
evalshiftThe Action pushes the run, upserts one PR comment, and fails the check on the Cloud policy verdict.
DeepEvaldeepeval test run gates from pytest; Confident AI offers required eval checks that block a merge.
Where run data lives
evalshiftRuns, outputs and the HTML report stay under .evalshift/; a bundle uploads only on evalshift push.
DeepEvalEvaluations run in your own environment; results upload to Confident AI once you run deepeval login.
Breadth of prebuilt metrics
evalshiftStructural, semantic-similarity, pairwise judge and tool evaluators — the set a two-model comparison needs.
DeepEval50+ metrics across RAG, agents, conversation, safety and multimodal, plus G-Eval and DAG for custom criteria.
RAG-specific scoring
evalshiftNo retriever-level metrics; scoring stops at the agent's outputs and its tool calls.
DeepEvalContextual precision, recall and relevancy for the retriever; faithfulness and answer relevancy for the generator.
Languages
evalshiftPython — CLI on 3.14+, capture SDK on 3.10+.
DeepEvalPython and TypeScript, running under pytest and Vitest respectively.
// 04
which one to reach for
honest routing — both answers are real
reach for evalshift when
- You are swapping a model, a provider or a prompt underneath an agent that is already in production.
- The failure you are worried about is a dropped tool call or a changed argument, not worse prose.
- You need to say how confident you are that a difference is real, not only that a score moved.
- The decision has to be enforced on the pull request, against budgets you set for regressions, cost and latency.
reach for DeepEval when
- You are measuring one system's quality — faithfulness, answer relevancy, retrieval precision — rather than comparing two models.
- You want a large library of prebuilt metrics to pick from instead of configuring evaluators for a comparison.
- Your evals belong in the pytest or Vitest suite the team already runs on every commit.
- You need safety and red-teaming coverage, or a synthesizer to build a dataset you do not have yet.
They answer different questions and nothing stops you running both. DeepEval tells you whether the app you ship is good enough, measured against metric thresholds inside your existing test suite. EvalShift tells you whether a candidate model behaves like the one already serving traffic, on examples captured from that traffic, and holds the merge until the paired numbers say it does. A team migrating a RAG agent might keep DeepEval's faithfulness and relevancy metrics on the pipeline and add an EvalShift run for the model swap itself.
// 05
questions
the ones people ask
- Can I use EvalShift and DeepEval together?
- Yes. They are separate Python packages that read different inputs and answer different questions — DeepEval scores the app you ship against metric thresholds, EvalShift compares two models on the same captured examples.
- Does DeepEval do model migration testing?
- DeepEval evaluates a test run against per-metric thresholds, and Confident AI's testing reports compare two runs side by side and highlight cases that improved or regressed. EvalShift's unit is the paired run itself: both models answer the same example, and the delta is what gets tested for significance.
- Which one tests tool calls?
- Both. DeepEval has Tool Correctness and Argument Correctness metrics plus trajectory metrics such as Plan Adherence and Step Efficiency. EvalShift scores tool selection, argument values and call structure on both sides of a paired run and gives tool-argument drift its own policy budget.
- Do my prompts and outputs leave my machine?
- Not with EvalShift unless you run evalshift push — the run, the scoring and the HTML report stay under .evalshift/. DeepEval also runs evaluations in your own environment, with optional upload to Confident AI after deepeval login.
- Is EvalShift a replacement for a metric library?
- No. EvalShift ships the evaluators a migration comparison needs — structural, semantic, pairwise LLM judge and tool evaluators. If you want breadth of prebuilt quality metrics, DeepEval has 50+ of them.
sources
run your own migration diff
the CLI is open source and runs offline — no account, no upload, nothing leaves your machine until you push a run.