Can I use EvalShift and Promptfoo together?

Yes, and it is a sensible split. Promptfoo handles red teaming and broad output-quality evals; EvalShift handles the paired baseline-vs-candidate run and the migration gate. They share nothing but your provider keys and run as separate CI checks.

Does Promptfoo do statistical significance testing?

Its documentation describes a test case's score as a weighted average of its assertion scores, compared against optional thresholds — we found no significance testing described there as of August 2026. EvalShift runs one paired test per (prompt, evaluator, slice) and corrects across the whole run with Benjamini-Hochberg.

Can Promptfoo evaluate agent tool calls?

Yes — its trajectory assertions check which tools an agent used, whether the arguments matched, the order they ran in, and how many steps it took. EvalShift's difference is that it scores both models' traces on the same example and reports the delta, because it is asking whether a migration changed the behaviour rather than whether one run was correct.

Is EvalShift open source like Promptfoo?

Partly. The capture SDK and the GitHub Action are MIT; the CLI is AGPL-3.0-or-later and still alpha. EvalShift Cloud, which stores run history, baselines and trends, is a paid product. Promptfoo is MIT with a free community tier.

Does EvalShift send my prompts to a server?

Not unless you ask it to. The run, the scoring, the analysis and the HTML report all happen on your machine under .evalshift/. Only evalshift push uploads, and it sends the finalized run — example inputs, both models' outputs, scores and analysis — never your provider keys and never the report.

compareevalshift-vs-promptfoo

EvalShift vs Promptfoo

Promptfoo is a general evaluation and red-teaming framework for LLM apps, and a good one. EvalShift does one narrower thing: it runs your agent on two models over the same suite and tells you, with a significance test, whether the swap is safe to ship.

claims checked 2026-08-19Promptfoo ↗

// 01

where each one sits

the category ladder, broad to narrow

llm evaluation — score prompts and outputs against assertions. a crowded category, well served.
llm observability — record what production did and watch it move.
llm migration testing — is the candidate model safe to swap in for the one already shipping?
llm migration testing for agents — did the tool calls survive the swap? this is the rung EvalShift is built on.

// 02

what each is built for

two jobs, not one job done twice

+ what EvalShift is built around

one question, asked well: is it safe to move this agent from the model in production to a candidate?
paired runs — both models see byte-identical input, and the unit of analysis is the per-example delta, not two independent pass rates.
the suite comes from the agent itself: the SDK records real model and tool calls in your app, and capture sync promotes them into golden JSONL.
agent behaviour is scored directly — tool selection, argument correctness, call ordering and parallelism, on both sides of every pair.
every comparison gets a significance test: Shapiro-Wilk screen, paired t or Wilcoxon, Cohen's d with a 95% CI, Benjamini-Hochberg across the whole run.
a migration policy turns that into one verdict, and the GitHub Action fails the check when the verdict is fail.

· what Promptfoo is for

one config runs your prompts across OpenAI, Anthropic, Azure, Bedrock, Ollama and more, and renders a side-by-side matrix — the docs frame it as building your own benchmark instead of trusting public ones.
a large assertion catalog: contains / regex / json checks, BLEU, ROUGE, METEOR, levenshtein, llm-rubric, g-eval, factuality, and RAG context metrics, so most output-quality questions have a check already written.
red teaming and vulnerability scanning — it generates adversarial inputs, runs them, and reports the risk. EvalShift has no equivalent and does not try to.
trajectory assertions cover agents: which tools were used, whether the arguments matched, in what order, and how many steps it took.
MIT licensed with a free community tier, and the README states LLM evals run 100% locally — your prompts never leave your machine.
developer ergonomics: live reload, caching, concurrency, and test logic in JavaScript, Python or Ruby.

// 03

side by side

every row checked against the sources below

dimension

evalshift

Promptfoo

Shape of the comparison

evalshiftpaired: source and target answer the same example, and the delta is the measurement.

Promptfooside-by-side: providers are scored independently and rendered as a comparison matrix.

Statistical significance

evalshiftShapiro-Wilk screen, paired t or Wilcoxon, Cohen's d with 95% CI, Benjamini-Hochberg FDR at 0.05.

Promptfootest scores are a weighted average of assertion scores against optional thresholds; the docs describe no significance testing.

Where test cases come from

evalshiftrecorded from the running agent by the SDK, then promoted — history, tool calls and all.

Promptfooauthored in config or loaded from CSV / JSON / JS / Python; promptfoo generate can synthesize datasets.

Ship / no-ship gate

evalshiftnamed budgets (regression rate, critical count, arg drift, cost, latency) → pass / conditional_pass / fail / inconclusive.

Promptfooassertions pass or fail against thresholds, and CI fails on that.

Run history and baselines

evalshiftpin one baseline per suite; every later run is diffed against it automatically, with verdict and regression trends.

Promptfooshare creates a link to an eval; Enterprise adds centralized dashboards and continuous monitoring.

Agent tool calls

evalshiftfour evaluators score both models' traces and compare them — selection, arguments, trace structure, imported traces.

Promptfootrajectory assertions check tool use, argument match, sequence and step count on a run.

Where the data lives

evalshiftruns, scoring and the HTML report stay on your machine; push is opt-in and sends the finalized run.

Promptfooevals run locally against your own provider keys; sharing and cloud are opt-in.

Breadth of built-in metrics

evalshifta focused set: structural, semantic, LLM judge, and the tool evaluators.

Promptfoomuch wider — dozens of deterministic and model-graded assertions, including RAG and text-overlap metrics.

Security testing

evalshiftnone. EvalShift does not do red teaming.

Promptfoored teaming, adversarial generation and vulnerability scanning are a first-class part of the product.

Install and ecosystem

evalshiftPython CLI (3.14+, AGPL, alpha) plus a stdlib-only capture SDK; models go through LiteLLM.

Promptfoonpm, npx or Homebrew on Node 22.22+; MIT, mature, and works with any LLM API or language.

// 04

which one to reach for

honest routing — both answers are real

reach for evalshift when

you have an agent in production and a model deprecation date, and 'it looked fine in staging' is not an answer you can defend.
'did anything regress?' has to come back with an effect size and a confidence interval, not a gap between two pass rates.
the failure you actually fear is a silent tool drop — the model stops calling notify_security_team and the prose still reads fine.
you want the merge blocked automatically when a candidate busts a budget you wrote down in advance.

reach for Promptfoo when

you need red teaming or vulnerability scanning — that is the whole point of Promptfoo's red-team side, and EvalShift has nothing to offer here.
you are choosing a model rather than migrating one: lining five candidates up on your own data is exactly what the matrix view is for.
the question is output quality — RAG faithfulness, rubric grading, text overlap — and you would rather pick an assertion off the shelf than write an evaluator.
your team is not on Python, or you want test logic in JavaScript or Ruby next to the rest of your suite.

They do not overlap much, and running both is reasonable. Promptfoo covers the broad surface — red teaming before launch, rubric and RAG checks on output quality, benchmarking candidate models on your own data. EvalShift covers the narrow one: once you have picked the candidate, it runs the paired suite, scores the tool calls, applies the statistics and gates the pull request. Both execute locally against your own provider keys, and neither needs to know about the other; in CI they are simply two checks, answering two different questions.

// 05

questions

the ones people ask

Can I use EvalShift and Promptfoo together?: Yes, and it is a sensible split. Promptfoo handles red teaming and broad output-quality evals; EvalShift handles the paired baseline-vs-candidate run and the migration gate. They share nothing but your provider keys and run as separate CI checks.
Does Promptfoo do statistical significance testing?: Its documentation describes a test case's score as a weighted average of its assertion scores, compared against optional thresholds — we found no significance testing described there as of August 2026. EvalShift runs one paired test per (prompt, evaluator, slice) and corrects across the whole run with Benjamini-Hochberg.
Can Promptfoo evaluate agent tool calls?: Yes — its trajectory assertions check which tools an agent used, whether the arguments matched, the order they ran in, and how many steps it took. EvalShift's difference is that it scores both models' traces on the same example and reports the delta, because it is asking whether a migration changed the behaviour rather than whether one run was correct.
Is EvalShift open source like Promptfoo?: Partly. The capture SDK and the GitHub Action are MIT; the CLI is AGPL-3.0-or-later and still alpha. EvalShift Cloud, which stores run history, baselines and trends, is a paid product. Promptfoo is MIT with a free community tier.
Does EvalShift send my prompts to a server?: Not unless you ask it to. The run, the scoring, the analysis and the HTML report all happen on your machine under .evalshift/. Only evalshift push uploads, and it sends the finalized run — example inputs, both models' outputs, scores and analysis — never your provider keys and never the report.

sources

run your own migration diff

the CLI is open source and runs offline — no account, no upload, nothing leaves your machine until you push a run.

get started →how scoring works