Promptfoo alternatives in 2026: evals, red teaming and run history

The best Promptfoo alternative depends on which part of Promptfoo you want to replace. As of November 2026: for evals in Python, DeepEval if you want ready-made metrics in pytest, Inspect AI for agent evals in sandboxes, or Pydantic Evals for typed, deterministic checks; for evals in TypeScript, vitest-evals; for run history and a UI your whole team can open, Langfuse, Braintrust or LangSmith; for red teaming, DeepTeam, garak or PyRIT.

The reason most teams started looking in 2026 is that OpenAI announced it was acquiring Promptfoo on March 9. Promptfoo is still MIT licensed and still releasing, and OpenAI now sends users of its own retiring Evals product to it. So before the list, this post covers what changed after the acquisition and what did not, the other reasons teams move, and what Promptfoo still does better than anything here. It ends with how to translate a promptfooconfig.yaml into each kind of alternative.

A disclosure: I build EvalShift. It replaces one narrow use of Promptfoo, comparing two models before a swap, and it is on the list for that job only. Every claim about another tool comes from that tool's docs, repository or pricing page, read on November 13, 2026.

Why teams look for a Promptfoo alternative

Promptfoo is a good tool, and none of these reasons is that it stopped working.

1. +OpenAI owns it. Some teams do not want the tool that compares OpenAI, Anthropic and Google models, or that red-teams an agent built on one of them, to belong to one of the three. That is a policy position, not a defect: nothing in the code has been shown to favour OpenAI. What changed and what did not is in the next section.
2. +Node and YAML. Promptfoo needs Node 22.22 or newer. The promptfoo package on PyPI ↗ is a wrapper that still needs Node and runs the CLI through npx. Tests live in a YAML file, with JavaScript or Python snippets inside it for custom logic. Python teams often want evals as code next to the code they test.
3. +Results have no shared home. The viewer is local. The open-source server can be self-hosted, but the docs ↗ say that is not recommended for production: it stores results in SQLite, has no built-in authentication or roles, and cannot run more than one replica. Dashboards, SSO and continuous monitoring are on the Enterprise plan ↗, which is priced on request.
4. +Red teaming leans on Promptfoo's servers. Your target is always called locally, but attack generation is remote by default ↗. Without an OpenAI key, generation and grading go to Promptfoo's API, and the data-handling page ↗ lists what goes with them: your application's purpose, the prompts sent to your target, its responses and your email address. Plugins and strategies marked remote in the docs use hosted inference even with your own key, including 11 of the 34 strategies. You can turn remote generation off, and the docs warn that quality drops when you do.
5. +Telemetry is opt-out. Usage telemetry is on by default and PROMPTFOO_DISABLE_TELEMETRY=1 turns it off. An open issue ↗ reports one request still going out with it disabled; that is a user report I have not reproduced.
6. +It checks before release only. Promptfoo evaluates what you run it on. Scoring live traffic and alerting on it is an Enterprise feature.

One reason that no longer holds: "it has no statistics on repeated runs". Since version 0.125.0, from October 2026, --repeat reports a Wilson 95% confidence interval on each repeated test's pass rate. What it still does not do is test whether two models differ by more than chance.

What changed after OpenAI acquired Promptfoo

The facts, as of November 13, 2026:

Two things did change. The homepage now leads with agent security, with evals as one product among several. And the open-source project's future is now a decision made inside OpenAI, which has said the technology will be integrated into its Frontier enterprise platform.

A fair reading: the acquisition has not yet made Promptfoo worse at anything, and the MIT license means the code you depend on today cannot be taken back. What the license does not promise is which provider gets a new feature first, or how much attention the eval side gets next year. If that uncertainty is acceptable, you do not need an alternative. Pin the version and carry on.

What Promptfoo does well

Worth listing before replacing it, because an alternative has to cover what you actually use:

• +The matrix. One config runs every prompt against every model and every test case, and a local web viewer shows the outputs side by side. No alternative here does this as directly.
• +Deterministic assertions. About 40 assertion types run without a model: string, regex, JSON and SQL checks, text-overlap scores, cost, latency, and trajectory checks on an agent's tool calls. A suite can run with no judge at all.
• +Red teaming. 157 plugins ↗ and 34 strategies ↗, with 10,000 probes a month free. No open-source alternative states coverage that wide.
• +Any language. The application under test can be anything that answers over HTTP or a script.
• +CI. An exit code, a pass-rate threshold and an official GitHub Action ↗ that comments on the pull request.

Promptfoo alternatives at a glance

Promptfoo itself, for reference: MIT, Node 22.22+, 0.125.1 from October 25, 2026, owned by OpenAI.

Alternatives for evals

DeepEval

DeepEval ↗ is the alternative most Python teams try first. An eval is a pytest test: build a test case, attach metrics, and deepeval test run fails the test when a metric misses its threshold. It ships more than 50 metrics for RAG, agents, conversations and safety, and it now has a TypeScript package on npm.

What you give up: the matrix and the local viewer. And where Promptfoo's checks are mostly code, DeepEval's metrics page ↗ says almost all of its metrics are graded by an LLM judge, so a run costs at least one judge call per metric per case. Run history and comparison are in Confident AI ↗, the proprietary platform from the same team: free for 2 seats and 5 test runs a week, then $200 a month per organization. Telemetry is on by default here too, and DEEPEVAL_TELEMETRY_OPT_OUT=1 turns it off.

Choose it when you want evals in pytest and a metric library you do not have to write. Promptfoo vs DeepEval compares the two in full.

Inspect AI

Inspect AI ↗ comes from the UK AI Security Institute. A task is a dataset, a solver and a scorer in Python. It runs agents inside Docker or Kubernetes sandboxes, repeats each sample with epochs, and reports a standard error with every score. It is MIT, has no commercial edition, and its docs describe no telemetry.

What you give up: the prompt matrix, a red-team scanner, and a documented way to fail a pull request on a threshold. You write that last part yourself from the log files.

Choose it when you evaluate agents that run code or use a computer, you want error bars without building them, or you want a tool with no vendor behind it.

Pydantic Evals

Pydantic Evals ↗ is the evaluation library in the Pydantic AI repository. A Dataset holds typed Case objects, and evaluators check the output or the OpenTelemetry spans your code emitted, so "the agent called lookup_order before refund" is a deterministic check on a span tree. An LLM judge is available when you need one.

What you give up: a CLI and an exit-code gate (you run it from your own test or script), the matrix and red teaming. The hosted results view is Logfire, Pydantic's paid observability product, with a free personal tier.

Choose it when you already use Pydantic and want most checks to be code.

vitest-evals

vitest-evals ↗ is the option for TypeScript teams who want to leave YAML but stay in Node. Evals are Vitest tests with scorers, and its GitHub Action gates on a pass rate. It is a small project, at 0.18.1 with a few hundred stars, so read the source before you depend on it.

Choose it when your application and its tests are TypeScript and you want evals in the same runner.

One to skip, one to check

• +OpenAI Evals is not an alternative. The hosted product is read-only and shuts down on November 30, 2026, and OpenAI's migration guide points to Promptfoo. The open-source openai/evals package last released in May 2024.
• +Ragas has the RAG metrics many teams want, but its last release was January 13, 2026 and its last commit February 24. Pin the version if you use it, and note that DeepEval and Promptfoo both ship faithfulness, context recall and answer relevance checks of their own.

Alternatives when you need history and a UI

If the reason you are leaving is that results live on one laptop, a framework will not fix it. These are platforms: they store every run, show two side by side and let non-engineers review outputs. All of them also trace production, which Promptfoo's open-source edition does not.

Langfuse

Langfuse ↗ is MIT licensed apart from its enterprise folders, and self-hosts for free with all core features. Its cloud has a free tier of 50,000 units a month and a first paid plan at $29 a month with unlimited users.

It is also the only tool here that publishes a migration guide from Promptfoo ↗. The guide supplies example scripts, not an importer, and it says to keep Promptfoo for red teaming, which Langfuse does not do.

Choose it when you want eval runs next to production traces in something you can host yourself.

Braintrust and LangSmith

Braintrust ↗ is built around experiments: an Eval() with a dataset, a task and scorers, compared row by row against a base run, with a GitHub Action that posts the result. It is proprietary and cloud-hosted, free for 1 GB of data and 10,000 scores a month, then $249 a month.

LangSmith ↗ runs datasets as pytest, Vitest or Jest tests and has ready-made trajectory evaluators for agents. It is proprietary, free for one seat and 5,000 traces a month, then $39 per seat.

Choose one of them when you would rather not operate a platform. Braintrust vs LangSmith compares the two.

Opik (Apache-2.0) and Arize Phoenix (Elastic License 2.0) are the other self-hostable options. Braintrust alternatives covers both.

Alternatives for red teaming

This is the part of Promptfoo that is hardest to replace, and no alternative matches its plugin count. What the alternatives offer is attack generation that runs on models you choose.

DeepTeam

DeepTeam ↗ is the red-teaming package from the DeepEval team: Apache-2.0, version 1.0.10. Its README claims more than 50 vulnerability types and more than 20 attack methods, mapped to the OWASP Top 10 for LLMs, NIST AI RMF and MITRE ATLAS. It generates and grades attacks with your own model, from Python or a YAML file. It releases less often than DeepEval: four times since July 2026.

Choose it when you want the closest thing to Promptfoo's application-level red teaming with no hosted generation step.

garak

garak ↗, from NVIDIA, is a vulnerability scanner for models. You point the CLI at a model endpoint and it runs probes for prompt injection, jailbreaks, data leakage, toxicity and more, then reports which ones landed. It is Apache-2.0 with 10k stars.

It tests a model more than an application: it does not know your system's purpose or tools the way a Promptfoo red-team config does.

Choose it when you are assessing a model, including one you host yourself, before building on it.

PyRIT

PyRIT ↗ is Microsoft's red-teaming framework, MIT licensed. It began as a Python library for security researchers building multi-turn attacks, and now also has a scanner CLI and a browser interface. It asks more of you than the others: you assemble targets, converters and scorers yourself.

Note that it does not answer a neutrality concern any better than Promptfoo does. Microsoft sells models too.

Choose it when you have a security team that wants to script its own attacks.

Giskard

Giskard ↗ version 3, Apache-2.0, generates attacks from a plain-language description of your application and runs them as pytest scenarios. Version 2 is no longer maintained and version 3 is a rewrite, at 3.0.1 since October 2026, so treat it as a new project.

One more thing on the red-teaming search results: Promptfoo's own "Promptfoo vs PyRIT" page ranks for this query and dates from June 2025, before PyRIT had a CLI.

Alternative for comparing two models

EvalShift

Many teams use Promptfoo's matrix for one recurring job: the model under a production agent is being retired or replaced, and they need to know whether the candidate behaves the same. The matrix shows the two models side by side and scores each on its own.

EvalShift is built for that job only. It runs your recorded production cases through both models, scores tool selection, arguments and trace rules on each side, and tests the per-example differences with a paired t-test or Wilcoxon, an effect size with a confidence interval and a false-discovery-rate correction. A migration policy turns that into a verdict that passes or fails the pull request.

What you give up is most of Promptfoo: there is no red teaming, no five-model matrix and a much smaller set of checks. The CLI is Apache-2.0, has no telemetry and runs on your machine against your own provider keys; the hosted run history is a paid plan.

Choose it when the question is "is this model swap safe", and keep another tool for everything else. EvalShift vs Promptfoo goes dimension by dimension.

Moving off Promptfoo

Only Langfuse publishes a migration guide, and nobody ships an importer, so a move is a rewrite. It is a mechanical one, because a promptfooconfig.yaml has few concepts:

A Promptfoo test with one deterministic and one judged assertion:

tests:
  - vars:
      message: Where is my order 4417?
    assert:
      - type: icontains
        value: '4417'
      - type: llm-rubric
        value: Does not promise a delivery date

becomes this in pytest with DeepEval:

import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams

from app import reply  # your function that calls the model

no_promise = GEval(
    name="No delivery promise",
    criteria="The actual output does not promise a delivery date.",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT],
)


@pytest.mark.parametrize("message, must_contain", [("Where is my order 4417?", "4417")])
def test_reply(message, must_contain):
    output = reply(message)
    assert must_contain.lower() in output.lower()  # was: icontains
    assert_test(LLMTestCase(input=message, actual_output=output), [no_promise])  # was: llm-rubric

Four things to plan for:

1. +Deterministic checks port cleanly. They become ordinary assertions and will give the same results.
2. +Judged checks do not. A rubric graded by a different judge prompt, model or threshold will pass and fail different cases. Run the old and new suites on the same outputs and read the disagreements before you trust the new numbers.
3. +Keep the cases, not the config. Export tests to CSV or JSONL first. The cases are the asset, and every alternative reads a file.
4. +Red teaming can stay. Moving evals does not force you to move red teaming. Langfuse's own migration guide says to keep Promptfoo for it.

If you are staying, three settings cover most of the concerns above: pin the version in CI, set PROMPTFOO_DISABLE_TELEMETRY=1, and set PROMPTFOO_DISABLE_REMOTE_GENERATION=true if red-team traffic must not leave your own model accounts.

How to choose

Then port ten test cases to your top choice before you port two hundred. The judged checks are where the surprises are.

FAQ

What is the best Promptfoo alternative? It depends on what you use Promptfoo for. DeepEval for evals as pytest tests with ready-made metrics, Inspect AI for agent evals in sandboxes, Pydantic Evals for typed deterministic checks, vitest-evals for TypeScript, Langfuse, Braintrust or LangSmith for run history and a UI, and DeepTeam, garak or PyRIT for red teaming.

Is Promptfoo still open source after the OpenAI acquisition? Yes. OpenAI announced the acquisition on March 9, 2026. As of November 2026 the repository is MIT licensed, the license file has not changed since January 2025, and the project is still releasing, with 0.125.1 on October 25, 2026.

Should you stop using Promptfoo because OpenAI owns it? Not for that reason alone. Nothing in the code has been shown to favour OpenAI, other providers are still supported, and the MIT license covers the version you run today. Teams with a policy against tools owned by a model provider have a reason to move; others can pin the version and stay.

Is there a Promptfoo alternative for Python? Yes. DeepEval, Inspect AI and Pydantic Evals are Python libraries where evals are code. The promptfoo package on PyPI is a wrapper around the Node CLI and still needs Node 22.22 or newer.

What is the best Promptfoo alternative for red teaming? DeepTeam is the closest: an Apache-2.0 package that red-teams an application with your own model. garak, from NVIDIA, scans models for vulnerabilities, and PyRIT, from Microsoft, is a framework for scripting attacks. None states coverage as wide as Promptfoo's 157 plugins and 34 strategies.

Is there a free open-source alternative to Promptfoo? Yes. DeepEval and vitest-evals (Apache-2.0), Inspect AI and Pydantic Evals (MIT) are free frameworks. Langfuse (MIT, except its enterprise folders) is a free self-hosted platform. DeepTeam, garak and PyRIT are free for red teaming. You still pay the model provider for every call.

Which Promptfoo alternatives are independent of a model provider? DeepEval (Confident AI), Inspect AI (the UK AI Security Institute), Pydantic Evals (Pydantic), Langfuse (ClickHouse) and Braintrust are not owned by a company that sells foundation models. PyRIT belongs to Microsoft and garak to NVIDIA.

How hard is it to migrate from Promptfoo? It is a rewrite, since no tool ships an importer, but a mechanical one. Test cases move as a CSV or JSONL file, deterministic assertions become plain code and give the same results, and judged assertions need re-validating because a different judge passes and fails different cases.

Keep reading

• +Promptfoo vs DeepEval: the two compared on test format, judge costs, repeats, agents, red teaming and CI.
• +The best open-source LLM evaluation frameworks in 2026: ten frameworks compared on license, maintenance and telemetry defaults.
• +Braintrust alternatives: the platforms, compared on hosting, license and price.
• +EvalShift vs Promptfoo: where a paired model comparison fits next to a general eval framework.