## Promptfoo alternatives in 2026: evals, red teaming and run history URL: https://www.evalshift.dev/blog/promptfoo-alternatives Published: 2026-11-24 Tag: tools Summary: Promptfoo alternatives for evals, red teaming and run history, what changed after OpenAI acquired Promptfoo, and how to move a promptfooconfig.yaml. Takeaways: - The best Promptfoo alternative depends on the part you are replacing: as of November 2026, DeepEval, Inspect AI or Pydantic Evals for evals in Python, vitest-evals for TypeScript, Langfuse, Braintrust or LangSmith for run history and a shared UI, and DeepTeam, garak or PyRIT for red teaming. - OpenAI announced it was acquiring Promptfoo on March 9, 2026. As of November 2026 Promptfoo is still MIT licensed, its license file is unchanged since January 2025, it still supports other providers, and its latest release is 0.125.1 from October 25, 2026. - OpenAI's own hosted Evals product is read-only and shuts down on November 30, 2026, and OpenAI's deprecations page names Promptfoo as the replacement. - Promptfoo red teaming generates attacks remotely by default: without an OpenAI key, generation and grading go to Promptfoo's API, and 11 of its 34 strategies use Promptfoo's hosted inference even with your own key. - No tool ships an importer for a promptfooconfig.yaml. Deterministic assertions port to plain code with the same results; judged assertions must be re-validated, because a different judge passes and fails different cases. The best Promptfoo alternative depends on which part of Promptfoo you want to replace. As of November 2026: for evals in Python, DeepEval if you want ready-made metrics in pytest, Inspect AI for agent evals in sandboxes, or Pydantic Evals for typed, deterministic checks; for evals in TypeScript, vitest-evals; for run history and a UI your whole team can open, Langfuse, Braintrust or LangSmith; for red teaming, DeepTeam, garak or PyRIT. The reason most teams started looking in 2026 is that OpenAI announced it was acquiring Promptfoo on March 9. Promptfoo is still MIT licensed and still releasing, and OpenAI now sends users of its own retiring Evals product to it. So before the list, this post covers what changed after the acquisition and what did not, the other reasons teams move, and what Promptfoo still does better than anything here. It ends with how to translate a `promptfooconfig.yaml` into each kind of alternative. A disclosure: I build EvalShift. It replaces one narrow use of Promptfoo, comparing two models before a swap, and it is on the list for that job only. Every claim about another tool comes from that tool's docs, repository or pricing page, read on November 13, 2026. ## Why teams look for a Promptfoo alternative Promptfoo is a good tool, and none of these reasons is that it stopped working. 1. **OpenAI owns it.** Some teams do not want the tool that compares OpenAI, Anthropic and Google models, or that red-teams an agent built on one of them, to belong to one of the three. That is a policy position, not a defect: nothing in the code has been shown to favour OpenAI. What changed and what did not is in the next section. 2. **Node and YAML.** Promptfoo needs Node 22.22 or newer. The [`promptfoo` package on PyPI](https://pypi.org/project/promptfoo/) is a wrapper that still needs Node and runs the CLI through `npx`. Tests live in a YAML file, with JavaScript or Python snippets inside it for custom logic. Python teams often want evals as code next to the code they test. 3. **Results have no shared home.** The viewer is local. The open-source server can be self-hosted, but the [docs](https://www.promptfoo.dev/docs/usage/self-hosting/) say that is not recommended for production: it stores results in SQLite, has no built-in authentication or roles, and cannot run more than one replica. Dashboards, SSO and continuous monitoring are on the [Enterprise plan](https://www.promptfoo.dev/pricing/), which is priced on request. 4. **Red teaming leans on Promptfoo's servers.** Your target is always called locally, but [attack generation is remote by default](https://www.promptfoo.dev/docs/red-team/configuration/). Without an OpenAI key, generation and grading go to Promptfoo's API, and the [data-handling page](https://www.promptfoo.dev/docs/red-team/troubleshooting/data-handling/) lists what goes with them: your application's purpose, the prompts sent to your target, its responses and your email address. Plugins and strategies marked remote in the docs use hosted inference even with your own key, including 11 of the 34 strategies. You can turn remote generation off, and the docs warn that quality drops when you do. 5. **Telemetry is opt-out.** Usage telemetry is on by default and `PROMPTFOO_DISABLE_TELEMETRY=1` turns it off. An [open issue](https://github.com/promptfoo/promptfoo/issues/9968) reports one request still going out with it disabled; that is a user report I have not reproduced. 6. **It checks before release only.** Promptfoo evaluates what you run it on. Scoring live traffic and alerting on it is an Enterprise feature. One reason that no longer holds: "it has no statistics on repeated runs". Since version 0.125.0, from October 2026, `--repeat` reports a Wilson 95% confidence interval on each repeated test's pass rate. What it still does not do is test whether two models differ by more than chance. ## What changed after OpenAI acquired Promptfoo The facts, as of November 13, 2026: | | Status | | --- | --- | | Announcement | [OpenAI](https://openai.com/index/openai-to-acquire-promptfoo/) and [Promptfoo](https://www.promptfoo.dev/blog/promptfoo-joining-openai/) both posted on March 9, 2026. No price was disclosed. | | What was promised | OpenAI said it would keep building the open-source project. Promptfoo said it would remain open source and keep supporting a range of providers and models. | | License | Still MIT. The license file was last changed in January 2025. | | Repository | Still under the `promptfoo` organization, 27.1k stars, with commits this week. | | Releases | 0.125.1 on October 25, 2026; eight releases in the last three months. | | Providers | Anthropic, Google, Mistral, Azure, Bedrock, Ollama and others are still supported. | | Pricing | Community (free), Enterprise and On-Premise, same shape as before. | | Forks | No community fork with a following. The most-starred fork has 11 stars and predates the deal. | | OpenAI's own Evals | The [deprecations page](https://developers.openai.com/api/docs/deprecations) lists the Evals dashboard and API as read-only, shutting down November 30, 2026, with "Migrate to Promptfoo" as the replacement. | Two things did change. The homepage now leads with agent security, with evals as one product among several. And the open-source project's future is now a decision made inside OpenAI, which has said the technology will be integrated into its Frontier enterprise platform. A fair reading: the acquisition has not yet made Promptfoo worse at anything, and the MIT license means the code you depend on today cannot be taken back. What the license does not promise is which provider gets a new feature first, or how much attention the eval side gets next year. If that uncertainty is acceptable, you do not need an alternative. Pin the version and carry on. ## What Promptfoo does well Worth listing before replacing it, because an alternative has to cover what you actually use: - **The matrix.** One config runs every prompt against every model and every test case, and a local web viewer shows the outputs side by side. No alternative here does this as directly. - **Deterministic assertions.** About 40 assertion types run without a model: string, regex, JSON and SQL checks, text-overlap scores, cost, latency, and trajectory checks on an agent's tool calls. A suite can run with no judge at all. - **Red teaming.** [157 plugins](https://www.promptfoo.dev/docs/red-team/plugins/) and [34 strategies](https://www.promptfoo.dev/docs/red-team/strategies/), with 10,000 probes a month free. No open-source alternative states coverage that wide. - **Any language.** The application under test can be anything that answers over HTTP or a script. - **CI.** An exit code, a pass-rate threshold and an [official GitHub Action](https://github.com/promptfoo/promptfoo-action) that comments on the pull request. ## Promptfoo alternatives at a glance | Tool | Replaces | License | Runs on | Latest release | Owner | | --- | --- | --- | --- | --- | --- | | [DeepEval](#deepeval) | Evals | Apache-2.0 | Python, pytest | 4.3.4, Nov 11, 2026 | Confident AI | | [Inspect AI](#inspect-ai) | Evals, agent evals | MIT | Python | 0.3.289, Nov 10, 2026 | UK AI Security Institute | | [Pydantic Evals](#pydantic-evals) | Evals | MIT | Python | 2.68.0, Nov 12, 2026 | Pydantic | | [vitest-evals](#vitest-evals) | Evals | Apache-2.0 | TypeScript, Vitest | 0.18.1, Nov 7, 2026 | Sentry | | [Langfuse](#langfuse) | Evals with history | MIT, except `ee` | Self-hosted or cloud | v4.80.0, Nov 13, 2026 | ClickHouse | | [Braintrust](#braintrust-and-langsmith) | Evals with history | Proprietary | Cloud | Hosted service | Braintrust | | [LangSmith](#braintrust-and-langsmith) | Evals with history | Proprietary | Cloud | Hosted service | LangChain | | [DeepTeam](#deepteam) | Red teaming | Apache-2.0 | Python | 1.0.10, Oct 16, 2026 | Confident AI | | [garak](#garak) | Red teaming | Apache-2.0 | Python | 0.17.1, Oct 16, 2026 | NVIDIA | | [PyRIT](#pyrit) | Red teaming | MIT | Python | 1.2.0, Nov 13, 2026 | Microsoft | | [EvalShift](#evalshift) | Comparing two models | Apache-2.0 | Python | 1.6.2, Oct 27, 2026 | EvalShift | Promptfoo itself, for reference: MIT, Node 22.22+, 0.125.1 from October 25, 2026, owned by OpenAI. ## Alternatives for evals ### DeepEval [DeepEval](https://deepeval.com/docs/getting-started) is the alternative most Python teams try first. An eval is a pytest test: build a test case, attach metrics, and `deepeval test run` fails the test when a metric misses its threshold. It ships more than 50 metrics for RAG, agents, conversations and safety, and it now has a TypeScript package on npm. What you give up: the matrix and the local viewer. And where Promptfoo's checks are mostly code, DeepEval's [metrics page](https://deepeval.com/docs/metrics-introduction) says almost all of its metrics are graded by an LLM judge, so a run costs at least one judge call per metric per case. Run history and comparison are in [Confident AI](https://www.confident-ai.com/pricing), the proprietary platform from the same team: free for 2 seats and 5 test runs a week, then $200 a month per organization. Telemetry is on by default here too, and `DEEPEVAL_TELEMETRY_OPT_OUT=1` turns it off. **Choose it when** you want evals in pytest and a metric library you do not have to write. [Promptfoo vs DeepEval](/blog/promptfoo-vs-deepeval) compares the two in full. ### Inspect AI [Inspect AI](https://inspect.aisi.org.uk/) comes from the UK AI Security Institute. A task is a dataset, a solver and a scorer in Python. It runs agents inside Docker or Kubernetes sandboxes, repeats each sample with `epochs`, and reports a standard error with every score. It is MIT, has no commercial edition, and its docs describe no telemetry. What you give up: the prompt matrix, a red-team scanner, and a documented way to fail a pull request on a threshold. You write that last part yourself from the log files. **Choose it when** you evaluate agents that run code or use a computer, you want error bars without building them, or you want a tool with no vendor behind it. ### Pydantic Evals [Pydantic Evals](https://github.com/pydantic/pydantic-ai/blob/main/docs/evals.md) is the evaluation library in the Pydantic AI repository. A `Dataset` holds typed `Case` objects, and evaluators check the output or the OpenTelemetry spans your code emitted, so "the agent called `lookup_order` before `refund`" is a deterministic check on a span tree. An LLM judge is available when you need one. What you give up: a CLI and an exit-code gate (you run it from your own test or script), the matrix and red teaming. The hosted results view is Logfire, Pydantic's paid observability product, with a free personal tier. **Choose it when** you already use Pydantic and want most checks to be code. ### vitest-evals [vitest-evals](https://github.com/getsentry/vitest-evals) is the option for TypeScript teams who want to leave YAML but stay in Node. Evals are Vitest tests with scorers, and its GitHub Action gates on a pass rate. It is a small project, at 0.18.1 with a few hundred stars, so read the source before you depend on it. **Choose it when** your application and its tests are TypeScript and you want evals in the same runner. ### One to skip, one to check - **OpenAI Evals** is not an alternative. The hosted product is read-only and shuts down on November 30, 2026, and OpenAI's migration guide points to Promptfoo. The open-source `openai/evals` package last released in May 2024. - **Ragas** has the RAG metrics many teams want, but its last release was January 13, 2026 and its last commit February 24. Pin the version if you use it, and note that DeepEval and Promptfoo both ship faithfulness, context recall and answer relevance checks of their own. ## Alternatives when you need history and a UI If the reason you are leaving is that results live on one laptop, a framework will not fix it. These are platforms: they store every run, show two side by side and let non-engineers review outputs. All of them also trace production, which Promptfoo's open-source edition does not. ### Langfuse [Langfuse](https://langfuse.com/pricing) is MIT licensed apart from its enterprise folders, and self-hosts for free with all core features. Its cloud has a free tier of 50,000 units a month and a first paid plan at $29 a month with unlimited users. It is also the only tool here that publishes a [migration guide from Promptfoo](https://langfuse.com/resources/engineering/migrate-from-promptfoo). The guide supplies example scripts, not an importer, and it says to keep Promptfoo for red teaming, which Langfuse does not do. **Choose it when** you want eval runs next to production traces in something you can host yourself. ### Braintrust and LangSmith [Braintrust](https://www.braintrust.dev/pricing) is built around experiments: an `Eval()` with a dataset, a task and scorers, compared row by row against a base run, with a GitHub Action that posts the result. It is proprietary and cloud-hosted, free for 1 GB of data and 10,000 scores a month, then $249 a month. [LangSmith](https://www.langchain.com/pricing) runs datasets as pytest, Vitest or Jest tests and has ready-made trajectory evaluators for agents. It is proprietary, free for one seat and 5,000 traces a month, then $39 per seat. **Choose one of them when** you would rather not operate a platform. [Braintrust vs LangSmith](/blog/braintrust-vs-langsmith) compares the two. Opik (Apache-2.0) and Arize Phoenix (Elastic License 2.0) are the other self-hostable options. [Braintrust alternatives](/blog/braintrust-alternatives) covers both. ## Alternatives for red teaming This is the part of Promptfoo that is hardest to replace, and no alternative matches its plugin count. What the alternatives offer is attack generation that runs on models you choose. ### DeepTeam [DeepTeam](https://github.com/confident-ai/deepteam) is the red-teaming package from the DeepEval team: Apache-2.0, version 1.0.10. Its README claims more than 50 vulnerability types and more than 20 attack methods, mapped to the OWASP Top 10 for LLMs, NIST AI RMF and MITRE ATLAS. It generates and grades attacks with your own model, from Python or a YAML file. It releases less often than DeepEval: four times since July 2026. **Choose it when** you want the closest thing to Promptfoo's application-level red teaming with no hosted generation step. ### garak [garak](https://github.com/NVIDIA/garak), from NVIDIA, is a vulnerability scanner for models. You point the CLI at a model endpoint and it runs probes for prompt injection, jailbreaks, data leakage, toxicity and more, then reports which ones landed. It is Apache-2.0 with 10k stars. It tests a model more than an application: it does not know your system's purpose or tools the way a Promptfoo red-team config does. **Choose it when** you are assessing a model, including one you host yourself, before building on it. ### PyRIT [PyRIT](https://github.com/microsoft/PyRIT) is Microsoft's red-teaming framework, MIT licensed. It began as a Python library for security researchers building multi-turn attacks, and now also has a scanner CLI and a browser interface. It asks more of you than the others: you assemble targets, converters and scorers yourself. Note that it does not answer a neutrality concern any better than Promptfoo does. Microsoft sells models too. **Choose it when** you have a security team that wants to script its own attacks. ### Giskard [Giskard](https://docs.giskard.ai/oss/scan) version 3, Apache-2.0, generates attacks from a plain-language description of your application and runs them as pytest scenarios. Version 2 is no longer maintained and version 3 is a rewrite, at 3.0.1 since October 2026, so treat it as a new project. One more thing on the red-teaming search results: Promptfoo's own "Promptfoo vs PyRIT" page ranks for this query and dates from June 2025, before PyRIT had a CLI. ## Alternative for comparing two models ### EvalShift Many teams use Promptfoo's matrix for one recurring job: the model under a production agent is being retired or replaced, and they need to know whether the candidate behaves the same. The matrix shows the two models side by side and scores each on its own. [EvalShift](/docs) is built for that job only. It runs your recorded production cases through both models, scores tool selection, arguments and trace rules on each side, and tests the per-example differences with a paired t-test or Wilcoxon, an effect size with a confidence interval and a false-discovery-rate correction. A [migration policy](/docs/migration-policy) turns that into a verdict that passes or fails the pull request. What you give up is most of Promptfoo: there is no red teaming, no five-model matrix and a much smaller set of checks. The CLI is Apache-2.0, has no telemetry and runs on your machine against your own provider keys; the hosted run history is a paid plan. **Choose it when** the question is "is this model swap safe", and keep another tool for everything else. [EvalShift vs Promptfoo](/compare/evalshift-vs-promptfoo) goes dimension by dimension. ## Moving off Promptfoo Only Langfuse publishes a migration guide, and nobody ships an importer, so a move is a rewrite. It is a mechanical one, because a `promptfooconfig.yaml` has few concepts: | In `promptfooconfig.yaml` | In a Python framework | On a platform | | --- | --- | --- | | `prompts` | The function under test, or a prompt constant | A versioned prompt | | `providers` | A parameter of that function; parametrize the test over models | One experiment run per model | | `tests` and `vars` | Parametrized cases, `Case` objects or a JSONL dataset | Dataset items | | `contains`, `regex`, `is-json`, `javascript`, `python` | A plain `assert`, or a code evaluator | A code evaluator | | `llm-rubric`, `g-eval`, `factuality` | A judged metric such as G-Eval | An LLM-as-a-judge evaluator | | `defaultTest` | A fixture or a shared list of evaluators | Evaluators attached to the experiment | | Exit code and pass threshold | The pytest exit code | The platform's CI action | | `--repeat` | `epochs` in Inspect AI, `--repeat` in DeepEval | Repetitions on the experiment | | `redteam:` | DeepTeam, garak or PyRIT, or keep Promptfoo for this | Not covered | A Promptfoo test with one deterministic and one judged assertion: ```yaml tests: - vars: message: Where is my order 4417? assert: - type: icontains value: '4417' - type: llm-rubric value: Does not promise a delivery date ``` becomes this in pytest with DeepEval: ```python import pytest from deepeval import assert_test from deepeval.metrics import GEval from deepeval.test_case import LLMTestCase, SingleTurnParams from app import reply # your function that calls the model no_promise = GEval( name="No delivery promise", criteria="The actual output does not promise a delivery date.", evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT], ) @pytest.mark.parametrize("message, must_contain", [("Where is my order 4417?", "4417")]) def test_reply(message, must_contain): output = reply(message) assert must_contain.lower() in output.lower() # was: icontains assert_test(LLMTestCase(input=message, actual_output=output), [no_promise]) # was: llm-rubric ``` Four things to plan for: 1. **Deterministic checks port cleanly.** They become ordinary assertions and will give the same results. 2. **Judged checks do not.** A rubric graded by a different judge prompt, model or threshold will pass and fail different cases. Run the old and new suites on the same outputs and read the disagreements before you trust the new numbers. 3. **Keep the cases, not the config.** Export `tests` to CSV or JSONL first. The cases are the asset, and every alternative reads a file. 4. **Red teaming can stay.** Moving evals does not force you to move red teaming. Langfuse's own migration guide says to keep Promptfoo for it. If you are staying, three settings cover most of the concerns above: pin the version in CI, set `PROMPTFOO_DISABLE_TELEMETRY=1`, and set `PROMPTFOO_DISABLE_REMOTE_GENERATION=true` if red-team traffic must not leave your own model accounts. ## How to choose | If you are leaving because | Look at | | --- | --- | | You want an eval tool no model provider owns | DeepEval, Inspect AI or Pydantic Evals | | You want evals in pytest | DeepEval | | You evaluate agents that run code | Inspect AI | | You want typed, mostly deterministic checks | Pydantic Evals | | You want to stay in TypeScript without YAML | vitest-evals | | Results need a shared home you can self-host | Langfuse | | Results need a shared home and you would rather not run it | Braintrust or LangSmith | | Red-team traffic must stay in your own model accounts | DeepTeam | | You are scanning a model, not an application | garak | | You only used the matrix to check model swaps | EvalShift | | You are not sure you need to leave | Stay, pin the version, turn off telemetry | Then port ten test cases to your top choice before you port two hundred. The judged checks are where the surprises are. ## FAQ **What is the best Promptfoo alternative?** It depends on what you use Promptfoo for. DeepEval for evals as pytest tests with ready-made metrics, Inspect AI for agent evals in sandboxes, Pydantic Evals for typed deterministic checks, vitest-evals for TypeScript, Langfuse, Braintrust or LangSmith for run history and a UI, and DeepTeam, garak or PyRIT for red teaming. **Is Promptfoo still open source after the OpenAI acquisition?** Yes. OpenAI announced the acquisition on March 9, 2026. As of November 2026 the repository is MIT licensed, the license file has not changed since January 2025, and the project is still releasing, with 0.125.1 on October 25, 2026. **Should you stop using Promptfoo because OpenAI owns it?** Not for that reason alone. Nothing in the code has been shown to favour OpenAI, other providers are still supported, and the MIT license covers the version you run today. Teams with a policy against tools owned by a model provider have a reason to move; others can pin the version and stay. **Is there a Promptfoo alternative for Python?** Yes. DeepEval, Inspect AI and Pydantic Evals are Python libraries where evals are code. The `promptfoo` package on PyPI is a wrapper around the Node CLI and still needs Node 22.22 or newer. **What is the best Promptfoo alternative for red teaming?** DeepTeam is the closest: an Apache-2.0 package that red-teams an application with your own model. garak, from NVIDIA, scans models for vulnerabilities, and PyRIT, from Microsoft, is a framework for scripting attacks. None states coverage as wide as Promptfoo's 157 plugins and 34 strategies. **Is there a free open-source alternative to Promptfoo?** Yes. DeepEval and vitest-evals (Apache-2.0), Inspect AI and Pydantic Evals (MIT) are free frameworks. Langfuse (MIT, except its enterprise folders) is a free self-hosted platform. DeepTeam, garak and PyRIT are free for red teaming. You still pay the model provider for every call. **Which Promptfoo alternatives are independent of a model provider?** DeepEval (Confident AI), Inspect AI (the UK AI Security Institute), Pydantic Evals (Pydantic), Langfuse (ClickHouse) and Braintrust are not owned by a company that sells foundation models. PyRIT belongs to Microsoft and garak to NVIDIA. **How hard is it to migrate from Promptfoo?** It is a rewrite, since no tool ships an importer, but a mechanical one. Test cases move as a CSV or JSONL file, deterministic assertions become plain code and give the same results, and judged assertions need re-validating because a different judge passes and fails different cases. ## Keep reading - [Promptfoo vs DeepEval](/blog/promptfoo-vs-deepeval): the two compared on test format, judge costs, repeats, agents, red teaming and CI. - [The best open-source LLM evaluation frameworks in 2026](/blog/open-source-llm-evaluation-frameworks): ten frameworks compared on license, maintenance and telemetry defaults. - [Braintrust alternatives](/blog/braintrust-alternatives): the platforms, compared on hosting, license and price. - [EvalShift vs Promptfoo](/compare/evalshift-vs-promptfoo): where a paired model comparison fits next to a general eval framework.