The best LLM evaluation tools in 2026, compared by the job they do

The best LLM evaluation tool depends on the job. As of November 2026: for comparing prompts and models, and for red teaming, Promptfoo; for prebuilt quality metrics inside a pytest suite, DeepEval, with Ragas for RAG pipelines; for agent benchmarks run in sandboxes with error bars, Inspect AI; for tracing production and scoring it with evals in one place, Langfuse, Arize Phoenix or Opik if you want to self-host, and LangSmith or Braintrust if you want a commercial platform; for teams already on Databricks, MLflow. For one narrow job, checking that a production agent still behaves the same after a model swap, EvalShift.

Most teams end up with two of these, not one: something that watches production, and something that checks a change before it merges. The rest of this post compares the tools on what they grade, how they compare two versions, how they plug into CI, where your data goes and what they cost, and lists what changed in the category this year, including OpenAI shutting down its hosted Evals product on November 30, 2026.

A disclosure: I build EvalShift, which is on this list. It covers one narrow job, and its section says what it does not do. Every claim about another tool links to that tool's own docs, checked in November 2026.

LLM evaluation tools at a glance

Three more come up in searches and are covered briefly at the end: TruLens, W&B Weave and Galileo, along with OpenAI Evals and Humanloop, which are no longer options for new projects.

LLM evaluation framework vs LLM evaluation platform

The tools split into two shapes, and the split matters more than any feature list.

An LLM evaluation framework is a library or CLI you run in your own code: you write test cases, pick or write scorers, and run the suite locally or in CI. Promptfoo, DeepEval, Inspect AI and Ragas are frameworks. They are free, your data stays on your machine unless you opt into a hosted dashboard, and they fit naturally into a pull-request check. What they do not do is watch production.

An LLM evaluation platform starts from tracing. Your application sends every request, model call and tool call to a server, and evals run over those traces: on datasets before release, and on sampled live traffic after it. LangSmith, Langfuse, Braintrust, Phoenix, Opik and MLflow are platforms. They are where you find out what went wrong for a specific user and turn that case into a test. The cost is a server, either theirs or one you host, and an SDK in your app.

The line is blurring, since most frameworks now have a hosted dashboard and most platforms now have a CI integration. A useful rule still holds: pick a platform for what production is doing, and make sure something in your stack can fail a pull request before a change ships.

What to look for in an LLM evaluation tool

Six questions separate the tools faster than a feature checklist:

1. +What does it grade? The final text only, or the trace: which tools an agent called, with which arguments, in what order. For an agent, a correct-sounding reply after the wrong tool call is the failure that matters most.
2. +Where do test cases come from? Hand-written in config, generated synthetically, or collected from production traces. Suites built from real traffic find the failures users will actually hit; build a golden suite from production traffic covers that loop.
3. +How does it compare two versions? Almost every tool shows two runs side by side. Few tell you whether the difference between them is larger than the noise. More on that below.
4. +Can it gate a pull request? A GitHub Action, a pytest exit code, or a threshold in a script that fails the job.
5. +Where does your data go? Locally only, to a vendor's cloud, or to a server you host.
6. +What can you run without paying? Open-source license, free self-hosting, or a free cloud tier with limits.

Agent and CI support compared

"No" in this table means the tool's documentation does not describe the feature as of November 2026, not that it cannot be built on top.

Promptfoo

Promptfoo ↗ is an open-source CLI for evaluating and red teaming LLM apps. You describe prompts, providers and test cases in a YAML file, run promptfoo eval, and get a matrix of every prompt against every model, which makes it the quickest way to line several candidates up on your own data. Its assertion catalog runs to dozens of types, from contains and JSON schema checks to llm-rubric, and its trajectory assertions ↗ check which tools an agent used, whether the arguments matched, the order and the step count.

Two things set it apart. Red teaming is a first-class part of the product: it generates adversarial inputs, runs them and reports the risk. And evals run locally against your own provider keys. An official GitHub Action ↗ runs a before-and-after eval on a pull request and comments the result.

Promptfoo is now part of OpenAI, which announced the acquisition ↗ on March 9, 2026. The README ↗ says it remains open source under MIT. Requires Node 22.22 or newer.

Choose it when you are picking a model or a prompt, you need red teaming, or you want assertions off the shelf in a config file. EvalShift vs Promptfoo goes dimension by dimension.

DeepEval

DeepEval ↗ is an open-source Python library that treats evals as unit tests: you write assert_test calls with metrics and run them with deepeval test run, which wraps pytest and fails when a metric misses its threshold. It ships 50+ metrics across RAG, agents, conversations and safety, plus G-Eval and DAG for writing a custom judged criterion. For agents it has Tool Correctness and Argument Correctness at the component level, and Task Completion, Step Efficiency and Plan Adherence over the whole trajectory. A Synthesizer generates test cases from documents when you have none yet.

A TypeScript version ↗ runs under Vitest and is in beta. Red teaming lives in a separate open-source project, DeepTeam. Results and production tracing go to Confident AI, the commercial platform from the same team, after deepeval login; its free plan covers 2 seats and 5 test runs a week, and paid plans start at $200 a month.

Choose it when your evals belong in the pytest suite the team already runs, and you would rather pick a metric than write one. EvalShift vs DeepEval has the full comparison.

Inspect AI

Inspect AI ↗ is an open-source Python framework from the UK AI Security Institute, built for rigorous evaluations of models and agents. An eval is a @task that combines a dataset, a solver (the prompt chain or agent) and a scorer; you run it with inspect eval and browse the logs with inspect view. It has a built-in ReAct agent, bash and editor tools, MCP support, and sandboxes in Docker, Kubernetes or Modal, so an agent can act in a real environment that is reset between samples. Inspect Evals collects more than 200 prebuilt benchmark implementations.

It is the only tool on this list whose docs describe uncertainty as a built-in part of every result: metrics ↗ include standard errors (clustered or bootstrapped), confidence intervals and Wilson intervals, and repeated epochs can be reduced to pass@k.

Choose it when you are running capability or safety evals in sandboxes, or reproducing public benchmarks, and you want error bars on the numbers. It is not built for production tracing or a pull-request gate.

Ragas

Ragas ↗ is an open-source Python library of evaluation metrics, best known for RAG: faithfulness, context precision, context recall, answer relevancy. It has about 30 metrics in all, including four for agents (Tool Call Accuracy, Tool Call F1, Agent Goal Accuracy and Topic Adherence), and can generate synthetic test sets from your documents. It is a library, not a platform: no dashboard, CI integration or tracing of its own, and its metrics are often run inside other tools, such as MLflow. The repository now lives at vibrantlabsai/ragas ↗.

Choose it when the question is whether your retrieval and grounding are good, and you want the metrics without a platform around them.

LangSmith

LangSmith ↗ is LangChain's platform for tracing, evaluating, deploying and monitoring agents. It is framework-agnostic: it traces the OpenAI, Anthropic and Vercel AI SDKs, plain code and OpenTelemetry spans, not only LangChain. Datasets run as experiments scored by code, LLM-as-a-judge, human or pairwise evaluators, and a comparison view marks each example that improved or regressed against a baseline experiment. Evals can also run as pytest, Vitest or Jest tests. Online evaluators score live production runs, and annotation queues route traces to human reviewers.

For agents, the open-source agentevals ↗ package matches a trajectory against a reference in strict, unordered, subset or superset mode, or has an LLM judge it. The Developer plan is free for one seat and 5,000 traces a month; Plus is $39 per seat per month. Self-hosting is an add-on to the Enterprise plan.

Choose it when you want tracing, evals, prompts and deployment on one commercial platform, especially if you build on LangChain or LangGraph. EvalShift vs LangSmith compares the two.

Langfuse

Langfuse ↗ is an open-source LLM engineering platform built around tracing. It records each request end to end, with model calls, retrievals and tool executions as nested observations, on OpenTelemetry. On top of the traces it runs LLM-as-a-judge and code evaluators, on datasets before release and on live traffic after it, alongside prompt management, a playground and annotation queues. Evaluators can read structured tool calls, and trajectory checks such as step counts and ordering are code you write.

Its experiment-action ↗ for GitHub runs your experiment script on a pull request, comments the result and fails the job when the script raises a RegressionError. The code is MIT licensed except for its ee folders, and self-hosting is free. The Cloud Hobby tier is free; paid plans start at $29 a month. Langfuse joined ClickHouse ↗ in January 2026, with no licensing changes planned.

Choose it when you want production tracing and evals on infrastructure you control. EvalShift vs Langfuse covers how the two fit together.

Braintrust

Braintrust ↗ is a commercial eval and observability platform built around experiments. You define an Eval() in TypeScript or Python with a dataset, a task and scorers, run it with the CLI, and compare experiments row by row, with each change marked as an improvement, a regression or a trade-off. Its open-source autoevals ↗ library supplies about 25 scorers (LLM-judged, RAG, heuristic, embedding), and trace-level scorers can read the whole trace of an agent run, tool calls included. Scoring rules evaluate production logs online, and eval-action ↗ posts experiment results on a pull request.

It has the widest SDK coverage here: TypeScript and Python, plus Go, Java, Ruby and C#. The Starter plan is free and Pro is $249 a month. Self-hosting, including a hybrid mode where you run the data plane, is on Enterprise.

Choose it when your team iterates on prompts and models through experiments and wants the comparison workflow, the playground and production scoring in one commercial product.

Arize Phoenix

Phoenix ↗ is Arize's open-source observability and evaluation tool, built on OpenTelemetry and the OpenInference conventions, so traces are portable to any OTel backend. It runs LLM-judged evals from templates, datasets and experiments, and has prebuilt evals for agents' tool selection, tool invocation and handling of tool responses. It is free to self-host with pip, Docker or Helm.

The license is the Elastic License 2.0 ↗, which is source-available rather than OSI open source: fine for running it yourself, restrictive if you want to offer it as a hosted service. Arize AX is the commercial product, with a free tier and Pro at $50 a month.

Choose it when you are standardizing on OpenTelemetry and want tracing and evals you host.

Opik

Opik ↗, from Comet, is an open-source observability and evaluation platform under Apache-2.0, and the full platform is free to self-host. It has 43 built-in metrics, including agent metrics such as Trajectory Accuracy, Agent Task Completion and Tool Correctness, a pytest integration for running evals on every commit, online evaluation rules for production traces, and guardrails for PII and prompt injection. Comet's cloud has a free tier, and Pro starts at $19 a month.

Choose it when you want a platform like Langfuse or Phoenix under a permissive open-source license, with agent metrics built in.

MLflow

MLflow ↗ added GenAI tracing and evaluation to the ML lifecycle platform many data teams already run. mlflow.genai.evaluate() takes a dataset, a prediction function and scorers: built-in LLM judges for correctness, guidelines and groundedness, multi-turn judges, and scorers from DeepEval, Ragas, Phoenix and TruLens. Tool-call correctness and efficiency scorers are experimental. Automatic evaluation runs judges on incoming production traces. MLflow is Apache-2.0 and free to self-host; Databricks runs a managed version, and a few judges are only available there.

Choose it when your team already tracks models in MLflow or works in Databricks.

EvalShift

EvalShift is built for one decision: is it safe to move this agent from the model it runs on today to a candidate? The capture SDK records real model and tool calls from your agent, and evalshift capture sync promotes them into a golden suite. One run sends every example through both models, scores tool selection, arguments and call structure on both sides, holds both to trace rules you write, and tests each difference for significance (paired t-test or Wilcoxon, effect size with a confidence interval, and a false-discovery-rate correction across the run). A migration policy turns that into one verdict, and the GitHub Action fails the pull request when the verdict is fail.

What it does not do: red teaming, production monitoring, RAG retrieval metrics, or a broad library of quality metrics. It runs offline, in Python, and pairs naturally with one of the platforms above for production. The CLI and SDK are free and open source; Pro, the hosted gate with run history and baselines, is $199 a month.

Choose it when you are switching or upgrading the model, or editing the prompt, under an agent that is already in production.

The question most tools leave to you: is the difference real?

Every tool above will show you that a candidate scored 83% and the current version 86%. Almost none will tell you whether that gap is a regression or noise. Of the tools here, only Inspect AI documents standard errors and confidence intervals as part of its results. The comparison views in LangSmith and Braintrust count or color the examples that improved and regressed, and Langfuse compares aggregate scores; their docs describe no significance test.

That matters because eval scores are noisy. On a suite of 200 cases, a pass rate of 86% has a 95% Wilson interval of roughly 80.5% to 90.1%, so a three-point drop sits comfortably inside it. Model outputs also vary between runs of the same input. Two ways to stop reading noise as signal, whatever tool you use:

• +Compare per example, not per run. Run both versions on the same cases and look at the per-case differences. A paired comparison is far more sensitive than two independent pass rates.
• +Run each case more than once, and size the suite so each slice you would act on has enough cases. How many eval cases do you need? works through the numbers.

What changed in LLM evaluation tools in 2025 and 2026

The category consolidated fast. If you are reading an older "best tools" list, check these first:

• +OpenAI's hosted Evals is shutting down. OpenAI deprecated ↗ the Evals dashboard and API in June 2026. Existing evals became read-only on October 31, 2026, and the product shuts down on November 30, 2026. OpenAI's migration guide points to Promptfoo. The open-source openai/evals ↗ repository is a separate project and is not affected.
• +Promptfoo is part of OpenAI since its March 2026 acquisition, and remains MIT licensed.
• +Langfuse joined ClickHouse in January 2026; self-hosting and the license are unchanged.
• +Weights & Biases is part of CoreWeave. The acquisition closed in May 2025, and W&B's products moved into CoreWeave Forge in September 2026. CoreWeave's docs describe Agent Lens, in public preview, as the successor to W&B Weave.
• +Galileo is now Splunk Agent Observability, after Cisco completed its acquisition in May 2026.
• +Humanloop shut down in September 2025. Its founders and part of the team joined Anthropic, and customer data was deleted after September 8, 2025.

Other tools you will see on these lists

• +TruLens ↗: an MIT-licensed eval and tracing library, now maintained by Snowflake, with RAG feedback functions and seven agent metrics, including tool selection and plan adherence.
• +W&B Weave ↗: tracing and weave.Evaluation with scorers, Apache-2.0 SDK; see the CoreWeave note above before starting a new project on it.
• +Galileo ↗: agent metrics such as Tool Selection Quality and Action Completion, plus small evaluator models for guardrails; now sold through Splunk.

How to choose an LLM evaluation tool

Start from the job, not the feature list:

Then check the six questions above against your shortlist, and run two or three tools on the same small suite of your own cases before committing. A tool that grades only final text will report green on an agent that skipped a tool call; what breaks when you switch LLMs shows what that looks like in practice.

FAQ

What are the best LLM evaluation tools? It depends on the job. Promptfoo for comparing prompts and models and for red teaming; DeepEval for metrics in pytest; Ragas for RAG; Inspect AI for agent evals in sandboxes; Langfuse, Phoenix or Opik for self-hosted tracing with evals; LangSmith or Braintrust for a commercial platform; MLflow on Databricks; and EvalShift for gating a model swap under a production agent.

What is an LLM evaluation framework? A library or CLI you run in your own code to test an LLM application: you write test cases, choose scorers such as exact checks, metrics or an LLM judge, and run the suite locally or in CI. Promptfoo, DeepEval, Inspect AI and Ragas are frameworks; platforms such as LangSmith and Langfuse add tracing and production monitoring around evals.

Which LLM evaluation tools are open source? Promptfoo (MIT), DeepEval (Apache-2.0), Inspect AI (MIT), Ragas (Apache-2.0), Langfuse (MIT except its ee folders), Opik (Apache-2.0), MLflow (Apache-2.0) and TruLens (MIT). Arize Phoenix is source-available under the Elastic License 2.0. LangSmith and Braintrust are proprietary platforms with MIT-licensed evaluator libraries. The EvalShift CLI is Apache-2.0, and its SDK and GitHub Action are MIT.

Which LLM evaluation tool is best for AI agents? One that grades the trace, not only the final reply. Promptfoo, DeepEval, LangSmith's agentevals, Opik, Ragas and TruLens ship tool-call or trajectory checks; Inspect AI runs agents in sandboxes; EvalShift compares two models' tool calls on the same recorded runs. Pick by whether you are testing one version or comparing two.

Is OpenAI Evals still available? OpenAI's hosted Evals dashboard and API became read-only on October 31, 2026 and shut down on November 30, 2026, and OpenAI's migration guide points to Promptfoo. The open-source openai/evals repository is separate and still available.

Do I need more than one LLM evaluation tool? Usually two: a platform that traces production and scores live traffic, and a framework or gate that tests a change before it merges. They answer different questions, and most of the tools here run side by side without integration.

Keep reading

• +How to evaluate AI agents: what to measure, at which level, and against which environment, whatever tool you pick.
• +AI evals for agents: what an eval is made of, and how to write one.
• +LLM-as-a-judge: the scorer most of these tools rely on, and how to validate it.
• +LangSmith alternatives: Langfuse vs LangSmith on pricing, self-hosting and retention, and the other platforms to consider.
• +EvalShift compared: side-by-side pages for Promptfoo, DeepEval, LangSmith and Langfuse.