Can I use EvalShift and Langfuse together?

Yes, and it is the common case. Langfuse covers production tracing, cost and latency dashboards and live scoring; EvalShift covers the pre-merge question of whether a model or prompt change regresses your golden suite.

Is Langfuse an alternative to EvalShift?

They overlap on datasets, evaluators and CI gating, but they answer different questions. Langfuse is built around what your application did in production; EvalShift is built around whether a candidate model is safe to ship.

Does Langfuse do model migration testing?

It has datasets and experiments that compare prompt, model or code changes side by side, and a GitHub Action that fails the job when your script raises RegressionError. EvalShift is narrower on the same job: paired runs on identical examples, built-in tool-call evaluators, and significance testing over the deltas.

Do I still need an observability tool if I use EvalShift?

Yes. EvalShift runs offline suites against a fixed golden suite and never sees live traffic, so it cannot tell you what is happening in production right now.

Does EvalShift send my prompts and responses to a server?

Only when you run evalshift push. The run, the scoring and the HTML report stay under .evalshift/ on your machine; the push uploads a finalized bundle containing example inputs, both models' outputs, scores and the analysis.

compareevalshift-vs-langfuse

EvalShift vs Langfuse

Langfuse is an open-source AI engineering platform built around tracing what your application did in production. EvalShift is a migration and regression testing tool: it runs one golden suite through two models and decides whether the candidate is safe to ship.

claims checked 2026-08-19Langfuse ↗

// 01

where each one sits

the category ladder, broad to narrow

general LLM evaluation
LLM observability — what production did
LLM migration testing — whether a change is safe to ship
migration testing for agents — tool calls, arguments, ordering

// 02

what each is built for

two jobs, not one job done twice

+ what EvalShift is built around

The SDK records what your agent actually did — model calls, tool calls, retrievals — and evalshift capture sync promotes those captures into a golden suite instead of hand-written fixtures.
One run sends every example through both models, source and target, so each example yields a paired delta rather than two independent averages.
Tool selection, argument correctness, call ordering, parallelism and refusals are built-in scored evaluators, with severity floors per evaluator.
Deltas go through a Shapiro-Wilk screen into a paired t-test or Wilcoxon, with Cohen's d, 95% CIs and Benjamini-Hochberg FDR across every comparison in the run.
A migration policy collapses that into one verdict — budgets for regression rate, critical count, equivalence, tool-argument drift, cost and latency, with per-slice overrides.
The GitHub Action runs the suite on the PR, keeps one comment updated, and fails the check when the policy says the candidate is not safe to ship.

· what Langfuse is for

It traces a request end to end — LLM calls, retrieval steps, tool executions and custom logic, with timing, inputs, outputs and metadata on each observation.
It is OpenTelemetry-based and ships integrations for LangChain, the OpenAI SDK, LiteLLM and others, so instrumenting an app that already exists is mostly configuration.
Cost, latency, volume and quality land in custom dashboards, with alerts when a metric crosses a threshold.
Managed LLM-as-a-judge scores live production traces; annotation queues, manual scores and end-user feedback cover what a judge cannot.
Prompt management, a playground and datasets live in the same app, so a prompt version, the traces it produced and the scores on them stay linked.
The repository is MIT licensed apart from its ee folders, and self-hosts for free via Docker Compose, Kubernetes or Terraform.

// 03

side by side

every row checked against the sources below

dimension

evalshift

Langfuse

Baseline vs candidate

evalshiftOne paired run puts every example through both models; the unit of analysis is the per-example delta.

LangfuseExperiments run a dataset against a configuration; runs are compared side by side in the UI, on aggregate scores.

Statistical significance

evalshiftShapiro-Wilk screen, paired t-test or Wilcoxon, Cohen's d with 95% CI, Benjamini-Hochberg FDR at α=0.05.

LangfuseAggregate scores per run. Its docs describe no significance test or multiple-comparison correction.

Agent tool-call scoring

evalshiftTool selection, arguments, call count, parallelism and refusals ship as evaluators with severity levels.

LangfuseTool executions appear as nested observations in the trace; scoring them is code you write as a custom evaluator.

Ship / no-ship verdict

evalshiftA server-side migration policy — six budgets plus per-slice overrides — that can be edited and re-applied to past runs.

LangfuseYour experiment script raises RegressionError when a score misses a threshold you define in code.

Where raw data lives

evalshiftRun, scoring and HTML report stay under .evalshift/ on your machine; only evalshift push uploads a finalized bundle.

LangfuseTraces are sent to Langfuse Cloud or your self-hosted instance; masking functions can redact fields before they leave the app.

Where test cases come from

evalshiftCaptured production runs, promoted turn by turn into golden.jsonl, deduplicated across syncs.

LangfuseDataset items from the SDK, the UI, CSV upload, or + Add to dataset on any production observation.

PR gate

evalshiftThe Action pushes the run, keeps one PR comment updated, and fails the check on the policy verdict.

Langfuselangfuse/experiment-action runs your experiment script, posts or updates a PR comment, and fails the job on regression.

LLM-as-a-judge

evalshiftPairwise A/B between the two outputs, order-randomized, with an explicit tie score.

LangfuseManaged judges that score live production traces as well as dataset runs, configured in the app.

Production monitoring

evalshiftNone. EvalShift runs offline suites before a change ships and does not watch live traffic.

LangfuseThe core of the product — every request traced, with cost, latency, volume and quality dashboards and threshold alerts.

Human review loop

evalshiftNone. Judgement is encoded in evaluators and policy budgets, not collected from reviewers.

LangfuseAnnotation queues, manual scores and notes, plus end-user feedback collected from your app.

// 04

which one to reach for

honest routing — both answers are real

reach for evalshift when

You are swapping models or providers and need evidence the candidate holds before it reaches users.
What breaks is agent behaviour — a tool that stops being called, an argument that drifts, an extra round — and text-only scoring reads green.
You want the merge blocked by a stated budget, not by someone remembering to read a dashboard.
Raw prompts and responses should stay on your machine, with only finalized run data pushed anywhere.

reach for Langfuse when

You need to know what production is doing right now — which requests are slow, which are expensive, which failed.
The question is why one specific user's request went wrong, and only the full trace answers it.
You want prompt versions, the traces they produced and their scores in one place, with a playground next to them.
Your quality loop is human — annotation queues, labelled examples, end-user feedback — rather than a comparison of two model versions.

They sit at opposite ends of the same loop, and running both is normal. Langfuse watches production and tells you which requests went wrong; EvalShift takes those cases and asks whether a model or prompt change makes them worse, before the change ships. A workable split: keep Langfuse instrumented in production for tracing, cost and live judge scores, run the EvalShift SDK alongside it so the agent's model and tool calls also land in .evalshift/captures/, promote those into a golden suite, and let the EvalShift Action gate the PR that changes the model. The two SDKs are independent — Langfuse exports spans to its server, EvalShift writes JSON captures to disk — so running both is a configuration question, not an integration.

// 05

questions

the ones people ask

Can I use EvalShift and Langfuse together?: Yes, and it is the common case. Langfuse covers production tracing, cost and latency dashboards and live scoring; EvalShift covers the pre-merge question of whether a model or prompt change regresses your golden suite.
Is Langfuse an alternative to EvalShift?: They overlap on datasets, evaluators and CI gating, but they answer different questions. Langfuse is built around what your application did in production; EvalShift is built around whether a candidate model is safe to ship.
Does Langfuse do model migration testing?: It has datasets and experiments that compare prompt, model or code changes side by side, and a GitHub Action that fails the job when your script raises RegressionError. EvalShift is narrower on the same job: paired runs on identical examples, built-in tool-call evaluators, and significance testing over the deltas.
Do I still need an observability tool if I use EvalShift?: Yes. EvalShift runs offline suites against a fixed golden suite and never sees live traffic, so it cannot tell you what is happening in production right now.
Does EvalShift send my prompts and responses to a server?: Only when you run evalshift push. The run, the scoring and the HTML report stay under .evalshift/ on your machine; the push uploads a finalized bundle containing example inputs, both models' outputs, scores and the analysis.

sources

run your own migration diff

the CLI is open source and runs offline — no account, no upload, nothing leaves your machine until you push a run.

get started →how scoring works