## RAG evaluation: metrics, frameworks and how to test a RAG pipeline

URL: https://www.evalshift.dev/blog/rag-evaluation
Published: 2026-11-21
Tag: rag
Summary: RAG evaluation metrics, frameworks and benchmarks compared, how far to trust an LLM judge, and how to test a retriever or model change in a RAG pipeline.

Takeaways:
- RAG evaluation scores retrieval and generation separately: retrieval with recall@k, nDCG or LLM-judged context precision and recall, generation with faithfulness, answer relevancy and correctness. An end-to-end score alone does not say which half to fix.
- Ragas, DeepEval and Promptfoo use the same metric names with different definitions. DeepEval's faithfulness passes a claim the context never mentions; Ragas's counts it against the answer.
- LLM judges are reliable for faithfulness and weak for context relevance: the RAGAS paper measured 0.95 and 0.70 agreement with human judgments. Label 100 to 200 of your own cases before a judged metric gates a release.
- Compare a retrieval change query by query, not by averages. A new embedding model can raise nDCG and still lose every query that hinges on an exact identifier.
- Test one component at a time: freeze the retrieved context when testing a prompt or generator model change, and freeze the chunks when testing a retriever.

RAG evaluation measures a retrieval-augmented generation pipeline in two halves, then end to
end. The retrieval half asks whether the right chunks came back: recall@k, precision@k, MRR and
nDCG against labeled relevant documents, or LLM-judged context precision and context recall when
you have no labels. The generation half asks whether the answer used those chunks well:
faithfulness (every claim supported by the retrieved context), answer relevancy (it addresses the
question) and correctness against a reference answer. Score the halves separately, because an
end-to-end score of 70% does not tell you whether to fix the retriever or the prompt.

The tools that compute these metrics, Ragas, DeepEval, TruLens, MLflow, Phoenix and Promptfoo
among them, agree on the names and disagree on the definitions. This post covers what each
metric measures and needs, how the frameworks differ, how far to trust an LLM judge for this, how
to build a test set, and how to test a change to one part of the pipeline without the rest moving.
It includes a short script, tested on a small fixture set, that compares two retrievers query by
query with a confidence interval, a step the top-ranking guides I read leave out.

A disclosure: I build EvalShift, a tool for testing model changes on agents. It has a short section
at the end, which says what it does not do for RAG. Everything before it works without it.

## Evaluate retrieval and generation separately

A wrong RAG answer has one of three causes, and each has a different fix:

| What went wrong | What you see | Where the fix is |
| --- | --- | --- |
| The relevant chunk was not retrieved | Low context recall; the answer is wrong or a refusal | Chunking, embedding model, hybrid search, query rewriting, top-k |
| It was retrieved, and the model ignored or misread it | High context recall, low faithfulness or correctness | Prompt, context order, the generator model |
| Nothing relevant exists in the corpus | No retriever can find it; the right answer is "I don't know" | The corpus, and the model's willingness to abstain |

The second row is more common than it sounds. In [Lost in the Middle](https://aclanthology.org/2024.tacl-1.9.pdf)
(Liu et al., TACL 2024), models used information at the start and end of a long context much better
than information in the middle: GPT-3.5-Turbo's multi-document QA accuracy dropped by more than 20%
when the answer moved to the middle, and with 20 or 30 documents and the answer in the middle it
fell below its closed-book score of 56.1%. Retrieving the right chunk is necessary and not sufficient. Raising top-k improves recall
and can lower answer quality at the same time.

The third row is the one test sets forget. If every question in your suite has an answer in the
corpus, you never measure whether the system admits it does not know. RGB
([Chen et al., AAAI 2024](https://arxiv.org/abs/2309.01431)) calls this negative rejection, and
found models struggled with it, as with integrating information across documents and with
retrieved text that contradicts what they know.

## RAG evaluation metrics

| Metric | Measures | Needs | Model calls |
| --- | --- | --- | --- |
| Hit rate (success@k) | Share of queries with at least one relevant chunk in the top k | Relevant chunk IDs per query | None |
| Recall@k | Share of the relevant chunks that are in the top k | Relevant chunk IDs | None |
| Precision@k | Share of the top k that is relevant | Relevant chunk IDs | None |
| MRR | 1 / rank of the first relevant chunk, averaged | Relevant chunk IDs | None |
| nDCG@k | Ranking quality with graded relevance, discounted by position | Graded relevance labels | None |
| Context precision (LLM) | Whether relevant chunks are ranked above irrelevant ones | A reference answer | One judgment per chunk |
| Context recall (LLM) | Share of the reference answer's claims the retrieved context supports | A reference answer | Claim extraction and checks |
| Faithfulness, groundedness | Share of the answer's claims the retrieved context supports | The retrieved context | Claim extraction and checks |
| Answer relevancy | Whether the answer addresses the question | Only the question and answer | One or more calls, sometimes embeddings |
| Answer correctness | Whether the answer matches a reference | A reference answer | Claim-level comparison |
| Noise sensitivity | How often irrelevant retrieved text makes the answer wrong | A reference answer | Claim extraction and checks |

### Retrieval metrics without a model

If you can label which chunks answer each question, the classic information-retrieval metrics are
free, deterministic and fast. The definitions below follow the
[ranx documentation](https://amenra.github.io/ranx/metrics/):

- **Recall@k** is the relevant chunks in the top k divided by all relevant chunks for the query.
  This is the metric to watch first, because a chunk the retriever never returned is one the
  generator cannot use.
- **Precision@k** is the relevant chunks in the top k divided by k. It is capped by how many
  relevant chunks exist: with one relevant chunk and k = 5, the best possible score is 0.2.
- **Hit rate** is the share of queries with at least one relevant chunk in the top k. It is the
  coarsest of these metrics and often the first one reported.
- **MRR** averages 1 divided by the rank of the first relevant chunk, so it rewards putting one
  good chunk at the top and ignores the rest.
- **nDCG@k** sums each retrieved chunk's relevance grade divided by log2 of its position plus one,
  then divides by the best possible ordering's score. It is the only one of these that uses graded
  labels ("answers the question" vs "helps").

BEIR ([Thakur et al., NeurIPS 2021](https://arxiv.org/abs/2104.08663)) reports nDCG@10 on every
dataset because precision and recall ignore rank and MRR cannot use graded labels, and MTEB's
retrieval tasks use the same main metric. If you report one retrieval number, report nDCG@10, and
keep recall@k beside it for the question "did the chunk come back at all".

One caution on labels: eRAG ([Salemi and Zamani, SIGIR 2024](https://arxiv.org/abs/2404.13781))
found that query-document relevance labels have only a small correlation with how well the RAG
system answers downstream. A chunk a person marks relevant is not always a chunk the model can use.
Its fix is to run the generator on each retrieved document alone and use the result as the label,
which costs a model call per document.

### LLM-judged context metrics

Without labeled chunks, a judge model decides which retrieved chunks are relevant. Ragas
[Context Precision](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/)
judges each chunk against a reference answer and computes a rank-weighted precision, the mean of
precision@k over the positions that hold a relevant chunk. Ragas
[Context Recall](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_recall/)
splits the reference answer into claims and reports the share the retrieved context supports. Both
need a reference answer but not chunk labels, which makes them cheaper to set up and more
expensive to run.

Ragas also has `NonLLMContextRecall` and `NonLLMContextPrecisionWithReference`, which compare
retrieved chunks to reference chunks by string similarity, and `IDBasedContextPrecision` and
`IDBasedContextRecall`, which compare chunk IDs. Those are the labeled metrics above under other
names, and they make no model call.

### Generation metrics

**Faithfulness** is the generation metric that matters most: the share of the answer's claims the
retrieved context supports. It needs no reference answer, so it also works on production traffic.
The [hallucination detection](/blog/llm-hallucination-detection) post covers the cheaper ways to
compute it, including small entailment models such as HHEM and MiniCheck that do the per-claim check
without a large model.

**Answer relevancy** checks that the answer addresses the question, not that it is right. An
answer can be relevant and wrong, or correct and padded with material nobody asked for.

**Answer correctness** compares the answer to a reference. Ragas `FactualCorrectness` splits both
into claims and reports precision, recall or F1 over them. It needs reference answers, which is
the expensive part of a test set.

**Noise sensitivity** (Ragas) is the share of the answer's claims that are incorrect, so lower is
better, and a mode restricts it to errors caused by irrelevant retrieved chunks. It measures what
happens when you raise top-k.

## Same metric name, different metric

The libraries reuse each other's names, and the definitions behind them differ in ways that move
scores. Read the definition before comparing a number across tools:

| Name | Ragas | DeepEval | Promptfoo |
| --- | --- | --- | --- |
| Faithfulness | Claims **supported** by the context / all claims | Claims that do **not contradict** the context / all claims, so an unsupported claim the context never mentions passes | `context-faithfulness`: supported claims / all claims |
| Answer relevancy | Generates questions from the answer (3 by default) and averages their embedding similarity to the real question; the docs say the score is not guaranteed to fall between 0 and 1 | Share of the answer's statements an LLM classifies as relevant to the input; no embeddings | `answer-relevance`: generates questions from the output, compares them by embedding |
| Context recall | Reference claims the context supports / all reference claims | Statements in `expected_output` attributable to the context / all statements | `context-recall`: attributable statements / all statements |
| Context relevance | Not in the RAG list (the Nvidia metric `ContextRelevance` rates 0, 1 or 2 twice and averages) | `ContextualRelevancyMetric`: relevant statements in the context / all statements | `context-relevance`: required sentences / all sentences |

The faithfulness difference is the one that bites. On a support bot that adds a plausible phone
number the knowledge base never mentions, Ragas Faithfulness counts that claim against the answer
and DeepEval's FaithfulnessMetric does not. Neither is wrong; they answer different questions.
Decide which question you are asking before you pick the metric.

## Can you trust an LLM judge for RAG metrics

Partly, and it depends on the metric. The research gives these numbers:

- The [RAGAS paper](https://aclanthology.org/2024.eacl-demo.16.pdf) (Es et al., EACL 2024 demos)
  measured agreement with human pairwise judgments on its WikiEval set: 0.95 for faithfulness,
  0.78 for answer relevance and 0.70 for context relevance, which it calls "the hardest quality
  dimension to evaluate". The two human annotators agreed about 95% of the time on faithfulness and
  context relevance and 90% on answer relevance.
- [ARES](https://aclanthology.org/2024.naacl-long.20.pdf) (Saad-Falcon et al., NAACL 2024)
  fine-tunes small judges on synthetic data and uses prediction-powered inference with about 150 or
  more human-labeled examples to put confidence intervals on the scores. It reports evaluation
  accuracy 59.3 points above RAGAS on context relevance and 14.4 points above on answer relevance.
- [RAGChecker](https://arxiv.org/abs/2408.08067) (Ru et al., 2024) checks answers claim by claim
  and correlated with human ratings at 0.50 to 0.62 (Pearson) across correctness, completeness and
  overall quality. Two humans correlated at 0.64 to 0.72, so the authors note "still a clear gap".
- [RAGBench](https://arxiv.org/abs/2407.11005) (Friel et al., 2024) found that LLM-based RAG
  evaluation methods struggled to compete with a fine-tuned RoBERTa model.

The pattern holds across these papers: faithfulness judges are reliable, context relevance judges
are not, and small fine-tuned classifiers compete with large judges. Before a judged metric gates
a release, label 100 to 200 of your own cases by hand and measure the judge against them, as the
[LLM-as-a-judge](/blog/llm-as-a-judge) post describes. For context relevance, prefer labeled
recall@k and nDCG where you can get labels.

## How to build a RAG test set

A RAG test case needs a question and, depending on the metrics you want, some of: the relevant
documents, a reference answer, and the context your pipeline retrieved for it. Five things make a
test set useful:

1. **Start from real questions.** Pull them from logs, support tickets and search queries. They
   have the vocabulary, typos and vagueness your users actually bring.
2. **Label relevance at the document level, with the answer's location.** Chunk IDs change every
   time you re-chunk, which silently invalidates chunk-level labels. Store the source document and
   the quoted passage that answers the question, and map them to whatever chunks exist at
   evaluation time.
3. **Include questions the corpus cannot answer.** A share of the suite should have "I don't know"
   or a clarifying question as the correct output. Without them you cannot see a model that answers
   everything.
4. **Include the queries retrievers find hard.** Exact identifiers (error codes, SKUs, version
   numbers), questions that need two documents, and questions phrased nothing like the document
   that answers them.
5. **Use synthetic questions to fill gaps, not to replace real ones.** Ragas
   [`TestsetGenerator`](https://docs.ragas.io/en/stable/getstarted/rag_testset_generation/) builds a
   knowledge graph from your documents and by default generates 50% single-hop and 50% multi-hop
   questions. DeepEval's `Synthesizer` (`generate_goldens_from_docs`) and LlamaIndex's
   `generate_question_context_pairs` do similar jobs. A question written by reading a chunk tends to
   reuse that chunk's wording, which flatters retrieval, so read a sample and keep real questions
   in the mix.

How many cases you need depends on how small a difference you want to detect. The
[eval case sizing](/blog/how-many-eval-cases-do-you-need) post has the arithmetic; the example below
shows what 10 queries can and cannot tell you.

## Comparing two retrievers query by query

The most common RAG change is a retrieval change: a new embedding model, a different chunk size, a
reranker. Averages hide what such a change does, because it usually helps some queries and breaks
others. This script scores two retrieval runs against the same labels, prints the averages, then
compares recall@k query by query with a bootstrap confidence interval and lists the queries the
candidate lost. It needs only the Python standard library:

```python
"""Score two retrieval runs against labeled relevant chunks, and compare them query by query.

Usage: python retrieval_eval.py labels.jsonl baseline.jsonl candidate.jsonl [k]
  labels.jsonl:  {"id": "q1", "relevant": {"doc-12": 2, "doc-40": 1}}   grade 2 = answers it, 1 = helps
  run files:     {"id": "q1", "retrieved": ["doc-40", "doc-7", "doc-12", ...]}   ranked, best first
"""
import json
import math
import random
import sys


def load(path):
    return {row["id"]: row for row in map(json.loads, open(path))}


def score(retrieved, relevant, k):
    top = retrieved[:k]
    hits = [doc for doc in top if doc in relevant]
    first = next((i for i, doc in enumerate(retrieved) if doc in relevant), None)
    dcg = sum(relevant.get(doc, 0) / math.log2(i + 2) for i, doc in enumerate(top))
    ideal = sorted(relevant.values(), reverse=True)[:k]
    idcg = sum(grade / math.log2(i + 2) for i, grade in enumerate(ideal))
    return {
        f"hit@{k}": 1.0 if hits else 0.0,
        f"recall@{k}": len(hits) / len(relevant),
        f"precision@{k}": len(hits) / k,
        "mrr": 0.0 if first is None else 1 / (first + 1),
        f"ndcg@{k}": dcg / idcg,
    }


def run_scores(labels, run, k):
    return {qid: score(run[qid]["retrieved"], labels[qid]["relevant"], k) for qid in labels}


def bootstrap_ci(diffs, rounds=10_000, seed=0):
    rng = random.Random(seed)
    means = sorted(
        sum(rng.choice(diffs) for _ in diffs) / len(diffs) for _ in range(rounds)
    )
    return means[int(0.025 * rounds)], means[int(0.975 * rounds) - 1]


if __name__ == "__main__":
    labels, base, cand = (load(p) for p in sys.argv[1:4])
    k = int(sys.argv[4]) if len(sys.argv) > 4 else 5
    a, b = run_scores(labels, base, k), run_scores(labels, cand, k)
    print(f"{'metric':<14}{'baseline':>10}{'candidate':>11}")
    for metric in a[next(iter(a))]:
        mean_a = sum(s[metric] for s in a.values()) / len(a)
        mean_b = sum(s[metric] for s in b.values()) / len(b)
        print(f"{metric:<14}{mean_a:>10.3f}{mean_b:>11.3f}")
    key = f"recall@{k}"
    diffs = [b[q][key] - a[q][key] for q in labels]
    low, high = bootstrap_ci(diffs)
    better = sum(d > 0 for d in diffs)
    worse = [q for q in labels if b[q][key] < a[q][key]]
    print(f"\n{key}: candidate better on {better}, worse on {len(worse)} of {len(diffs)} queries")
    print(f"mean change {sum(diffs) / len(diffs):+.3f}, 95% bootstrap CI [{low:+.3f}, {high:+.3f}]")
    for q in worse:
        print(f"  lost: {q}")
    sys.exit(1 if high < 0 else 0)
```

I ran it on a small fixture set: 10 support questions with graded labels, and two made-up runs
standing in for the current retriever and a candidate embedding model. These are fixtures written
to show the output, not measurements of real models:

```text
metric          baseline  candidate
hit@5              0.900      0.800
recall@5           0.767      0.800
precision@5        0.260      0.280
mrr                0.783      0.800
ndcg@5             0.672      0.800

recall@5: candidate better on 4, worse on 2 of 10 queries
mean change +0.033, 95% bootstrap CI [-0.350, +0.383]
  lost: err-E1042
  lost: sku-TX-900
```

Read it from the bottom. The candidate improves nDCG@5 from 0.672 to 0.800 and recall@5 on four
queries, and it loses both questions that hinge on an exact identifier, an error code and a SKU.
Four of the five averages say ship it. Only hit rate drops, from 0.900 to 0.800, because those two
queries now retrieve nothing relevant while one that retrieved nothing before now does. Check
exact-identifier queries separately whenever a change reduces the weight of keyword matching,
because an average lets a few wins hide them. The interval runs from
-0.35 to +0.38, so 10 queries cannot tell you whether recall improved. The script exits 1 only
when the whole interval is below zero, which makes it safe to run in CI on a suite of a few
hundred queries, and the `lost:` lines are the cases to read either way.

## RAG evaluation frameworks compared

Versions and dates as of November 17, 2026:

| Framework | Latest release | RAG metrics | Notes |
| --- | --- | --- | --- |
| [Ragas](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/) | 0.4.4, Nov 14, 2026 | Context Precision, Context Recall, Context Entities Recall, Noise Sensitivity, Answer Relevancy, Faithfulness, Factual Correctness; ID-based and non-LLM variants | Test set generator; usage analytics on by default (`RAGAS_DO_NOT_TRACK=true`) |
| [DeepEval](https://deepeval.com/docs/metrics-faithfulness) | 4.3.4, Nov 11, 2026 | `AnswerRelevancyMetric`, `FaithfulnessMetric`, `ContextualPrecisionMetric`, `ContextualRecallMetric`, `ContextualRelevancyMetric` | Runs in pytest; every metric returns a reason; `Synthesizer` for test data |
| [TruLens](https://www.trulens.org/getting_started/core_concepts/rag_triad/) | 2.16.0, Oct 19, 2026 | The RAG triad: context relevance, groundedness, answer relevance | Published by Snowflake; OpenTelemetry tracing |
| [MLflow](https://mlflow.org/docs/latest/genai/eval-monitor/scorers/llm-judge/rag/groundedness) | 3.18.0, Nov 3, 2026 | `RetrievalRelevance`, `RetrievalGroundedness`, `RetrievalSufficiency`, plus `Correctness` and `RelevanceToQuery` | The retrieval judges need a traced span of type `RETRIEVER`; Databricks uses the same judges |
| [Arize Phoenix](https://arize.com/docs/phoenix/cookbook/evaluation/evaluate-rag) | phoenix-evals 3.10.0, Nov 10, 2026 | `RetrievalRelevanceEvaluator`, `FaithfulnessEvaluator`, `CorrectnessEvaluator` | Its RAG cookbook computes nDCG, precision and hit rate from the relevance labels with pandas |
| [Promptfoo](https://www.promptfoo.dev/docs/configuration/expected-outputs/model-graded/) | 0.125.1, Oct 25, 2026 | `context-faithfulness`, `context-recall`, `context-relevance`, `answer-relevance` | YAML assertions, default threshold 0.5 |
| [Braintrust autoevals](https://github.com/braintrustdata/autoevals) | 0.4.1, Oct 21, 2026 | `ContextPrecision`, `ContextRecall`, `ContextRelevancy`, `ContextEntityRecall`, `Faithfulness`, `AnswerRelevancy`, `AnswerCorrectness` | Its source says they are "ported from the RAGAS project with some enhancements" |
| [LlamaIndex](https://developers.llamaindex.ai/python/framework/module_guides/evaluating/) | core 0.14.26, Nov 13, 2026 | `RetrieverEvaluator` (hit rate, MRR) against expected IDs; `FaithfulnessEvaluator`, `RelevancyEvaluator` | `generate_question_context_pairs` builds synthetic retrieval data |

A note on Ragas, since "ragas evaluation" is one of the most searched terms in this area: after
0.4.3 in January 2026 the repository went quiet from late February until November 13, 2026, and
0.4.4 followed on November 14. That release fixes `import ragas` failing against recent
langchain-community versions and a security issue (CVE-2026-6587) in its multimodal metrics, so
upgrade if you pinned 0.4.3. The metric API is also moving: `ragas.metrics.collections` is the
current one, and the docs say the older `ragas.metrics` classes will be removed in 1.0.

Which to pick: Ragas or DeepEval if you want the standard metrics as a Python library, DeepEval if
your tests already live in pytest, MLflow or Phoenix if you already trace with them, Promptfoo if
your evals are YAML, and plain code like the script above for labeled retrieval metrics, which need
no framework. The [open-source evaluation frameworks](/blog/open-source-llm-evaluation-frameworks)
post compares these tools beyond RAG.

## RAG benchmarks, and why they do not replace your test set

Public benchmarks are useful for choosing a component, and weak at predicting how it will do on
your corpus:

| Benchmark | What it contains | Use it for |
| --- | --- | --- |
| [BEIR](https://arxiv.org/abs/2104.08663) (NeurIPS 2021) | 18 retrieval datasets, zero-shot | Comparing retrievers out of domain; BM25 is a strong baseline |
| [MTEB](https://huggingface.co/spaces/mteb/leaderboard), [MMTEB](https://arxiv.org/abs/2502.13595) | Embedding tasks including retrieval; MMTEB has 500+ tasks in 250+ languages | Shortlisting embedding models |
| [BRIGHT](https://arxiv.org/abs/2407.12883) (ICLR 2025) | 1,384 real queries whose relevant documents take reasoning to find | Retrieval beyond keyword and topic match |
| [CRAG](https://arxiv.org/abs/2406.04744) (NeurIPS 2024) | 4,409 questions across five domains with mock web and knowledge-graph search | End-to-end RAG with a penalty for wrong answers |
| [RGB](https://arxiv.org/abs/2309.01431) (AAAI 2024) | English and Chinese questions testing noise robustness, negative rejection, integration and counterfactuals | How the generator handles bad context |
| [MultiHop-RAG](https://arxiv.org/abs/2401.15391) (COLM 2024) | 2,556 queries over news articles, some with no answer | Questions that need several documents |
| [FRAMES](https://arxiv.org/abs/2409.12941) (NAACL 2025) | 824 questions, each needing 2 to 15 Wikipedia articles | Multi-step retrieval |
| [RAGTruth](https://aclanthology.org/2024.acl-long.585.pdf) (ACL 2024) | Nearly 18,000 RAG responses labeled for hallucination at word level | Evaluating faithfulness detectors |

Three numbers from these papers make the case for testing on your own data:

- On BRIGHT, SFR-Embedding-Mistral scores nDCG@10 of 59.0 on MTEB and 18.3 on BRIGHT. A
  leaderboard position measures the leaderboard's queries.
- In CRAG, the most advanced LLMs reached at most 34% accuracy alone and 44% with straightforward
  RAG, and state-of-the-art industry RAG systems answered only 63% of questions without
  hallucinating.
- In MultiHop-RAG, GPT-4 scored 0.56 with the chunks its retriever found and 0.89 with the
  ground-truth evidence. The gap between those two numbers is the retriever's share of the error.
  FRAMES shows the same from the other side: accuracy went from 0.408 with no retrieval to 0.66
  with multi-step retrieval.

## How to test a change to a RAG pipeline

Most RAG regressions arrive with a change to one component. Hold the others still and measure the
one that moved:

| What changed | Hold constant | Measure |
| --- | --- | --- |
| Chunking, index rebuild | The labels (document plus quoted passage) | Recall@k, nDCG@10, query by query |
| Embedding model, hybrid weights | The chunks | Recall@k, nDCG@10, and the exact-identifier queries separately |
| Top-k, reranker | The retriever's candidates | Recall@k and faithfulness together, since more context can lower answer quality |
| Prompt | The retrieved context, recorded per case | Faithfulness, correctness, and abstention on the unanswerable cases |
| Generator model | The retrieved context, recorded per case | The same, plus cost and latency |

Holding the context constant is the step teams skip. If you rerun retrieval while testing a new
generator model, a ranking change in the vector store shows up as a model difference. Record the
retrieved chunks with each test case, feed the same chunks to both models, and score retrieval in
a separate run.

Then compare against the current pipeline, not against an absolute threshold. A faithfulness score
of 0.91 means little alone; 0.91 against 0.94 on the same cases, with an interval that excludes
zero, is a regression. Gate on the metrics with explainable failures (labeled recall, correctness
on cases with references, abstention on the unanswerable cases) and keep the judged metrics
advisory until you have measured the judge.

### When retrieval is a tool call

In an agent, retrieval is usually a tool such as `search_kb`, and the model decides whether to
search, what to search for, and how many times. That adds three things to evaluate: whether it
searched when the answer needed it, the query it wrote, and whether the answer used what came back.
BrowseComp-Plus ([Chen et al., 2025](https://arxiv.org/abs/2508.06600)) fixed the corpus for
deep-research agents to separate the two: its abstract reports GPT-5 at 55.9% accuracy, and 70.1%,
with fewer search calls, when paired with the Qwen3-Embedding-8B retriever. The
[tool-call evaluation](/blog/evaluating-agent-tool-calls) post covers grading the search calls
themselves.

## Doing this with EvalShift

Disclosure again: I'm the founder. [EvalShift](/docs) does not compute any of the RAG metrics in
this post. It has no recall@k, nDCG, context precision or faithfulness score, and it does not
compare embedding models or rerun your retriever. For those, use the script above or one of the
frameworks.

What it does is the generator row of the table above: test whether a different model or a changed
prompt behaves differently on recorded cases, with retrieval held constant.

- **Recorded context.** A case promoted from a [captured](/docs/sdk-capture) run replays the
  recorded model input, so if your pipeline put the retrieved chunks in the prompt, both models get
  the same text. When retrieval is a tool, `--rounds all` feeds the recorded tool results back to
  the candidate, so it answers from the same documents the current model saw, whatever query it
  would have written.
- **Search behavior.** With retrieval as a tool, `tool_selection` checks whether the candidate still
  searches where the current model did, `tool_arguments` compares the query it writes, and
  `trace_invariants` can require a search before an answer or cap the number of calls. The
  [evaluators docs](/docs/evaluators) cover each.
- **The verdict.** A [`migration_policy`](/docs/migration-policy) turns the differences into pass,
  conditional pass, fail or inconclusive, and inconclusive means the suite was too small to decide.

The pairwise LLM judge does not see the retrieved documents, so it cannot score groundedness. Run a
faithfulness metric from one of the frameworks above on both models' answers for that.

## FAQ

**What is RAG evaluation?** It is measuring a retrieval-augmented generation pipeline in two
halves and end to end: whether the retriever returns the chunks that answer the question, measured
with recall@k, nDCG or LLM-judged context precision and recall, and whether the generator's answer
is faithful to those chunks, relevant to the question and correct.

**What are the main RAG evaluation metrics?** For retrieval: hit rate, recall@k, precision@k, MRR
and nDCG@k against labeled relevant chunks, or context precision and context recall judged by an
LLM against a reference answer. For generation: faithfulness or groundedness, answer relevancy and
answer correctness. Noise sensitivity measures how irrelevant retrieved text affects the answer.

**What is the RAG triad?** It is TruLens's name for three reference-free RAG checks: context
relevance (are the retrieved chunks relevant to the query), groundedness (is each claim in the
answer supported by the chunks) and answer relevance (does the answer address the query).

**How do you evaluate RAG without ground truth?** Use reference-free metrics: faithfulness or
groundedness, which checks the answer against the retrieved context, LLM-judged context relevance
and answer relevancy. They need no reference answers, so they also run on production traffic. They
cannot tell you whether the answer is correct, or whether a better chunk existed that was not
retrieved.

**What is the difference between Ragas and DeepEval?** Both are open-source Python libraries with
LLM-judged RAG metrics. Ragas has more retrieval variants, including ID-based and non-LLM ones, and
a knowledge-graph test set generator. DeepEval runs in pytest and returns a reason with every
score. Their metrics with the same name differ: DeepEval's faithfulness counts an unsupported claim
as faithful unless the context contradicts it, while Ragas counts it against the answer.

**Is Ragas still maintained?** Yes. After a gap from late February 2026, its repository became
active again on November 13, 2026, and version 0.4.4 was released on November 14, 2026, with fixes
for import errors and a security issue in its multimodal metrics.

## Keep reading

- [LLM hallucination detection](/blog/llm-hallucination-detection): faithfulness checks in depth,
  including small grounding models.
- [LLM-as-a-judge](/blog/llm-as-a-judge): how to write and validate the judge behind a RAG metric.
- [How many eval cases do you need](/blog/how-many-eval-cases-do-you-need): sizing a test set to
  detect the difference you care about.
- [Open-source LLM evaluation frameworks](/blog/open-source-llm-evaluation-frameworks): Ragas,
  DeepEval and the rest compared beyond RAG.
