← all posts
rag·Lukas Babaliauskas·Nov 21, 2026·18 min read

RAG evaluation: metrics, frameworks and how to test a RAG pipeline

RAG evaluation metrics, frameworks and benchmarks compared, how far to trust an LLM judge, and how to test a retriever or model change in a RAG pipeline.

RAG evaluation measures a retrieval-augmented generation pipeline in two halves, then end to end. The retrieval half asks whether the right chunks came back: recall@k, precision@k, MRR and nDCG against labeled relevant documents, or LLM-judged context precision and context recall when you have no labels. The generation half asks whether the answer used those chunks well: faithfulness (every claim supported by the retrieved context), answer relevancy (it addresses the question) and correctness against a reference answer. Score the halves separately, because an end-to-end score of 70% does not tell you whether to fix the retriever or the prompt.

The tools that compute these metrics, Ragas, DeepEval, TruLens, MLflow, Phoenix and Promptfoo among them, agree on the names and disagree on the definitions. This post covers what each metric measures and needs, how the frameworks differ, how far to trust an LLM judge for this, how to build a test set, and how to test a change to one part of the pipeline without the rest moving. It includes a short script, tested on a small fixture set, that compares two retrievers query by query with a confidence interval, a step the top-ranking guides I read leave out.

A disclosure: I build EvalShift, a tool for testing model changes on agents. It has a short section at the end, which says what it does not do for RAG. Everything before it works without it.

Evaluate retrieval and generation separately

A wrong RAG answer has one of three causes, and each has a different fix:

What went wrongWhat you seeWhere the fix is
The relevant chunk was not retrievedLow context recall; the answer is wrong or a refusalChunking, embedding model, hybrid search, query rewriting, top-k
It was retrieved, and the model ignored or misread itHigh context recall, low faithfulness or correctnessPrompt, context order, the generator model
Nothing relevant exists in the corpusNo retriever can find it; the right answer is "I don't know"The corpus, and the model's willingness to abstain

The second row is more common than it sounds. In Lost in the Middle ↗ (Liu et al., TACL 2024), models used information at the start and end of a long context much better than information in the middle: GPT-3.5-Turbo's multi-document QA accuracy dropped by more than 20% when the answer moved to the middle, and with 20 or 30 documents and the answer in the middle it fell below its closed-book score of 56.1%. Retrieving the right chunk is necessary and not sufficient. Raising top-k improves recall and can lower answer quality at the same time.

The third row is the one test sets forget. If every question in your suite has an answer in the corpus, you never measure whether the system admits it does not know. RGB (Chen et al., AAAI 2024 ↗) calls this negative rejection, and found models struggled with it, as with integrating information across documents and with retrieved text that contradicts what they know.

RAG evaluation metrics

MetricMeasuresNeedsModel calls
Hit rate (success@k)Share of queries with at least one relevant chunk in the top kRelevant chunk IDs per queryNone
Recall@kShare of the relevant chunks that are in the top kRelevant chunk IDsNone
Precision@kShare of the top k that is relevantRelevant chunk IDsNone
MRR1 / rank of the first relevant chunk, averagedRelevant chunk IDsNone
nDCG@kRanking quality with graded relevance, discounted by positionGraded relevance labelsNone
Context precision (LLM)Whether relevant chunks are ranked above irrelevant onesA reference answerOne judgment per chunk
Context recall (LLM)Share of the reference answer's claims the retrieved context supportsA reference answerClaim extraction and checks
Faithfulness, groundednessShare of the answer's claims the retrieved context supportsThe retrieved contextClaim extraction and checks
Answer relevancyWhether the answer addresses the questionOnly the question and answerOne or more calls, sometimes embeddings
Answer correctnessWhether the answer matches a referenceA reference answerClaim-level comparison
Noise sensitivityHow often irrelevant retrieved text makes the answer wrongA reference answerClaim extraction and checks

Retrieval metrics without a model

If you can label which chunks answer each question, the classic information-retrieval metrics are free, deterministic and fast. The definitions below follow the ranx documentation ↗:

  • +Recall@k is the relevant chunks in the top k divided by all relevant chunks for the query. This is the metric to watch first, because a chunk the retriever never returned is one the generator cannot use.
  • +Precision@k is the relevant chunks in the top k divided by k. It is capped by how many relevant chunks exist: with one relevant chunk and k = 5, the best possible score is 0.2.
  • +Hit rate is the share of queries with at least one relevant chunk in the top k. It is the coarsest of these metrics and often the first one reported.
  • +MRR averages 1 divided by the rank of the first relevant chunk, so it rewards putting one good chunk at the top and ignores the rest.
  • +nDCG@k sums each retrieved chunk's relevance grade divided by log2 of its position plus one, then divides by the best possible ordering's score. It is the only one of these that uses graded labels ("answers the question" vs "helps").

BEIR (Thakur et al., NeurIPS 2021 ↗) reports nDCG@10 on every dataset because precision and recall ignore rank and MRR cannot use graded labels, and MTEB's retrieval tasks use the same main metric. If you report one retrieval number, report nDCG@10, and keep recall@k beside it for the question "did the chunk come back at all".

One caution on labels: eRAG (Salemi and Zamani, SIGIR 2024 ↗) found that query-document relevance labels have only a small correlation with how well the RAG system answers downstream. A chunk a person marks relevant is not always a chunk the model can use. Its fix is to run the generator on each retrieved document alone and use the result as the label, which costs a model call per document.

LLM-judged context metrics

Without labeled chunks, a judge model decides which retrieved chunks are relevant. Ragas Context Precision ↗ judges each chunk against a reference answer and computes a rank-weighted precision, the mean of precision@k over the positions that hold a relevant chunk. Ragas Context Recall ↗ splits the reference answer into claims and reports the share the retrieved context supports. Both need a reference answer but not chunk labels, which makes them cheaper to set up and more expensive to run.

Ragas also has NonLLMContextRecall and NonLLMContextPrecisionWithReference, which compare retrieved chunks to reference chunks by string similarity, and IDBasedContextPrecision and IDBasedContextRecall, which compare chunk IDs. Those are the labeled metrics above under other names, and they make no model call.

Generation metrics

Faithfulness is the generation metric that matters most: the share of the answer's claims the retrieved context supports. It needs no reference answer, so it also works on production traffic. The hallucination detection post covers the cheaper ways to compute it, including small entailment models such as HHEM and MiniCheck that do the per-claim check without a large model.

Answer relevancy checks that the answer addresses the question, not that it is right. An answer can be relevant and wrong, or correct and padded with material nobody asked for.

Answer correctness compares the answer to a reference. Ragas FactualCorrectness splits both into claims and reports precision, recall or F1 over them. It needs reference answers, which is the expensive part of a test set.

Noise sensitivity (Ragas) is the share of the answer's claims that are incorrect, so lower is better, and a mode restricts it to errors caused by irrelevant retrieved chunks. It measures what happens when you raise top-k.

Same metric name, different metric

The libraries reuse each other's names, and the definitions behind them differ in ways that move scores. Read the definition before comparing a number across tools:

NameRagasDeepEvalPromptfoo
FaithfulnessClaims supported by the context / all claimsClaims that do not contradict the context / all claims, so an unsupported claim the context never mentions passescontext-faithfulness: supported claims / all claims
Answer relevancyGenerates questions from the answer (3 by default) and averages their embedding similarity to the real question; the docs say the score is not guaranteed to fall between 0 and 1Share of the answer's statements an LLM classifies as relevant to the input; no embeddingsanswer-relevance: generates questions from the output, compares them by embedding
Context recallReference claims the context supports / all reference claimsStatements in expected_output attributable to the context / all statementscontext-recall: attributable statements / all statements
Context relevanceNot in the RAG list (the Nvidia metric ContextRelevance rates 0, 1 or 2 twice and averages)ContextualRelevancyMetric: relevant statements in the context / all statementscontext-relevance: required sentences / all sentences

The faithfulness difference is the one that bites. On a support bot that adds a plausible phone number the knowledge base never mentions, Ragas Faithfulness counts that claim against the answer and DeepEval's FaithfulnessMetric does not. Neither is wrong; they answer different questions. Decide which question you are asking before you pick the metric.

Can you trust an LLM judge for RAG metrics

Partly, and it depends on the metric. The research gives these numbers:

  • +The RAGAS paper ↗ (Es et al., EACL 2024 demos) measured agreement with human pairwise judgments on its WikiEval set: 0.95 for faithfulness, 0.78 for answer relevance and 0.70 for context relevance, which it calls "the hardest quality dimension to evaluate". The two human annotators agreed about 95% of the time on faithfulness and context relevance and 90% on answer relevance.
  • +ARES ↗ (Saad-Falcon et al., NAACL 2024) fine-tunes small judges on synthetic data and uses prediction-powered inference with about 150 or more human-labeled examples to put confidence intervals on the scores. It reports evaluation accuracy 59.3 points above RAGAS on context relevance and 14.4 points above on answer relevance.
  • +RAGChecker ↗ (Ru et al., 2024) checks answers claim by claim and correlated with human ratings at 0.50 to 0.62 (Pearson) across correctness, completeness and overall quality. Two humans correlated at 0.64 to 0.72, so the authors note "still a clear gap".
  • +RAGBench ↗ (Friel et al., 2024) found that LLM-based RAG evaluation methods struggled to compete with a fine-tuned RoBERTa model.

The pattern holds across these papers: faithfulness judges are reliable, context relevance judges are not, and small fine-tuned classifiers compete with large judges. Before a judged metric gates a release, label 100 to 200 of your own cases by hand and measure the judge against them, as the LLM-as-a-judge post describes. For context relevance, prefer labeled recall@k and nDCG where you can get labels.

How to build a RAG test set

A RAG test case needs a question and, depending on the metrics you want, some of: the relevant documents, a reference answer, and the context your pipeline retrieved for it. Five things make a test set useful:

  1. +Start from real questions. Pull them from logs, support tickets and search queries. They have the vocabulary, typos and vagueness your users actually bring.
  2. +Label relevance at the document level, with the answer's location. Chunk IDs change every time you re-chunk, which silently invalidates chunk-level labels. Store the source document and the quoted passage that answers the question, and map them to whatever chunks exist at evaluation time.
  3. +Include questions the corpus cannot answer. A share of the suite should have "I don't know" or a clarifying question as the correct output. Without them you cannot see a model that answers everything.
  4. +Include the queries retrievers find hard. Exact identifiers (error codes, SKUs, version numbers), questions that need two documents, and questions phrased nothing like the document that answers them.
  5. +Use synthetic questions to fill gaps, not to replace real ones. Ragas TestsetGenerator ↗ builds a knowledge graph from your documents and by default generates 50% single-hop and 50% multi-hop questions. DeepEval's Synthesizer (generate_goldens_from_docs) and LlamaIndex's generate_question_context_pairs do similar jobs. A question written by reading a chunk tends to reuse that chunk's wording, which flatters retrieval, so read a sample and keep real questions in the mix.

How many cases you need depends on how small a difference you want to detect. The eval case sizing post has the arithmetic; the example below shows what 10 queries can and cannot tell you.

Comparing two retrievers query by query

The most common RAG change is a retrieval change: a new embedding model, a different chunk size, a reranker. Averages hide what such a change does, because it usually helps some queries and breaks others. This script scores two retrieval runs against the same labels, prints the averages, then compares recall@k query by query with a bootstrap confidence interval and lists the queries the candidate lost. It needs only the Python standard library:

python
"""Score two retrieval runs against labeled relevant chunks, and compare them query by query.

Usage: python retrieval_eval.py labels.jsonl baseline.jsonl candidate.jsonl [k]
  labels.jsonl:  {"id": "q1", "relevant": {"doc-12": 2, "doc-40": 1}}   grade 2 = answers it, 1 = helps
  run files:     {"id": "q1", "retrieved": ["doc-40", "doc-7", "doc-12", ...]}   ranked, best first
"""
import json
import math
import random
import sys


def load(path):
    return {row["id"]: row for row in map(json.loads, open(path))}


def score(retrieved, relevant, k):
    top = retrieved[:k]
    hits = [doc for doc in top if doc in relevant]
    first = next((i for i, doc in enumerate(retrieved) if doc in relevant), None)
    dcg = sum(relevant.get(doc, 0) / math.log2(i + 2) for i, doc in enumerate(top))
    ideal = sorted(relevant.values(), reverse=True)[:k]
    idcg = sum(grade / math.log2(i + 2) for i, grade in enumerate(ideal))
    return {
        f"hit@{k}": 1.0 if hits else 0.0,
        f"recall@{k}": len(hits) / len(relevant),
        f"precision@{k}": len(hits) / k,
        "mrr": 0.0 if first is None else 1 / (first + 1),
        f"ndcg@{k}": dcg / idcg,
    }


def run_scores(labels, run, k):
    return {qid: score(run[qid]["retrieved"], labels[qid]["relevant"], k) for qid in labels}


def bootstrap_ci(diffs, rounds=10_000, seed=0):
    rng = random.Random(seed)
    means = sorted(
        sum(rng.choice(diffs) for _ in diffs) / len(diffs) for _ in range(rounds)
    )
    return means[int(0.025 * rounds)], means[int(0.975 * rounds) - 1]


if __name__ == "__main__":
    labels, base, cand = (load(p) for p in sys.argv[1:4])
    k = int(sys.argv[4]) if len(sys.argv) > 4 else 5
    a, b = run_scores(labels, base, k), run_scores(labels, cand, k)
    print(f"{'metric':<14}{'baseline':>10}{'candidate':>11}")
    for metric in a[next(iter(a))]:
        mean_a = sum(s[metric] for s in a.values()) / len(a)
        mean_b = sum(s[metric] for s in b.values()) / len(b)
        print(f"{metric:<14}{mean_a:>10.3f}{mean_b:>11.3f}")
    key = f"recall@{k}"
    diffs = [b[q][key] - a[q][key] for q in labels]
    low, high = bootstrap_ci(diffs)
    better = sum(d > 0 for d in diffs)
    worse = [q for q in labels if b[q][key] < a[q][key]]
    print(f"\n{key}: candidate better on {better}, worse on {len(worse)} of {len(diffs)} queries")
    print(f"mean change {sum(diffs) / len(diffs):+.3f}, 95% bootstrap CI [{low:+.3f}, {high:+.3f}]")
    for q in worse:
        print(f"  lost: {q}")
    sys.exit(1 if high < 0 else 0)

I ran it on a small fixture set: 10 support questions with graded labels, and two made-up runs standing in for the current retriever and a candidate embedding model. These are fixtures written to show the output, not measurements of real models:

text
metric          baseline  candidate
hit@5              0.900      0.800
recall@5           0.767      0.800
precision@5        0.260      0.280
mrr                0.783      0.800
ndcg@5             0.672      0.800

recall@5: candidate better on 4, worse on 2 of 10 queries
mean change +0.033, 95% bootstrap CI [-0.350, +0.383]
  lost: err-E1042
  lost: sku-TX-900

Read it from the bottom. The candidate improves nDCG@5 from 0.672 to 0.800 and recall@5 on four queries, and it loses both questions that hinge on an exact identifier, an error code and a SKU. Four of the five averages say ship it. Only hit rate drops, from 0.900 to 0.800, because those two queries now retrieve nothing relevant while one that retrieved nothing before now does. Check exact-identifier queries separately whenever a change reduces the weight of keyword matching, because an average lets a few wins hide them. The interval runs from -0.35 to +0.38, so 10 queries cannot tell you whether recall improved. The script exits 1 only when the whole interval is below zero, which makes it safe to run in CI on a suite of a few hundred queries, and the lost: lines are the cases to read either way.

RAG evaluation frameworks compared

Versions and dates as of November 17, 2026:

FrameworkLatest releaseRAG metricsNotes
Ragas ↗0.4.4, Nov 14, 2026Context Precision, Context Recall, Context Entities Recall, Noise Sensitivity, Answer Relevancy, Faithfulness, Factual Correctness; ID-based and non-LLM variantsTest set generator; usage analytics on by default (RAGAS_DO_NOT_TRACK=true)
DeepEval ↗4.3.4, Nov 11, 2026AnswerRelevancyMetric, FaithfulnessMetric, ContextualPrecisionMetric, ContextualRecallMetric, ContextualRelevancyMetricRuns in pytest; every metric returns a reason; Synthesizer for test data
TruLens ↗2.16.0, Oct 19, 2026The RAG triad: context relevance, groundedness, answer relevancePublished by Snowflake; OpenTelemetry tracing
MLflow ↗3.18.0, Nov 3, 2026RetrievalRelevance, RetrievalGroundedness, RetrievalSufficiency, plus Correctness and RelevanceToQueryThe retrieval judges need a traced span of type RETRIEVER; Databricks uses the same judges
Arize Phoenix ↗phoenix-evals 3.10.0, Nov 10, 2026RetrievalRelevanceEvaluator, FaithfulnessEvaluator, CorrectnessEvaluatorIts RAG cookbook computes nDCG, precision and hit rate from the relevance labels with pandas
Promptfoo ↗0.125.1, Oct 25, 2026context-faithfulness, context-recall, context-relevance, answer-relevanceYAML assertions, default threshold 0.5
Braintrust autoevals ↗0.4.1, Oct 21, 2026ContextPrecision, ContextRecall, ContextRelevancy, ContextEntityRecall, Faithfulness, AnswerRelevancy, AnswerCorrectnessIts source says they are "ported from the RAGAS project with some enhancements"
LlamaIndex ↗core 0.14.26, Nov 13, 2026RetrieverEvaluator (hit rate, MRR) against expected IDs; FaithfulnessEvaluator, RelevancyEvaluatorgenerate_question_context_pairs builds synthetic retrieval data

A note on Ragas, since "ragas evaluation" is one of the most searched terms in this area: after 0.4.3 in January 2026 the repository went quiet from late February until November 13, 2026, and 0.4.4 followed on November 14. That release fixes import ragas failing against recent langchain-community versions and a security issue (CVE-2026-6587) in its multimodal metrics, so upgrade if you pinned 0.4.3. The metric API is also moving: ragas.metrics.collections is the current one, and the docs say the older ragas.metrics classes will be removed in 1.0.

Which to pick: Ragas or DeepEval if you want the standard metrics as a Python library, DeepEval if your tests already live in pytest, MLflow or Phoenix if you already trace with them, Promptfoo if your evals are YAML, and plain code like the script above for labeled retrieval metrics, which need no framework. The open-source evaluation frameworks post compares these tools beyond RAG.

RAG benchmarks, and why they do not replace your test set

Public benchmarks are useful for choosing a component, and weak at predicting how it will do on your corpus:

BenchmarkWhat it containsUse it for
BEIR ↗ (NeurIPS 2021)18 retrieval datasets, zero-shotComparing retrievers out of domain; BM25 is a strong baseline
MTEB ↗, MMTEB ↗Embedding tasks including retrieval; MMTEB has 500+ tasks in 250+ languagesShortlisting embedding models
BRIGHT ↗ (ICLR 2025)1,384 real queries whose relevant documents take reasoning to findRetrieval beyond keyword and topic match
CRAG ↗ (NeurIPS 2024)4,409 questions across five domains with mock web and knowledge-graph searchEnd-to-end RAG with a penalty for wrong answers
RGB ↗ (AAAI 2024)English and Chinese questions testing noise robustness, negative rejection, integration and counterfactualsHow the generator handles bad context
MultiHop-RAG ↗ (COLM 2024)2,556 queries over news articles, some with no answerQuestions that need several documents
FRAMES ↗ (NAACL 2025)824 questions, each needing 2 to 15 Wikipedia articlesMulti-step retrieval
RAGTruth ↗ (ACL 2024)Nearly 18,000 RAG responses labeled for hallucination at word levelEvaluating faithfulness detectors

Three numbers from these papers make the case for testing on your own data:

  • +On BRIGHT, SFR-Embedding-Mistral scores nDCG@10 of 59.0 on MTEB and 18.3 on BRIGHT. A leaderboard position measures the leaderboard's queries.
  • +In CRAG, the most advanced LLMs reached at most 34% accuracy alone and 44% with straightforward RAG, and state-of-the-art industry RAG systems answered only 63% of questions without hallucinating.
  • +In MultiHop-RAG, GPT-4 scored 0.56 with the chunks its retriever found and 0.89 with the ground-truth evidence. The gap between those two numbers is the retriever's share of the error. FRAMES shows the same from the other side: accuracy went from 0.408 with no retrieval to 0.66 with multi-step retrieval.

How to test a change to a RAG pipeline

Most RAG regressions arrive with a change to one component. Hold the others still and measure the one that moved:

What changedHold constantMeasure
Chunking, index rebuildThe labels (document plus quoted passage)Recall@k, nDCG@10, query by query
Embedding model, hybrid weightsThe chunksRecall@k, nDCG@10, and the exact-identifier queries separately
Top-k, rerankerThe retriever's candidatesRecall@k and faithfulness together, since more context can lower answer quality
PromptThe retrieved context, recorded per caseFaithfulness, correctness, and abstention on the unanswerable cases
Generator modelThe retrieved context, recorded per caseThe same, plus cost and latency

Holding the context constant is the step teams skip. If you rerun retrieval while testing a new generator model, a ranking change in the vector store shows up as a model difference. Record the retrieved chunks with each test case, feed the same chunks to both models, and score retrieval in a separate run.

Then compare against the current pipeline, not against an absolute threshold. A faithfulness score of 0.91 means little alone; 0.91 against 0.94 on the same cases, with an interval that excludes zero, is a regression. Gate on the metrics with explainable failures (labeled recall, correctness on cases with references, abstention on the unanswerable cases) and keep the judged metrics advisory until you have measured the judge.

When retrieval is a tool call

In an agent, retrieval is usually a tool such as search_kb, and the model decides whether to search, what to search for, and how many times. That adds three things to evaluate: whether it searched when the answer needed it, the query it wrote, and whether the answer used what came back. BrowseComp-Plus (Chen et al., 2025 ↗) fixed the corpus for deep-research agents to separate the two: its abstract reports GPT-5 at 55.9% accuracy, and 70.1%, with fewer search calls, when paired with the Qwen3-Embedding-8B retriever. The tool-call evaluation post covers grading the search calls themselves.

Doing this with EvalShift

Disclosure again: I'm the founder. EvalShift does not compute any of the RAG metrics in this post. It has no recall@k, nDCG, context precision or faithfulness score, and it does not compare embedding models or rerun your retriever. For those, use the script above or one of the frameworks.

What it does is the generator row of the table above: test whether a different model or a changed prompt behaves differently on recorded cases, with retrieval held constant.

  • +Recorded context. A case promoted from a captured run replays the recorded model input, so if your pipeline put the retrieved chunks in the prompt, both models get the same text. When retrieval is a tool, --rounds all feeds the recorded tool results back to the candidate, so it answers from the same documents the current model saw, whatever query it would have written.
  • +Search behavior. With retrieval as a tool, tool_selection checks whether the candidate still searches where the current model did, tool_arguments compares the query it writes, and trace_invariants can require a search before an answer or cap the number of calls. The evaluators docs cover each.
  • +The verdict. A migration_policy turns the differences into pass, conditional pass, fail or inconclusive, and inconclusive means the suite was too small to decide.

The pairwise LLM judge does not see the retrieved documents, so it cannot score groundedness. Run a faithfulness metric from one of the frameworks above on both models' answers for that.

FAQ

What is RAG evaluation? It is measuring a retrieval-augmented generation pipeline in two halves and end to end: whether the retriever returns the chunks that answer the question, measured with recall@k, nDCG or LLM-judged context precision and recall, and whether the generator's answer is faithful to those chunks, relevant to the question and correct.

What are the main RAG evaluation metrics? For retrieval: hit rate, recall@k, precision@k, MRR and nDCG@k against labeled relevant chunks, or context precision and context recall judged by an LLM against a reference answer. For generation: faithfulness or groundedness, answer relevancy and answer correctness. Noise sensitivity measures how irrelevant retrieved text affects the answer.

What is the RAG triad? It is TruLens's name for three reference-free RAG checks: context relevance (are the retrieved chunks relevant to the query), groundedness (is each claim in the answer supported by the chunks) and answer relevance (does the answer address the query).

How do you evaluate RAG without ground truth? Use reference-free metrics: faithfulness or groundedness, which checks the answer against the retrieved context, LLM-judged context relevance and answer relevancy. They need no reference answers, so they also run on production traffic. They cannot tell you whether the answer is correct, or whether a better chunk existed that was not retrieved.

What is the difference between Ragas and DeepEval? Both are open-source Python libraries with LLM-judged RAG metrics. Ragas has more retrieval variants, including ID-based and non-LLM ones, and a knowledge-graph test set generator. DeepEval runs in pytest and returns a reason with every score. Their metrics with the same name differ: DeepEval's faithfulness counts an unsupported claim as faithful unless the context contradicts it, while Ragas counts it against the answer.

Is Ragas still maintained? Yes. After a gap from late February 2026, its repository became active again on November 13, 2026, and version 0.4.4 was released on November 14, 2026, with fixes for import errors and a security issue in its multimodal metrics.

Keep reading