## LLM hallucination detection: methods that work, and the hallucinations agents add

URL: https://www.evalshift.dev/blog/llm-hallucination-detection
Published: 2026-10-31
Tag: evals
Summary: LLM hallucination detection methods compared: grounding checks, claim-level judges, sampling, logprobs and deterministic checks, plus agent hallucinations.

Takeaways:
- LLM hallucination detection checks a model's output for claims that are false or that its source does not support. The main methods are grounding checks against a source, claim-level verification, sampling consistency, token probabilities and deterministic checks.
- For most production systems faithfulness matters more than world knowledge: check the answer against the retrieved documents and tool results the model actually saw.
- Sampling methods such as SelfCheckGPT and semantic entropy need no source, but they miss errors a model makes consistently, and they cost several generations per answer.
- Agents hallucinate where the reply never shows it: invented tools, invented arguments, and claims of actions with no matching tool call. Detect these from the trace and the tool definitions, without a model.
- A hallucination detector is a classifier. Label 100 to 200 of your own outputs and report its precision and recall with intervals before letting it block a release.

LLM hallucination detection is checking a model's output for claims that are false or that its
source does not support. There are four ways to do it. You can check the answer against a source
(the retrieved documents, a tool result, a knowledge base) with an entailment classifier or an LLM
judge. You can split the answer into individual claims and verify each. You can sample the same
question several times and treat disagreement between the samples as a warning. And for
structured output and agents, you can check deterministically: does the tool exist, does the
argument match the schema, did the action the reply describes actually run?

No single method covers everything. Source checks need a source. Sampling catches a model that is
guessing, but not one that is confidently wrong. Judges are expensive and have their own error
rate. Agents add hallucinations that never appear in the reply at all, such as an invented order
ID or a refund the model says it issued but never called a tool for. This post covers what each
method catches, what it costs, how to measure whether your detector works, and a script that
finds agent hallucinations in recorded traces.

A disclosure: I build EvalShift, a tool for testing model changes on agents. It has a short section
at the end. Everything before it works without it.

## What counts as a hallucination

The word covers several different failures, and a detector built for one is often blind to the
others. Two research taxonomies are worth knowing, because the detection literature uses their
terms.

Ji et al.'s [survey](https://arxiv.org/abs/2202.03629) (ACM Computing Surveys, 2023) splits
hallucinations by their relation to a source. An **intrinsic** hallucination contradicts the
source: the document says the refund window is 30 days and the summary says 60. An **extrinsic**
hallucination cannot be verified from the source at all: the summary adds a phone number the
document never mentioned. Extrinsic content is not necessarily wrong. It may be true, but nothing
in the source supports it.

Huang et al.'s [survey](https://arxiv.org/abs/2311.05232) (ACM TOIS, 2025) splits them by what the
output fails to match:

| Type | What the output fails to match | Example |
| --- | --- | --- |
| Factuality: contradiction | Facts about the world | "The Eiffel Tower is in Berlin" |
| Factuality: fabrication | Anything verifiable | A citation to a paper that does not exist |
| Faithfulness: instruction | The user's request | Asked to translate, the model answers the question instead |
| Faithfulness: context | The context it was given | A RAG answer that contradicts the retrieved passage |
| Faithfulness: logic | Itself | Step 3 of the reasoning contradicts step 1 |

For most production systems the faithfulness half matters more. A support bot, a RAG pipeline or
an agent has a source of truth in its context: the knowledge base, the retrieved documents, the
tool results. You can check an answer against that context. Checking an answer against "the
world" is a much harder problem, and most teams do not need to solve it.

## Why models hallucinate, in one paragraph

OpenAI researchers argued in [Why Language Models Hallucinate](https://arxiv.org/abs/2509.04664)
(Kalai et al., September 2025) that the cause is largely in how models are scored. Under
right-or-wrong grading, a guess sometimes scores and "I don't know" never does, so training and
benchmarks reward guessing. In their words, the procedures "reward guessing over acknowledging
uncertainty". OpenAI's own [SimpleQA](https://arxiv.org/abs/2411.04368) benchmark shows the
trade-off: it grades each answer correct, incorrect or not attempted, and in its paper GPT-4o got
38.2% correct and 60.8% incorrect, while Claude 3.5 Sonnet got 28.9% correct but only 36.1%
incorrect, because it declined 35.0% of questions. The practical consequence for detection: a
model that never abstains will produce confident wrong answers, and the methods that look for
uncertainty will miss some of them.

## Hallucination detection methods at a glance

| Method | What it needs | Catches | Misses | Cost per answer |
| --- | --- | --- | --- | --- |
| Grounding classifier (NLI) | The source text | Claims the source contradicts or does not support | Errors in a source that is itself wrong; claims with no source | One small-model call, milliseconds to seconds on a CPU |
| Claim-level check (LLM judge) | The source text, or a knowledge base | Each unsupported claim, with an explanation | Whatever the judge itself gets wrong | One or more large-model calls |
| Sampling consistency | Several samples of the same prompt | Answers the model is guessing at | Confident, consistent errors | 5–10 generations, plus comparisons |
| Token probabilities | Logprobs from the API | Low-confidence spans | Confident errors; unavailable on some APIs | Free, if the API returns them |
| Deterministic checks | A schema, tool definitions, a trace | Invented tools, invalid arguments, IDs from nowhere, actions that never ran | Anything in free text | Free |

The rest of this section takes them one at a time.

### 1. Check the answer against its source

If the system retrieved documents or called tools, the answer should be supported by what came
back. A grounding check takes the source as the premise and each sentence or claim of the answer
as the hypothesis, and asks whether the premise entails it. That is natural language inference
(NLI), and small dedicated models do it well:

- **Vectara HHEM-2.1-Open** ([model card](https://huggingface.co/vectara/hallucination_evaluation_model))
  is a flan-t5-base classifier under Apache-2.0. It takes (premise, hypothesis) pairs and returns a
  score from 0 to 1, runs on a CPU in under 600 MB of memory, and is English only. On the card's
  benchmarks it scores slightly above GPT-4 on balanced accuracy.
- **MiniCheck** ([Tang et al., EMNLP 2024](https://arxiv.org/abs/2404.10774)) scores a document
  against one sentence-level claim and returns a support probability. The paper reports
  "GPT-4-level performance but for 400x lower cost". The
  [repo](https://github.com/Liyan06/MiniCheck) ships checkpoints from RoBERTa-large up to a 7B
  model.
- **LettuceDetect** ([Kovács and Recski, 2025](https://arxiv.org/abs/2502.17125)) labels the
  unsupported tokens in an answer, not just the answer as a whole, which tells you which span to
  look at. It is MIT-licensed, trained on RAGTruth, and reports 79.22% example-level F1 on it.
- **Patronus Lynx** ([Ravi et al., 2024](https://arxiv.org/abs/2407.08488)) is a fine-tuned Llama 3
  judge in 8B and 70B sizes. The 70B model scores 87.4% on the paper's HaluBench against GPT-4o's
  86.5%. Its license is CC BY-NC 4.0, so it is not for commercial use.

The evaluation libraries wrap the same idea. RAGAS
[Faithfulness](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/)
extracts the claims in a response and reports the share the retrieved context supports, and its
`FaithfulnesswithHHEM` variant uses HHEM for the per-claim check. DeepEval has two metrics that are
easy to confuse:
[FaithfulnessMetric](https://deepeval.com/docs/metrics-faithfulness) checks the output's claims
against `retrieval_context`, what your pipeline actually retrieved, while
[HallucinationMetric](https://deepeval.com/docs/metrics-hallucination) checks the output against
`context`, ground-truth context you supply in the test case. FaithfulnessMetric counts a claim as
truthful unless it contradicts the context, so an unsupported but uncontradicted claim passes it.
If extrinsic hallucinations matter to you, check what your metric does with "not mentioned".

Two limits apply to every grounding check. It verifies the answer against the source, not the
source against the world, so a wrong document produces a confidently grounded wrong answer. And the
check is only as good as the context you pass it: if you give the detector the top 3 passages and
the model saw the top 10, correct answers drawn from passages 4 to 10 get flagged.

### 2. Split the answer into claims and verify each

Long answers mix supported and unsupported statements, and a single score for the whole answer
hides which is which. [FActScore](https://arxiv.org/abs/2305.14251) (Min et al., EMNLP 2023)
splits a generation into atomic facts and reports the share a knowledge source supports. On
biographies checked against Wikipedia by human annotators, ChatGPT scored 58%, and the paper's
automated estimator came within 2% of the human numbers.

The production version of this is an LLM judge with a two-step prompt: list the claims, then mark
each supported, contradicted or not mentioned, citing the passage. Ask for the verdict per claim,
not one overall score. Per-claim output gives you something to audit, and it is what you need to
compute precision and recall later. The [LLM-as-a-judge](/blog/llm-as-a-judge) post covers how to
write that prompt and validate the judge, and the biases to expect.

[Chain-of-Verification](https://arxiv.org/abs/2309.11495) (Dhuliawala et al., Findings of ACL 2024)
uses the same decomposition to reduce hallucinations rather than detect them: draft an answer,
write verification questions about it, answer each independently of the draft, then revise.

### 3. Sample several answers and compare them

When there is no source to check against, the model's own consistency is a signal. A fact the
model knows tends to come back the same way across samples. A fact it is making up tends to change.

[SelfCheckGPT](https://arxiv.org/abs/2303.08896) (Manakul et al., EMNLP 2023) samples several extra
responses and checks each sentence of the main answer against them. Its
[repo](https://github.com/potsawee/selfcheckgpt) reports sentence-level AUC-PR on its WikiBio
benchmark of 92.50 for the NLI variant and 93.42 for the variant that prompts gpt-3.5-turbo,
against 72.96 for random guessing. It needs only black-box access to the model.

Semantic entropy ([Farquhar et al., Nature 2024](https://www.nature.com/articles/s41586-024-07421-0))
refines the idea. It clusters the sampled answers by meaning, putting two answers in the same
cluster when each entails the other, and measures entropy over the clusters, so "Paris" and "It's
Paris, France" count as agreement. Averaged over 30 task and model combinations it reached an AUROC
of 0.790, against 0.691 for plain token entropy. The authors' blog puts the cost at roughly 10
times a single answer. [Semantic entropy probes](https://arxiv.org/abs/2406.15927) (Kossen et al.,
2024) estimate the same signal from the hidden states of one generation, which removes the
sampling cost but requires access to the model's internals, so it works only on models you host.

What sampling cannot catch: an error the model makes consistently. If it learned a wrong fact, it
will repeat it in every sample, and consistency reads that as confidence. The Nature paper is
explicit that it targets confabulations, answers that are both wrong and arbitrary.

### 4. Token probabilities

Low token probability can flag uncertain spans, and it is free when the API returns it. Support is
uneven. OpenAI's Chat Completions API takes `logprobs` and `top_logprobs` (up to 20), and its
Responses API returns them when you add `message.output_text.logprobs` to `include`. Gemini takes
`responseLogprobs` and `logprobs` in its generation config. Anthropic's
[Messages API](https://platform.claude.com/docs/en/api/messages/create) has no logprobs parameter,
so for Claude you need one of the other methods. Even where it is available, a low probability
marks uncertainty about wording as readily as uncertainty about facts, so treat it as a cheap
first filter, not a verdict.

### 5. Deterministic checks for structured output

A large share of production hallucinations are in fields, not prose: an enum value that does not
exist, a date in the wrong format, a product SKU the catalog has never had. These need no model to
detect:

- Validate the output against a JSON Schema, with enums and patterns, not just types.
- Check every identifier against the system it refers to: does the order, user or SKU exist?
- Resolve every citation and URL. A citation that does not resolve is a fabrication.
- Check numbers the output copies from the context against the context.

These checks have no false positives you cannot explain, they cost nothing, and they are the first
thing to add. Most teams add them last.

## Hallucinations in AI agents

An agent can hallucinate in places a chatbot cannot, and most of them never show up in the reply.
The [Relign paper](https://arxiv.org/abs/2412.04141) (Xu et al., 2024) splits tool hallucinations
into **selection** errors (a nonexistent or irrelevant tool, or a call at the wrong time) and
**usage** errors (malformed arguments, or values fabricated rather than taken from the request).
A 2025 [survey of agent hallucinations](https://arxiv.org/abs/2509.18970) adds the one I worry
about most: agents that claim to have completed steps they never performed.

In recorded traces, four kinds account for most of what I see:

| Kind | What the model does | Example | How to detect it |
| --- | --- | --- | --- |
| Invented tool | Calls a tool that was never offered | `cancel_order` on an agent that only has `lookup_order` and `issue_refund` | Compare each call's name against the tools sent with the request |
| Invented argument | Passes a value nothing in the context supports | `issue_refund(order_id="A-77310")` when the user said "my last order" and no lookup ran | Validate against the tool's JSON Schema, and check that every ID came from the user or an earlier tool result |
| Unbacked action claim | Says it did something it never called a tool for | "I've refunded your order" with no `issue_refund` call in the trace | Match past-tense action claims in the reply against the calls that were made |
| Ignored tool result | Answers from memory after a tool returned something else | The lookup says the order shipped on the 12th; the reply says it ships tomorrow | A grounding check, with the tool results as the source |

The unbacked claim is the dangerous one. A reply that claims a refund reads exactly like a reply
after a refund, so every check that reads only the text passes it. It is not hypothetical:
Transluce [reported](https://transluce.org/investigating-o3-truthfulness) in April 2025 that a
pre-release OpenAI o3 claimed to have run code it had no tool to run, and defended the claim when
asked. The trace is the evidence. If the claim has no matching call, the action did not happen.

The first three kinds need no model and no reference answer, just the trace and the tool
definitions. This script checks a JSONL file of recorded traces, one trace per line with the tools
offered, the user messages, the tool calls with their results and the final reply:

```python
"""Flag agent hallucinations in recorded traces. Needs: pip install jsonschema"""
import json
import re
import sys

from jsonschema import Draft202012Validator

# Past-tense claims of an action, mapped to the tools that would back them.
CLAIMS = {
    r"\bI(?:'ve| have)? (?:just |already )?(?:refunded|issued a refund)": {"issue_refund"},
    r"\bI(?:'ve| have)? (?:just |already )?(?:checked|looked up|searched)": {"lookup_order", "search_kb"},
    r"\bI(?:'ve| have)? (?:just |already )?(?:cancell?ed)": {"cancel_order"},
}
ID_FIELD = re.compile(r"(?:^|_)id$")


def check(trace):
    offered = {t["name"]: t for t in trace["tools"]}
    seen = " ".join(m["content"] for m in trace["messages"] if m["role"] == "user")
    called, problems = set(), []
    for call in trace["tool_calls"]:
        name, args = call["name"], call["arguments"]
        if name not in offered:
            problems.append(f"invented tool: {name}")
            continue
        for err in Draft202012Validator(offered[name]["parameters"]).iter_errors(args):
            problems.append(f"{name}: {err.message}")
        for key, value in args.items():
            # An ID the model passes must come from the user or an earlier tool result.
            if ID_FIELD.search(key) and str(value) not in seen:
                problems.append(f"{name}: {key}={value!r} appears nowhere in the context")
        called.add(name)
        seen += " " + json.dumps(call.get("result", ""))
    for pattern, tools in CLAIMS.items():
        claim = re.search(pattern, trace["reply"], re.IGNORECASE)
        if claim and not tools & called:
            problems.append(f"unbacked claim: {claim.group(0)!r} with no {' or '.join(sorted(tools))} call")
    return problems


if __name__ == "__main__":
    failed = 0
    for line in open(sys.argv[1]):
        trace = json.loads(line)
        for problem in check(trace):
            print(f"{trace['id']}: {problem}")
            failed += 1
    sys.exit(1 if failed else 0)
```

Run on five fixture traces of a refund agent, one clean and four broken, it prints this and exits 1,
so it can fail a pipeline:

```text
unbacked: unbacked claim: "I've refunded" with no issue_refund call
invented-id: issue_refund: order_id='A-77310' appears nowhere in the context
invented-tool: invented tool: cancel_order
invented-tool: unbacked claim: "I've cancelled" with no cancel_order call
bad-args: lookup_order: '10422' does not match '^A-\\d{5}$'
bad-args: lookup_order: Additional properties are not allowed ('include_history' was unexpected)
```

The clean trace, a lookup followed by a refund with the looked-up amount, prints nothing. The claim
patterns are English and hand-written, so they miss paraphrases ("your money is on its way back")
and flag some innocent sentences ("I checked the code you pasted"). Treat a hit as a case to read,
and grow the list from the misses you find. The ID check is a substring match, which is crude but
catches the common case of a model inventing a plausible-looking identifier. The fourth kind, an
ignored tool result, needs a grounding check from method 1, with the tool results as the source.

A related failure is calling a tool when the right answer was to call none. In the
[model upgrade example](/blog/llm-regression-testing-model-upgrade), asked to convert kilometres to
kilograms, the old model explained that the units don't convert and the new one called
`convert_units` anyway. The fix there was a test case that asserts no tool is called. The
[tool-call evaluation](/blog/evaluating-agent-tool-calls) post covers scoring the rest of the
trace.

## How to tell whether your detector works

A hallucination detector is a classifier, and it has a false positive rate and a false negative
rate like any other. Measure both before you let it block anything.

Public benchmarks give you a first comparison between methods:

| Benchmark | What it contains | Useful for |
| --- | --- | --- |
| [RAGTruth](https://arxiv.org/abs/2401.00396) (ACL 2024) | 17,790 RAG responses from six models across QA, data-to-text and summarization, labeled at word level | Grounding detectors, span-level detection |
| [HaluEval](https://arxiv.org/abs/2305.11747) (EMNLP 2023) | 35,000 samples across QA, dialogue, summarization and general queries | Response-level detection |
| [TruthfulQA](https://arxiv.org/abs/2109.07958) (ACL 2022) | 817 questions built to elicit common misconceptions | Factuality without a source |
| [SimpleQA](https://arxiv.org/abs/2411.04368) (2024) | 4,326 short fact questions, graded correct, incorrect or not attempted | Whether a model abstains instead of guessing |
| [HalluLens](https://arxiv.org/abs/2504.17550) (ACL 2025) | Generated short-form and long-form questions plus made-up entities | Factuality and refusal, with less test leakage |
| [ToolBeHonest](https://arxiv.org/abs/2406.20015) (2024) | 700 agent tasks with missing or limited tools | Whether an agent admits a task cannot be done |

Public leaderboards need careful reading. Vectara's
[hallucination leaderboard](https://github.com/vectara/hallucination-leaderboard) is the most
quoted. As of its October 22, 2026 update it listed 111 models, from 1.8% at the top to more than
20%. It measures one thing: how often a model's summary of a document adds or contradicts
something, as scored by Vectara's HHEM-2.3 model. Larger reasoning models often score worse on it
than small ones, and its own FAQ says it does not capture every type of hallucination. Read it as
a summarization faithfulness rate, not a general hallucination rate.

None of these measures your data. Before a detector gates anything, label 100 to 200 of your own
outputs by hand, including the failures you already know about, and run the detector on them.
Report three numbers:

- **Precision**: of the outputs the detector flagged, how many were real hallucinations. Low
  precision means your team learns to ignore it.
- **Recall**: of the real hallucinations, how many it flagged. Low recall means it passes the
  failures you built it for.
- **The interval** on each. On 40 labeled hallucinations, a recall of 35 out of 40 (87.5%) has a
  95% Wilson interval of roughly 74% to 95%. That range is the honest answer to "how good is the
  detector".

Then choose the threshold from those numbers, not from a default. A detector that blocks a
deploy should favor precision; one that queues cases for human review can favor recall.

## Hallucination detection in CI and when you change models

Detection in production tells you a hallucination happened. Detection before release tells you
whether a change made them more likely, and a model upgrade is the change most likely to.
The [LLM drift](/blog/llm-drift) post covers detecting model changes in general; for
hallucinations specifically:

1. **Keep a suite of cases that tempt the model to guess.** Questions the source does not answer,
   requests for tools that do not exist, IDs the user never gave. The right output for each is an
   abstention, a clarifying question or no tool call. Grow it from every hallucination you find in
   production.
2. **Run the cheap checks on every case.** Schemas, IDs, tool names and unbacked claims cost
   nothing and have explainable failures, so they can block.
3. **Run a grounding check on answers that have a source.** Use a small NLI model where you can, and
   an LLM judge where you need per-claim explanations.
4. **Compare against the current model, not against zero.** Every model hallucinates at some rate.
   The question for a release is whether the candidate's rate is higher than the current one's on
   the same cases, and whether the difference is larger than the noise.
5. **Keep a new judge-based check advisory until you have measured it.** A detector you have not
   validated should report, not block.

## Doing this with EvalShift

Disclosure again: I'm the founder. [EvalShift](/docs) is not a hallucination detector for live
traffic. It answers the question in step 4: does a different model, or a changed prompt,
hallucinate more than the current one on your recorded cases?

- **Cases from production.** The [capture SDK](/docs/sdk-capture) records agent runs with every
  tool call and result, and `evalshift capture sync` promotes them into a golden suite. Promotion
  refuses a turn that was offered tools, called none and whose reply claims a tool action, because
  saving it would make the hallucinated action the expected behavior. A turn that answered without
  a tool becomes a case that expects no tool call.
- **Tool hallucinations.** `tool_selection` grades each model's calls against the expected ones,
  and `tool_arguments` with `against: expected` scores argument values against the recorded ground
  truth. `trace_invariants` rules mark tools as forbidden or required, enforce an order, and check
  arguments against a JSON Schema. The [evaluators docs](/docs/evaluators) cover each.
- **Output hallucinations.** `json_schema` and `regex` checks cover structured fields, and a
  pairwise LLM judge with a groundedness criterion compares the two models' answers to the same
  case in randomized order.
- **The gate.** A [`migration_policy`](/docs/migration-policy) sets budgets such as
  `max_invariant_violations` and `max_tool_divergence`. The verdict is pass, conditional pass, fail
  or inconclusive, and inconclusive means the suite was too small to decide.

What it does not do: score a single model's output in production, or check facts against the
world. For those, use the grounding models and libraries above.

## FAQ

**What is LLM hallucination detection?** It is checking a model's output for claims that are false
or that its source does not support. The main methods are checking the answer against its source
with an entailment model or LLM judge, verifying each claim separately, sampling several answers
and measuring their disagreement, and deterministic checks on structured output and tool calls.

**How do you detect hallucinations in a RAG system?** Check each claim in the answer against the
retrieved passages the model actually saw. Small entailment models such as HHEM, MiniCheck or
LettuceDetect do this cheaply, and RAGAS Faithfulness and DeepEval FaithfulnessMetric do it with an
LLM. Decide whether a claim the passages never mention should count as a hallucination, because
metrics differ on this.

**Can you detect hallucinations without a reference answer?** Yes. Grounding checks need the
source, not a reference answer. Sampling methods such as SelfCheckGPT and semantic entropy need
neither: they generate several answers and treat disagreement as a sign the model is guessing.
They miss errors the model makes consistently.

**What is semantic entropy?** It is a hallucination detection method from a 2024 Nature paper by
Farquhar et al. It samples several answers, groups the ones that mean the same thing using
two-way entailment, and measures the entropy over the groups. High entropy means the model gives
different answers each time, a sign it is confabulating.

**What are tool hallucinations in AI agents?** They are errors in an agent's tool use: calling a
tool that does not exist, calling one at the wrong time, passing malformed arguments or values the
request never contained, and claiming to have taken an action without calling the tool. Most
never show in the reply text, so they are detected from the trace and the tool definitions.

**Which LLM hallucinates the least?** It depends on the task and on how hallucination is measured.
Vectara's leaderboard measures summarization faithfulness and ranks some small models above large
reasoning models. SimpleQA measures short factual answers and rewards abstaining over guessing.
Measure the models you are considering on your own cases.

## Keep reading

- [LLM-as-a-judge](/blog/llm-as-a-judge): how to write and validate the judge behind a claim-level
  check.
- [Evaluating agent tool calls](/blog/evaluating-agent-tool-calls): scoring the trace, not the
  reply.
- [LLM drift](/blog/llm-drift): detecting when the model behind your app changes.
- [Agent evaluation metrics](/blog/agent-evaluation-metrics): what to measure in an agent besides
  hallucinations.
