## AI evals for agents: what they are and how to write them

URL: https://www.evalshift.dev/blog/ai-evals-for-agents
Published: 2026-11-02
Tag: evals
Summary: AI evals are tests that grade what a model or agent did. How agent evals differ from LLM evals, the three grader types, and how to write one in 7 steps.

Takeaways:
- An AI eval is a test for an AI system: it gives a model or agent an input, lets it run, and grades the result with code, with an LLM judge, or with human review.
- LLM evals grade one response. Agent evals grade a whole run, including the tools called, their arguments, the state left behind and the final reply, because an agent can write a correct-sounding reply after doing the wrong thing.
- Grade an agent's outcome rather than its exact path, since agents often reach correct results by unanticipated routes, and check the path only for rules that must always hold, such as verifying before a write or charging at most once.
- Use code graders for everything checkable, an LLM judge only for the quality of prose that remains, and 30 to 50 human-labeled examples to calibrate the judge before it can block a release.
- Run each agent task several times and report pass^k, the share of tasks solved in every trial: in τ-bench, GPT-4o solved about 61% of retail tasks on one trial but only about 25% in all of eight.

An AI eval is a test for an AI system: you give it an input, let it run, and grade what it did
with code, with another model, or with a person. LLM evals grade a single response. Evals for AI
agents grade a whole run: the tools the agent called, the arguments it passed, the state it left
behind and the reply it ended with. That difference decides how you write them, because an agent
can reply "your refund is on its way" without ever calling the refund tool, and an eval that reads
only the reply will mark that run as a pass.

This guide defines the parts of an eval, explains how agent evals differ from LLM evals, covers the
three kinds of grader, and then walks through writing an eval for an agent in seven steps, with a
worked example you can adapt.

## What an AI eval is made of

Anthropic's guide to agent evals
([Demystifying evals for AI agents](https://anthropic.com/engineering/demystifying-evals-for-ai-agents),
January 2026) uses a vocabulary most eval tools now share. It is worth adopting, because "the eval
failed" can mean any of these:

| Term | What it is | Example for a support agent |
| --- | --- | --- |
| Task | One test case: an input plus what success means | "Order A1042 arrived broken, can I get my money back?" Success: one refund of $59.00 on A1042 |
| Trial | One attempt at a task. Agents vary between runs, so a task gets several | Run 3 of 5 |
| Grader | Logic that scores one aspect of a trial | "Exactly one refund exists for A1042" |
| Transcript (trace) | The full record of a trial: messages, tool calls, tool results, the final reply | `check_refund_policy` → `issue_refund(order_id="A1042", amount_usd=59.0)` → reply |
| Outcome | The state of the world after the trial | The row in the refunds table, not the sentence claiming it |
| Suite | The set of tasks you run together | 120 support tasks, tagged by route |

A task can have several graders, and usually should. One that checks the refund exists, one that
checks policy was looked up first, and one that checks the reply explains the timeline will each
fail for a different reason, and you want to know which.

## LLM evals vs agent evals

The terms overlap and are often used interchangeably, but they grade different things.

| | LLM eval | Agent eval |
| --- | --- | --- |
| Unit graded | One response to one prompt | A run: several model calls, tool calls and results |
| What can go wrong | Wrong, off-topic or malformed text | Wrong tool, wrong arguments, skipped step, repeated action, false claim of success, plus everything an LLM eval catches |
| Where the evidence is | The output text | The trace and the outcome, then the text |
| Determinism | Varies between samples | Varies more: one different tool call changes every step after it |

The practical consequence: an agent's worst regressions read as correct. A model that stops
calling `notify_security_team`, or reports an address change it never made, still writes fluent,
on-topic prose. Every eval that scores the text alone passes it.
[What actually breaks when you switch LLMs](/blog/what-breaks-when-you-switch-llms) goes through
the full change surface.

## The three kinds of grader

Every eval grader is one of three kinds, and a good suite uses all three for different jobs.

| Grader | Examples | Cost | Strength | Weakness |
| --- | --- | --- | --- | --- |
| Code | Schema validation, exact match, regex, tool-call checks, database state | Free, instant | Deterministic, cannot be argued with | Only checks what you can specify exactly |
| Model (LLM judge) | Rubric scoring, pairwise comparison, natural-language assertions | About $1 per 1,000 calls on a small judge | Handles open-ended text | Biased toward position, length and its own phrasing; needs calibration |
| Human | Expert review, spot checks, labeling | Reviewer hours | The ground truth everything else is calibrated against | Slow, expensive, drifts with fatigue |

The order matters. Use code for everything code can check, a model for the quality of prose that
is left, and people to write the criteria, label a calibration sample and read the failures.
[LLM-as-a-judge](/blog/llm-as-a-judge) covers the model grader in depth, including how accurate
judges are and how to validate one.

## Capability evals vs regression evals

Evals answer one of two questions, and mixing them up makes the numbers hard to read.

- **Capability evals** ask what the agent can do. They should start with a low pass rate, because
  a suite the agent already passes cannot show progress. You use them while building.
- **Regression evals** ask whether the agent still does what it did yesterday. They should sit near
  100%, so any drop is a signal. You run them on every change to the model, the prompt or the
  tools.

A capability task the agent passes reliably graduates into the regression suite. Most teams
building on a model provider spend more of their eval effort on regression than they expect,
because the things that change most often are not their code: a model upgrade, a provider switch,
a deprecated model with a retirement date.

## Agents are not deterministic: pass@k and pass^k

Run the same agent on the same task five times and you can get five different traces. One trial per
task measures luck as much as behavior. Two metrics describe the spread:

- **pass@k:** the chance that at least one of k trials succeeds. It rises with k, and it is the
  right number when one success is enough, as with a coding agent whose output a person will
  review.
- **pass^k:** the chance that all k trials succeed. It falls with k, and it is the right number for
  an agent customers talk to, because each of them gets one trial.

The gap is large. In τ-bench, a benchmark of agents serving simulated retail and airline customers
under written policies, GPT-4o with function calling solved about 61% of retail tasks on a single
trial, and its pass^8, solving the same task in all eight trials, fell to about 25%
([Yao et al., 2024](https://arxiv.org/abs/2406.12045)). An agent that succeeds 75% of the time per
trial passes three trials in a row only 42% of the time (0.75³).

## How to write an eval for an AI agent

The seven steps below take you from nothing to a suite you can gate a release on. They assume an
agent in or near production. If you are still prototyping, start with step 1 and stop after step 4.

## Step 1: start from real failures and real traffic

Do not start by imagining test cases. Start with the 20 to 50 cases that already went wrong: bug
reports, support escalations, transcripts someone flagged. Anthropic's guide makes the same
recommendation, because early on, the changes you make have large effects and a small suite
detects them.

Then add real traffic. Hand-written cases cover the situations you already thought of; production
logs contain the ones you did not. Sample 100 to 300 inputs, stratified by route or intent so a
rare but expensive route is not drowned out, strip personal data, and drop near-duplicates.
[Build a golden eval suite from production traffic](/blog/build-a-golden-suite-from-production-traffic)
covers capturing those runs, and [how many eval cases you need](/blog/how-many-eval-cases-do-you-need)
covers sizing.

Tag every case with its route and any known edge case. Pass rates per tag are what you will act
on later.

## Step 2: define success as an outcome, before you write a grader

Write down what done looks like for each task in terms of the world, not the reply:

- "One refund of $59.00 exists on order A1042", not "the agent says the refund was issued".
- "The booking exists for the 14:00 slot and no other booking was created", not "the reply
  confirms the booking".
- "No tool was called" for a question the agent should answer from its instructions.

If you cannot write success down, a grader cannot check it, and neither can a colleague. That is a
sign the task is ambiguous, not that you need a cleverer grader. For tasks with a fixed answer,
store the expected outcome on the task. For tasks with many valid answers, store the properties
every valid answer has.

## Step 3: grade the outcome, and the path only where it is a rule

Agents often reach a correct outcome by a route you did not anticipate: a different lookup order,
an extra search, a clarifying question. An eval that requires an exact sequence of tool calls fails
those runs and teaches you to ignore failures. Anthropic's advice is to
"grade what the agent produced, not the path it took."

There is one exception, and it matters most for agents that act. Some paths are rules, and
breaking them is a failure even when the outcome looks right:

- Authenticate or verify before any write.
- Look up the refund policy before issuing a refund.
- Charge a card at most once.
- Never call a deprecated or dangerous endpoint.
- Keep an amount inside its allowed bounds.

Write those as checks on the trace, one rule each. Leave everything else free.

## Step 4: use code first, a judge for the rest, people to calibrate

Go through the success definition from step 2 and assign each property to the cheapest grader that
can check it.

**Code graders** take the outcome and the trace rules: the refund row exists, its amount is right,
`check_refund_policy` ran before `issue_refund`, `issue_refund` ran once, the structured output
parses. A sketch in Python, independent of any framework:

```python
def grade(trial, db):
    tools = [call.name for call in trial.tool_calls]
    refunds = db.refunds_for(order_id="A1042")

    refund_issued = len(refunds) == 1 and refunds[0].amount_usd == 59.00
    policy_first = (
        "check_refund_policy" in tools
        and "issue_refund" in tools
        and tools.index("check_refund_policy") < tools.index("issue_refund")
    )
    # The reply may only claim a refund when one exists.
    honest_reply = refund_issued or "refund" not in trial.final_text.lower()

    return {
        "refund_issued": refund_issued,
        "policy_first": policy_first,
        "honest_reply": honest_reply,
    }
```

Return one result per check, not one combined score. A task that fails `policy_first` and passes
`refund_issued` is a different bug from one that fails both.

**A model grader** takes what code cannot: whether the reply explains when the money arrives,
whether the tone fits an upset customer. Give it one criterion per judge, phrased as observable
behavior, with a clear pass condition:

```text
Criterion: Does the reply tell the customer when the refund will reach them?
It passes if it gives a timeframe or a date. It fails if it says only that
the refund was issued, or gives no timing at all.
Answer PASS or FAIL, then one sentence of reasoning.
```

**People** label 30 to 50 trials by hand and compare their labels with the judge's. Every
disagreement is either a vague criterion or a judge you should not trust yet. Keep a new judge
advisory, reported but not blocking, until that comparison holds up.

## Step 5: run every task several times

Run each task at least three times, five where the stakes are high, and report pass^k for
customer-facing behavior: the share of tasks the agent got right every time. A task that passes
two runs out of three is not a pass. It is a flaky behavior your customers will meet one time in
three.

Several trials also separate noise from regressions. If the current model already disagrees with
itself on a task, a different result from a new model on that task proves nothing on its own.

## Step 6: read the transcripts, not just the score

An aggregate pass rate tells you that something moved, not what. Break results down per tag and
per grader, and read every failed transcript in the routes you care about. Most eval bugs show up
there first: a grader checking the wrong field, a task with an impossible success condition, a
test case that leaked into the prompt.

Read some passes too. A grader that never fails is either checking something trivial or broken.

## Step 7: freeze the suite and run it on every change

A regression suite only proves something if it holds still. Freeze the tasks and graders, version
them, and run the suite on every change that can alter behavior: the model name, the prompt, the
tool definitions, the agent's code. Comparing two runs on a suite that changed in between compares
two moving things.

When a change is intended, such as a new route or a corrected expectation, update the suite in its
own commit, so the next comparison starts from a known baseline. Then wire the run into CI, so a
pull request that regresses the agent fails before it merges.
[LLM regression testing in CI](/blog/llm-regression-testing-in-ci) shows a gate on GitHub Actions.

## Common mistakes in agent evals

- **Grading the claim instead of the outcome.** "The agent said it booked the slot" is not a
  booking. Check the state, or at least the tool call that would have changed it.
- **Requiring an exact path.** Pinning every tool call in order fails valid runs. Pin only the rules
  from step 3.
- **One averaged score.** "Quality dropped 3%" cannot be fixed. "Twelve tasks on the refund route
  skipped `check_refund_policy`" can.
- **One trial per task.** You will chase noise and miss flakiness.
- **A judge nobody validated.** A judge returns a confident number whether or not its criterion
  makes sense. Calibrate it against human labels first.
- **Changing two things at once.** A new model and a rewritten prompt in one comparison leave
  nobody able to say which one moved the score. Change one, measure, then change the other.

## Writing agent evals with EvalShift

EvalShift is built for the regression side of agent evals, and for the change most teams test least:
switching or upgrading the model. It replays a frozen suite against the current model and a
candidate, grades both sides of every task, and turns the differences into a verdict.

- **Tasks from real traffic.** The capture SDK records production runs, and `evalshift capture sync`
  turns them into a golden suite where every task carries its recorded tools and expected tool
  calls.
- **Trace graders, no code required.** `tool_selection` and `tool_arguments` check which tools were
  called and with which values, and `trace_invariants` holds both models to the rules from step 3:

```yaml
evaluators:
  trace_invariants:
    - name: refunds_contract
      owner: "@support-team"
      rules:
        - id: policy-before-refund
          type: order
          before: check_refund_policy
          after: issue_refund
        - id: one-refund
          type: call_count
          tool: issue_refund
          max_calls: 1
        - id: sane-amount
          type: arguments
          tool: issue_refund
          json_schema:
            type: object
            required: [order_id, amount_usd]
            properties:
              amount_usd: {type: number, minimum: 0.01, maximum: 500}
```

- **Judges that start advisory.** `llm_judge` criteria compare both models' replies pairwise, and
  `evalshift init` writes them as non-blocking until they are calibrated.
- **A verdict decided in advance.** A `migration_policy` block sets the budgets, such as zero
  invariant violations or at most 10% tool-selection divergence, before anyone sees a number.

One limit to know: EvalShift does not run your tools against a live system. It feeds each model the
tool results recorded in production, so it grades the calls a model makes, not the database state
after them. Outcome checks against a real environment, like the `refund_issued` check above, belong
in your own staging tests. The [agents docs](/docs/agents) cover replay in detail, and the
[trace_invariants reference](/docs/configuration#trace-invariants) lists every rule type.

## FAQ

**What are AI evals?** Tests for AI systems. An eval gives a model or agent an input, lets it run,
and grades the result with code, with another model acting as a judge, or with human review. A set
of evals run together is a suite, and its pass rates show whether a change to the model, prompt or
tools made the system better or worse.

**What is the difference between LLM evals and agent evals?** An LLM eval grades one response to one
prompt, usually its text. An agent eval grades a whole run: the tools the agent called, their
order and arguments, the state the run left behind, and the final reply. Agent evals need graders
on the trace and the outcome, because an agent can write a correct-sounding reply after doing the
wrong thing.

**How do you write an eval for an AI agent?** Start from 20 to 50 real failures plus a sample of real
traffic, define success as an outcome in the world, grade that outcome with code, add trace rules
only for steps that must never be skipped, use an LLM judge only for the quality of the reply, run
each task several times, read the failed transcripts, and freeze the suite so it can gate every
change.

**Should agent evals check the tool calls or only the final outcome?** Mostly the outcome, because
agents often reach a correct result by a path you did not anticipate, and exact-path checks fail
those runs. Check the path only for rules that must hold every time, such as verifying before a
write or charging at most once.

**How many eval cases do you need to start?** 20 to 50 cases drawn from real failures is enough to
start, because early changes have large effects. To compare two models or prompts per route, aim
for 100 to 300 cases from production, stratified so every route you would act on has about 20.

**What is the difference between pass@k and pass^k?** pass@k is the chance that at least one of k
trials succeeds, and rises with k. pass^k is the chance that all k trials succeed, and falls with
k. Use pass^k for customer-facing agents: in τ-bench, GPT-4o solved about 61% of retail tasks on one
trial but only about 25% in all of eight trials.

## Keep reading

- [How to evaluate AI agents](/blog/how-to-evaluate-ai-agents): the six dimensions to measure, the
  three levels to test at, and sandbox vs mocks vs replay.
- [Evaluating agent tool calls](/blog/evaluating-agent-tool-calls): the four tool-call evaluators,
  and what does not belong in trace expectations.
- [LLM-as-a-judge](/blog/llm-as-a-judge): how judges work, how accurate they are and how to
  validate one.
- [Build a golden eval suite from production traffic](/blog/build-a-golden-suite-from-production-traffic):
  step 1 in full.
- [How to test an LLM model migration before you ship it](/blog/test-llm-model-migration-before-you-ship):
  a regression eval for the model change itself.