## Evaluating agent tool calls: what text evals can't see

URL: https://www.evalshift.dev/blog/evaluating-agent-tool-calls
Published: 2026-08-11
Tag: agents
Summary: Agent behavior drifts in the trace, not the prose. The four tool-call evaluators, the modes worth changing, and the expectations not worth pinning.

Takeaways:
- Agent regressions live in the trace, not the prose — a refusal regression is fluent, well-formed text, so every output-scoring evaluator misses it.
- Four evaluators cover the surface: tool_selection, tool_arguments, tool_trace_structure, and agent_trace for loops that run outside EvalShift.
- All four are pure computation over recorded traces — no API calls, and cheaper together than a single judge criterion.
- Read the machine-readable regression labels, not the average: "twelve ARGUMENT_VALUE_DRIFT on one field" names the bug, "quality dropped 4%" does not.
- Keep circumstance out of expectations — timestamps, retry loops, and failed tool results are not behavior worth pinning.

The output looked identical. That is the sentence at the center of most agent migration incidents.
Both models answered "I've refunded your order, you'll see it in three to five business days" — and
one of them called `issue_refund` before `lookup_order`, on an order id it had not verified. Text
evaluators cannot see that, because the thing that changed was never in the text.

Agent behavior is a trace: which tools, in what order, with what arguments, and how many times. It
drifts independently of prose quality, which is why a migration can pass every judge criterion you
wrote and still be the wrong deploy. This post is what to score on the trace, and how the four
tool-call evaluators divide that work.

## Why text evaluators go quiet exactly here

On a turn where both models answered with tool calls and no prose, there is nothing to embed and
nothing to judge. The `semantic` evaluator skips the pair at 1.0/1.0 with `metadata.skipped` set —
rather than erroring on an empty embedding input — and `llm_judge` skips at 0.5/0.5 without spending
a judge call, since comparing two empty strings only ever returns a meaningless tie.

Both behaviors are correct, and together they mean your text evaluators contribute nothing on the
most agent-shaped turns in the suite. If tool calls are how your product does its work, the tool-call
evaluators are not an addition to your eval config. They are the config.

(The asymmetric case is still scored: a target that went silent where the source answered in prose is
exactly the regression `semantic` exists to catch.)

## Turning a prompt agent-style

A prompt becomes an agent prompt when its `tools_path` is set. The orchestrator then sends the
provider your tool definitions and records each response as a provider-agnostic `ToolTrace` — ordered
`ToolCall`s carrying `tool_name`, `arguments`, `call_id`, `parent_call_id`, and `sequence_index`,
plus `final_text` and refusal info.

The tool file accepts both provider shapes, YAML or JSON, a flat list or `{"tools": [...]}`:

```yaml
- name: issue_refund
  description: Issue a refund on an existing order.
  input_schema:
    type: object
    properties:
      order_id:   {type: string}
      amount_usd: {type: number}
    required: [order_id, amount_usd]
```

The model client serializes to whatever the target provider expects, so one file serves Anthropic,
OpenAI, and Gemini alike — which matters, because a cross-provider migration is the case where a
hand-maintained second copy of your tool schemas drifts first.

If the definitions already live in Python, do not maintain a parallel file at all:

```bash
evalshift tools sync app/prompts/tool_definitions.py
```

That extracts every module-level literal list of tool dicts, merges them into `.evalshift/tools.json`,
and sets `tools_path:` on your prompts in place, preserving comments and formatting. The file is
AST-parsed and **never imported** — computed values are rejected rather than executed. Two different
tools sharing a name abort the sync; identical duplicates collapse.

## The four evaluators

All of them are pure computation over recorded traces: no API calls, no cost, no opinion that can be
argued with. Configure all four and you are still paying less than one judge criterion.

### `tool_selection` — did it call the right tools?

Five modes, and picking the wrong one is the most common misconfiguration here:

| Mode | Compares |
| --- | --- |
| `expected` (default) | Both sides against the example's `expected_tools`, matched **in order** |
| `expected_set` | The same ground truth, order-insensitive — multiset recall, for parallel fan-outs |
| `exact` | Sequence equality against the source model |
| `set` | Jaccard over the tool-name sets |
| `first` | First call only |

The default compares both sides against recorded ground truth, which is what you want: the source
model is a baseline, not an oracle. The `exact` and `set` modes compare the target to the *source*,
which measures drift rather than correctness — useful when you have no ground truth, misleading when
you do, since a source that was already calling the wrong tool defines the yardstick.

Examples marked `expected_no_tools` score 1.0 if and only if zero calls were made. That is the
evaluator for "answer the policy question, don't hit the database."

One extra knob deserves to be used more than it is: `severity_floor: low|medium|high|critical` means
a regression on this evaluator can never be classified below the floor regardless of effect size. The
canonical agent migration failure — the candidate quietly stops calling `notify_security_team` on
security-sensitive tickets — is a small effect on a small slice, and a floor is what keeps it from
being filed as `low` next to a formatting nit.

### `tool_arguments` — same tools, different values?

Calls are matched greedily by `(tool_name, nearest sequence_index)`, then each argument field is
scored by a per-field strategy: `exact`, `subset`, `numeric` (relative error decaying linearly to
zero at `numeric_tolerance`, default `0.05`), or `semantic` (embedding cosine, borrowing the
configured `semantic` evaluator's model and cache — without one it degrades to `exact`). Fields you
do not list default to `exact`.

Two defaults carry real judgment:

- A field present on one side only scores **0.5**, not 0. Omitting an optional parameter is a real
  difference, not a wrong value. `optional_fields_scored: strict` restores the harsher 0.0.
- `against: expected` switches the whole comparison from drift-vs-source to correctness, scoring
  **both** sides against `expected_tools[].arguments`. Each expectation's `match_strategy` picks which
  keys get compared: `exact` compares the union, `subset` and `contains_per_field` compare only the
  recorded keys. An expected call the model never made scores 0; an example with no expected
  arguments is skipped neutrally at 1.0/1.0.

Default drift mode pins `source_score` at 1.0 by construction. That is fine for "did anything move"
and wrong for "is it right" — a hallucinating source scores a perfect 1.0 forever. If your suite came
from real captures, you have the ground truth; use `against: expected`.

### `tool_trace_structure` — the shape of the loop

Call-count drift within `call_count_tolerance` (default 1), parallelism match, `expected_tool_count`
when set, and refusal alignment. Each check toggles off independently via `check_call_count`,
`check_parallelism`, `check_refusals`.

Refusals are the sharp edge: a refusal mismatch forces severity to at least `high` and flags
`REFUSAL_REGRESSION`. A model that started refusing work it used to do, or stopped refusing work it
used to decline, is never a low-severity finding — and it is invisible to every evaluator that scores
output text, because a refusal is fluent, well-formed prose.

### `agent_trace` — for agents that run outside EvalShift

If your agent loop lives in LangChain, a custom orchestrator, or another language entirely, run the
model-call stage and then attach full timelines:

```bash
evalshift traces import <run-id> --source source_traces.jsonl --target target_traces.jsonl
```

Each line is one trace for a `(prompt_id, example_id, role)`, with the same event schema the capture
SDK writes. Then `agent_trace` scores order similarity (LCS-normalised), per-field argument equality
on matched calls, and — the check with no equivalent anywhere else — missing verification:

```yaml
evaluators:
  agent_trace:
    - name: safety
      check_missing_verification: true
      verification_tools: [confirm_with_user]
      dangerous_tools: [delete_record, transfer_funds]
```

An extra dangerous call on the target flags `DANGEROUS_ACTION_DRIFT`; a dangerous call with no
preceding verification tool flags `MISSING_VERIFICATION_STEP`. Those two are worth encoding before
you need them, because they are the failures that turn a quality regression into an incident report.

## Read the categories, not the average

Regressions carry machine-readable labels that the report and the hosted diff group by:
`TOOL_SELECTION_DRIFT`, `ARGUMENT_VALUE_DRIFT`, `TOOL_TRACE_STRUCTURE_DRIFT`, `TOOL_ORDER_DRIFT`,
`DANGEROUS_ACTION_DRIFT`, `MISSING_VERIFICATION_STEP`, `UNNECESSARY_TOOL_CALL`,
`REFUSAL_REGRESSION`, alongside the text-side ones.

The grouping is the point. "Agent quality dropped 4%" is not actionable and not even really a claim.
"Twelve `ARGUMENT_VALUE_DRIFT` on `issue_refund.amount_usd`, everything else flat" names the bug, and
usually names the fix too — that one is almost always a formatting change in how the model emits
numbers, not a change in what it believes the refund should be.

## Budget it in the policy

Two `migration_policy` fields exist specifically for the argument surface:

```yaml
migration_policy:
  max_tool_argument_drift: 0.01
  tool_argument_drift_floor: 0.9
```

The floor is what keeps the budget honest. Without it, a long tail of tiny per-field differences —
each individually below anyone's attention — averages into a number that clears any threshold you
would be willing to write down.

## What does not belong in the trace expectations

The failure mode on the other side is a suite so strict nobody can merge anything. Some of what a
capture records is behavior; the rest is circumstance.

- Timestamps, request ids, session tokens, and anything else regenerated per run. Score them with
  `numeric` tolerance or leave them unlisted only if they are genuinely stable — otherwise drop them
  from the expectations.
- Retry loops. A recorded run that called the same tool three times because the first two timed out
  is a story about your network, and pinning `expected_tool_count` on it makes an infrastructure
  flake into a permanent model regression.
- Tool results that failed. `capture sync` warns when a promoted turn contains a failed result
  (`error`, or `{"success": false}`) precisely because the model's next move was a reaction to a
  broken tool, not a decision worth reproducing.

Start with `--names-only` expectations, watch what the diff actually flags for a week, and tighten to
arguments once you know which fields carry meaning. A gate people learn to re-run until it passes is
worse than no gate.

## Keep reading

- [Build a golden suite from production traffic](/blog/build-a-golden-suite-from-production-traffic)
  — where `expected_tools` comes from in the first place.
- [How to test an LLM model migration before you ship it](/blog/test-llm-model-migration-before-you-ship)
  — the paired run these evaluators score.
- [Agent evaluation](/docs/agents) — every mode, strategy, and default.
- [Evaluators](/docs/evaluators) — the full evaluator reference.