Evaluating agent tool calls: what text evals can't see

The output looked identical. That is the sentence at the center of most agent migration incidents. Both models answered "I've refunded your order, you'll see it in three to five business days" — and one of them called issue_refund before lookup_order, on an order id it had not verified. Text evaluators cannot see that, because the thing that changed was never in the text.

Agent behavior is a trace: which tools, in what order, with what arguments, and how many times. It drifts independently of prose quality, which is why a migration can pass every judge criterion you wrote and still be the wrong deploy. This post is what to score on the trace, and how the four tool-call evaluators divide that work.

## Why text evaluators go quiet exactly here

On a turn where both models answered with tool calls and no prose, there is nothing to embed and nothing to judge. The semantic evaluator writes no record at all — rather than erroring on an empty embedding input or inventing a score — and llm_judge does the same without spending a judge call, since comparing two empty strings only ever returns a meaningless tie, and a fabricated tie is indistinguishable from a judged one.

Both behaviors are correct, and together they mean your text evaluators contribute nothing on the most agent-shaped turns in the suite. If tool calls are how your product does its work, the tool-call evaluators are not an addition to your eval config. They are the config.

(The asymmetric case is still scored: a target that went silent where the source answered in prose is exactly the regression semantic exists to catch.)

## The toolset travels with the example

Nothing on the prompt marks it as an agent prompt. The toolset rides on each golden-suite example instead — a content-addressed toolset_ref pointing at a sidecar file under .evalshift/toolsets/, or an inline tools list — so one suite freely mixes agent and text-only rows under the same prompt. For any example whose toolset is non-empty, the orchestrator sends the provider those tool definitions and records each response as a provider-agnostic ToolTrace — ordered ToolCalls carrying tool_name, arguments, call_id, parent_call_id, and sequence_index, plus final_text and refusal info.

A toolset accepts both provider shapes, as a flat list or {"tools": [...]}:

- name: issue_refund
  description: Issue a refund on an existing order.
  input_schema:
    type: object
    properties:
      order_id:   {type: string}
      amount_usd: {type: number}
    required: [order_id, amount_usd]

The model client serializes to whatever the target provider expects, so one file serves Anthropic, OpenAI, and Gemini alike — which matters, because a cross-provider migration is the case where a hand-maintained second copy of your tool schemas drifts first.

A capture-first suite never has that second copy at all. evalshift capture sync writes one sidecar per distinct toolset your captures recorded — content-addressed, so two captures offering the same tools share one file — and stamps each example's toolset_ref to match: the schemas your suite replays are the schemas your agent actually offered, recorded at the moment it offered them. Hand-authored suites inline the same shape as the example's tools: list instead, and tools: [] is a real value — "this example's agent had no tools available" is a first-class assertion, not an absence.

## The four evaluators

All of them are pure computation over recorded traces: no API calls, no cost, no opinion that can be argued with. Configure all four and you are still paying less than one judge criterion.

### tool_selection — did it call the right tools?

Two independent axes, each writing its own record, because a migration asks two different questions and the answers differ:

Conformance is the axis that measures correctness: the source model is a baseline, not an oracle, so both sides are graded against recorded ground truth — and a row where both sides missed it is tagged TOOL_GROUND_TRUTH_MISS and left out of the policy rates, because ground truth the source model itself fails is a broken harness, not a migration finding. Divergence measures drift, with the source as its own 1.0 baseline; its default is set rather than exact so reordered identical calls do not read as change. Turning both axes off is a config error.

Examples marked expected_no_tools score 1.0 on conformance if and only if zero calls were made. That is the evaluator for "answer the policy question, don't hit the database."

One extra knob deserves to be used more than it is: severity_floor: low|medium|high|critical means a regression on this evaluator can never be classified below the floor regardless of effect size. The canonical agent migration failure — the candidate quietly stops calling notify_security_team on security-sensitive tickets — is a small effect on a small slice, and a floor is what keeps it from being filed as low next to a formatting nit.

### tool_arguments — same tools, different values?

Calls are matched greedily by (tool_name, nearest sequence_index), then each argument field is scored by a per-field strategy: exact, subset, numeric (relative error decaying linearly to zero at numeric_tolerance, default 0.05), or semantic (embedding cosine, borrowing the configured semantic evaluator's model and cache — without one it degrades to exact). Fields you do not list default to exact.

Two defaults carry real judgment:

• +A field present on one side only scores 0.5, not 0. Omitting an optional parameter is a real difference, not a wrong value. optional_fields_scored: strict restores the harsher 0.0.
• +against: expected switches the whole comparison from drift-vs-source to correctness, scoring both sides against expected_tools[].arguments. Each expectation's match_strategy picks which keys get compared: exact compares the union, subset and contains_per_field compare only the recorded keys. An expected call the model never made scores 0; an example with no expected arguments is skipped neutrally at 1.0/1.0.

Default drift mode pins source_score at 1.0 by construction. That is fine for "did anything move" and wrong for "is it right" — a hallucinating source scores a perfect 1.0 forever. If your suite came from real captures, you have the ground truth; use against: expected.

### tool_trace_structure — the shape of the loop

Call-count drift within call_count_tolerance (default 1), parallelism match, expected_tool_count when set, and refusal alignment. Each check toggles off independently via check_call_count, check_parallelism, check_refusals.

Refusals are the sharp edge: a refusal mismatch forces severity to at least high and flags REFUSAL_REGRESSION. A model that started refusing work it used to do, or stopped refusing work it used to decline, is never a low-severity finding — and it is invisible to every evaluator that scores output text, because a refusal is fluent, well-formed prose.

### agent_trace — for agents that run outside EvalShift

If your agent loop lives in LangChain, a custom orchestrator, or another language entirely, run the model-call stage and then attach full timelines:

evalshift traces import <run-id> --source source_traces.jsonl --target target_traces.jsonl

Each line is one trace for a (prompt_id, example_id, role), with the same event schema the capture SDK writes. Then agent_trace scores order similarity (LCS-normalised), per-field argument equality on matched calls, and — the check with no equivalent anywhere else — missing verification:

evaluators:
  agent_trace:
    - name: safety
      check_missing_verification: true
      verification_tools: [confirm_with_user]
      dangerous_tools: [delete_record, transfer_funds]

An extra dangerous call on the target flags DANGEROUS_ACTION_DRIFT; a dangerous call with no preceding verification tool flags MISSING_VERIFICATION_STEP. Those two are worth encoding before you need them, because they are the failures that turn a quality regression into an incident report.

## Read the categories, not the average

Regressions carry machine-readable labels that the report and the Cloud diff group by: TOOL_SELECTION_DRIFT, ARGUMENT_VALUE_DRIFT, TOOL_TRACE_STRUCTURE_DRIFT, TOOL_ORDER_DRIFT, DANGEROUS_ACTION_DRIFT, MISSING_VERIFICATION_STEP, UNNECESSARY_TOOL_CALL, REFUSAL_REGRESSION, alongside the text-side ones.

The grouping is the point. "Agent quality dropped 4%" is not actionable and not even really a claim. "Twelve ARGUMENT_VALUE_DRIFT on issue_refund.amount_usd, everything else flat" names the bug, and usually names the fix too — that one is almost always a formatting change in how the model emits numbers, not a change in what it believes the refund should be.

## Budget it in the policy

Two migration_policy fields exist specifically for the argument surface:

migration_policy:
  max_tool_argument_drift: 0.01
  tool_argument_drift_floor: 0.9

The floor is what keeps the budget honest. Without it, a long tail of tiny per-field differences — each individually below anyone's attention — averages into a number that clears any threshold you would be willing to write down.

## What does not belong in the trace expectations

The failure mode on the other side is a suite so strict nobody can merge anything. Some of what a capture records is behavior; the rest is circumstance.

• +Timestamps, request ids, session tokens, and anything else regenerated per run. Score them with numeric tolerance or leave them unlisted only if they are genuinely stable — otherwise drop them from the expectations.
• +Retry loops. A recorded run that called the same tool three times because the first two timed out is a story about your network, and pinning expected_tool_count on it makes an infrastructure flake into a permanent model regression.
• +Tool results that failed. capture sync warns when a promoted turn contains a failed result (error, or {"success": false}) precisely because the model's next move was a reaction to a broken tool, not a decision worth reproducing.

Start with --names-only expectations, watch what the diff actually flags for a week, and tighten to arguments once you know which fields carry meaning. A gate people learn to re-run until it passes is worse than no gate.

## Keep reading

• +Build a golden suite from production traffic — where expected_tools comes from in the first place.
• +How to test an LLM model migration before you ship it — the paired run these evaluators score.
• +Agent evaluation — every mode, strategy, and default.
• +Evaluators — the full evaluator reference.