Evaluating agent tool calls: what text evals can't see

The output looked identical. That is the sentence at the center of most agent migration incidents. Both models answered "I've refunded your order, you'll see it in three to five business days" — and one of them called issue_refund before lookup_order, on an order id it had not verified. Text evaluators cannot see that, because the thing that changed was never in the text.

Agent behavior is a trace: which tools, in what order, with what arguments, and how many times. It drifts independently of prose quality, which is why a migration can pass every judge criterion you wrote and still be the wrong deploy. This post is what to score on the trace, and how the four tool-call evaluators divide that work.

## Why text evaluators go quiet exactly here

On a turn where both models answered with tool calls and no prose, there is nothing to embed and nothing to judge. The semantic evaluator skips the pair at 1.0/1.0 with metadata.skipped set — rather than erroring on an empty embedding input — and llm_judge skips at 0.5/0.5 without spending a judge call, since comparing two empty strings only ever returns a meaningless tie.

Both behaviors are correct, and together they mean your text evaluators contribute nothing on the most agent-shaped turns in the suite. If tool calls are how your product does its work, the tool-call evaluators are not an addition to your eval config. They are the config.

(The asymmetric case is still scored: a target that went silent where the source answered in prose is exactly the regression semantic exists to catch.)

## Turning a prompt agent-style

A prompt becomes an agent prompt when its tools_path is set. The orchestrator then sends the provider your tool definitions and records each response as a provider-agnostic ToolTrace — ordered ToolCalls carrying tool_name, arguments, call_id, parent_call_id, and sequence_index, plus final_text and refusal info.

The tool file accepts both provider shapes, YAML or JSON, a flat list or {"tools": [...]}:

- name: issue_refund
  description: Issue a refund on an existing order.
  input_schema:
    type: object
    properties:
      order_id:   {type: string}
      amount_usd: {type: number}
    required: [order_id, amount_usd]

The model client serializes to whatever the target provider expects, so one file serves Anthropic, OpenAI, and Gemini alike — which matters, because a cross-provider migration is the case where a hand-maintained second copy of your tool schemas drifts first.

If the definitions already live in Python, do not maintain a parallel file at all:

evalshift tools sync app/prompts/tool_definitions.py

That extracts every module-level literal list of tool dicts, merges them into .evalshift/tools.json, and sets tools_path: on your prompts in place, preserving comments and formatting. The file is AST-parsed and never imported — computed values are rejected rather than executed. Two different tools sharing a name abort the sync; identical duplicates collapse.

## The four evaluators

All of them are pure computation over recorded traces: no API calls, no cost, no opinion that can be argued with. Configure all four and you are still paying less than one judge criterion.

### tool_selection — did it call the right tools?

Five modes, and picking the wrong one is the most common misconfiguration here:

The default compares both sides against recorded ground truth, which is what you want: the source model is a baseline, not an oracle. The exact and set modes compare the target to the source, which measures drift rather than correctness — useful when you have no ground truth, misleading when you do, since a source that was already calling the wrong tool defines the yardstick.

Examples marked expected_no_tools score 1.0 if and only if zero calls were made. That is the evaluator for "answer the policy question, don't hit the database."

One extra knob deserves to be used more than it is: severity_floor: low|medium|high|critical means a regression on this evaluator can never be classified below the floor regardless of effect size. The canonical agent migration failure — the candidate quietly stops calling notify_security_team on security-sensitive tickets — is a small effect on a small slice, and a floor is what keeps it from being filed as low next to a formatting nit.

### tool_arguments — same tools, different values?

Calls are matched greedily by (tool_name, nearest sequence_index), then each argument field is scored by a per-field strategy: exact, subset, numeric (relative error decaying linearly to zero at numeric_tolerance, default 0.05), or semantic (embedding cosine, borrowing the configured semantic evaluator's model and cache — without one it degrades to exact). Fields you do not list default to exact.

Two defaults carry real judgment:

• +A field present on one side only scores 0.5, not 0. Omitting an optional parameter is a real difference, not a wrong value. optional_fields_scored: strict restores the harsher 0.0.
• +against: expected switches the whole comparison from drift-vs-source to correctness, scoring both sides against expected_tools[].arguments. Each expectation's match_strategy picks which keys get compared: exact compares the union, subset and contains_per_field compare only the recorded keys. An expected call the model never made scores 0; an example with no expected arguments is skipped neutrally at 1.0/1.0.

Default drift mode pins source_score at 1.0 by construction. That is fine for "did anything move" and wrong for "is it right" — a hallucinating source scores a perfect 1.0 forever. If your suite came from real captures, you have the ground truth; use against: expected.

### tool_trace_structure — the shape of the loop

Call-count drift within call_count_tolerance (default 1), parallelism match, expected_tool_count when set, and refusal alignment. Each check toggles off independently via check_call_count, check_parallelism, check_refusals.

Refusals are the sharp edge: a refusal mismatch forces severity to at least high and flags REFUSAL_REGRESSION. A model that started refusing work it used to do, or stopped refusing work it used to decline, is never a low-severity finding — and it is invisible to every evaluator that scores output text, because a refusal is fluent, well-formed prose.

### agent_trace — for agents that run outside EvalShift

If your agent loop lives in LangChain, a custom orchestrator, or another language entirely, run the model-call stage and then attach full timelines:

evalshift traces import <run-id> --source source_traces.jsonl --target target_traces.jsonl

Each line is one trace for a (prompt_id, example_id, role), with the same event schema the capture SDK writes. Then agent_trace scores order similarity (LCS-normalised), per-field argument equality on matched calls, and — the check with no equivalent anywhere else — missing verification:

evaluators:
  agent_trace:
    - name: safety
      check_missing_verification: true
      verification_tools: [confirm_with_user]
      dangerous_tools: [delete_record, transfer_funds]

An extra dangerous call on the target flags DANGEROUS_ACTION_DRIFT; a dangerous call with no preceding verification tool flags MISSING_VERIFICATION_STEP. Those two are worth encoding before you need them, because they are the failures that turn a quality regression into an incident report.

## Read the categories, not the average

Regressions carry machine-readable labels that the report and the hosted diff group by: TOOL_SELECTION_DRIFT, ARGUMENT_VALUE_DRIFT, TOOL_TRACE_STRUCTURE_DRIFT, TOOL_ORDER_DRIFT, DANGEROUS_ACTION_DRIFT, MISSING_VERIFICATION_STEP, UNNECESSARY_TOOL_CALL, REFUSAL_REGRESSION, alongside the text-side ones.

The grouping is the point. "Agent quality dropped 4%" is not actionable and not even really a claim. "Twelve ARGUMENT_VALUE_DRIFT on issue_refund.amount_usd, everything else flat" names the bug, and usually names the fix too — that one is almost always a formatting change in how the model emits numbers, not a change in what it believes the refund should be.

## Budget it in the policy

Two migration_policy fields exist specifically for the argument surface:

migration_policy:
  max_tool_argument_drift: 0.01
  tool_argument_drift_floor: 0.9

The floor is what keeps the budget honest. Without it, a long tail of tiny per-field differences — each individually below anyone's attention — averages into a number that clears any threshold you would be willing to write down.

## What does not belong in the trace expectations

The failure mode on the other side is a suite so strict nobody can merge anything. Some of what a capture records is behavior; the rest is circumstance.

• +Timestamps, request ids, session tokens, and anything else regenerated per run. Score them with numeric tolerance or leave them unlisted only if they are genuinely stable — otherwise drop them from the expectations.
• +Retry loops. A recorded run that called the same tool three times because the first two timed out is a story about your network, and pinning expected_tool_count on it makes an infrastructure flake into a permanent model regression.
• +Tool results that failed. capture sync warns when a promoted turn contains a failed result (error, or {"success": false}) precisely because the model's next move was a reaction to a broken tool, not a decision worth reproducing.

Start with --names-only expectations, watch what the diff actually flags for a week, and tighten to arguments once you know which fields carry meaning. A gate people learn to re-run until it passes is worse than no gate.

## Keep reading

• +Build a golden suite from production traffic — where expected_tools comes from in the first place.
• +How to test an LLM model migration before you ship it — the paired run these evaluators score.
• +Agent evaluation — every mode, strategy, and default.
• +Evaluators — the full evaluator reference.