Each golden-suite example carries its own toolset, and EvalShift compares agent behaviour — which tools the model called, what arguments it passed, and how it sequenced them — across two model versions, on top of the ordinary text comparison.

The killer scenario it catches:

## The moving parts

• +Per-example toolsets: every example carries exactly one of toolset_ref (a content-addressed sidecar pointer, what capture promote/sync write) or an inline tools list ([] is a real “no tools offered” value). run resolves it per example — one suite freely mixes agent and text-only rows under the same prompt.
• +ToolTrace data model: provider-agnostic, populated from Anthropic / OpenAI / Gemini responses.
• +Three tool evaluators: tool_selection, tool_arguments, tool_trace_structure. A fourth, agent_trace, scores full timelines imported with evalshift traces import.
• +Suite ground truth: optional expected_tools, expected_tool_rounds, tool_result_fixtures, expected_tool_count, expected_no_tools, expected_parallel per example — derived from captures by capture promote/sync, rarely written by hand.
• +HTML report: side-by-side trace diffs in place of text panes for tool-evaluator regressions, plus a “Tools called (source → target)” column on every per-example row.

## Walkthrough

The suite comes from promoted captures of your own agent, which carry expected_tools and the recorded toolset for free:

evalshift init                            # capture-first evalshift.yaml

# instrument the agent with evalshift-sdk, run real traffic:
EVALSHIFT_CAPTURE=1 python run_agent.py

evalshift capture sync                    # captures -> golden suite; every
                                          # example carries its recorded toolset
evalshift compare --suite-name <suite> --to <candidate-model> --open

evalshift run resolves each example’s toolset and, when it is non-empty, dispatches via the tool-aware client path — each Call row in raw.jsonl carries a parsed ToolTrace. evalshift evaluate then runs tool_selection (and any other configured tool evaluators) against that example’s ground truth.

## Configuration

A minimal agent config:

version: 1
prompts:
  - id: replay
    detection: manual
    content: "{input}"
    variables: [input]

defaults:
  source_model: gemini-2.5-flash
  target_model: gemini-3.1-flash-lite-preview

evaluators:
  tool_selection:
    - name: routing
      conformance: expected         # each side vs example.expected_tools
      divergence: set               # target vs source, Jaccard on names
      severity_floor: high

  # Optional — enable as needed:
  # tool_arguments:
  #   - name: routing_args
  # tool_trace_structure:
  #   - name: routing_structure

## Toolset shape

A toolset — a <base>/toolsets/<hex>.json sidecar, or an example’s inline tools — is a list of tool dicts. Both provider shapes are accepted and mixed freely; the client re-serialises per target provider, so one toolset serves Anthropic, OpenAI, and Gemini models alike:

# Anthropic-shape
- name: issue_refund
  description: Issue a refund on an existing order.
  input_schema:
    type: object
    properties:
      order_id:   {type: string}
      amount_usd: {type: number}
    required: [order_id, amount_usd]
  strict: true      # optional

# OpenAI-shape (equivalent)
- type: function
  function:
    name: issue_refund
    description: Issue a refund on an existing order.
    parameters: {type: object, properties: {...}}
    strict: true    # optional

You rarely write a sidecar by hand — capture promote/sync write one per distinct toolset your captures recorded, content-addressed so two captures offering the same tools share one file. A hand-authored suite inlines the same shape directly as each example’s tools: list instead.

A tool may carry an optional strict: true — top-level in the Anthropic shape, function.strict in the OpenAI shape. It is replayed as function.strict to OpenAI targets and top-level strict to Anthropic targets; both serialisers emit it only when set, so non-strict toolsets fingerprint unchanged. Gemini function declarations have no strict mode, so a Gemini arm records it under dropped_params as tools.strict and the report shows a Constraints not honoured banner. The other tool-use constraints production used — tool_choice and parallel_tool_calls — ride on the example’s generation_config instead; see example fields.

## Suite ground truth

{"id": "ex_security_01", "inputs": {"query": "..."}, "tags": ["security"], "expected_tools": [{"tool_name": "notify_security_team"}], "toolset_ref": "sha256:1a2b3c..."}
{"id": "ex_text_01",     "inputs": {"query": "what is your refund policy?"}, "tags": ["text_only"], "expected_no_tools": true, "toolset_ref": "sha256:1a2b3c..."}

### Offered, requested, executed

A capture records three things that are easy to conflate: offered (model_call.toolset_ref — the tools passed to the model), requested (model_call.requested_tool_calls — the calls the model asked for in its response) and executed (the tool_call / tool_result events — what the app actually ran). The three can legitimately differ. Promotion uses the requested calls as ground truth whenever the capture recorded them and the executed calls otherwise, stamping the case promotion_source: requested | executed; the offered toolset is what every example carries as its toolset_ref. See captures for the fallback rules.

### Agent rounds

A captured agent turn is usually a loop: call tools, read results, call more, answer. Promotion groups those calls into rounds — one per recorded model call — and keeps every round on the case as expected_tool_rounds; expected_tools is always round 1 (expected_tool_rounds[0]).

Default (--rounds first): single-shot replay. run issues one model call per example and feeds no tool result back, so the candidate can only produce round 1, and round 1 is the only round it is scored against — scoring it against round-2 calls would record a regression no model could avoid. A multi-round capture promoted this way warns, naming the later calls it will not replay.

--rounds all: teacher-forced multi-round replay. Promotion also carries the recorded tool results on the case as tool_result_fixtures, one inner list per covered round aligned by position with expected_tool_rounds, each entry {tool_name, result, error}. run then replays the example round by round:

• +Round k sees the history prefix (if any), the rendered prompt, and then the recorded rounds 1..k−1 — the recorded assistant tool calls and the recorded results as ordinary assistant / tool messages. The candidate’s own calls are never fed back: source, target and the recording all saw byte-identical context in every round, which is what makes a per-round comparison fair. A string result is sent verbatim, anything else as JSON, and a recorded tool error as {"error": "..."}.
• +The replay covers every round the fixtures cover plus the round after it. When every tool round is covered that last round is the answer round — the recorded agent called nothing and produced its final text — so the text evaluators get a real answer to compare, and a candidate that keeps calling tools when it should have answered is caught.
• +Fixture coverage stops at the first round with a call that has no recorded result; promotion warns, naming the rounds the replay will cover. Later rounds stay on the case as expected_tool_rounds but are never replayed. If round 1 itself is uncovered, replay stays single-shot.
• +One raw.jsonl row per example per model, as before: tokens, cost and latency summed over rounds, text the last round’s answer, the ToolTrace carrying round_count and a round_index on every call. A model error in round k fails the example as round k/n: …; completed rounds are discarded, so a partially replayed example is unmeasured, not half-scored. The cost pre-flight counts one call per replayed round.
• +Scoring is per round — each tool evaluator grades round k against its own ground truth and records the mean over replayed rounds, with the per-round detail under metadata.rounds. Because the mean drops below 1.0 as soon as one round differs, max_tool_divergence counts an example as diverged when any round diverged. See picking an evaluator.

## Picking an evaluator

You can run multiple at once. Each becomes an independent comparison in analysis.json, with the existing Benjamini-Hochberg correction already adjusting for the multi-test count.

On a multi-round replay the HTML report and report.json show one line per round in the “Tools called (source → target)” column and prefix each trace-diff item with Round k:, so a divergence names the round it happened in; bundle trace events carry the round as round_index. Single-shot pairs render exactly as before and carry no rounds key.

## Troubleshooting

• +“broken eval harness” row from evaluate — the source model failed the recorded ground truth on at least half of the tool_selection.conformance rows. The suite’s expectations were captured from the source model, so when the source fails them the run measured the harness (wrong toolset attached, wrong prompt, suite promoted from a different agent), not the migration. Fix the harness before collecting more examples.
• +TOOL_GROUND_TRUTH_MISS — a conformance row where both sides missed the ground truth. Those rows leave every policy rate (a shared miss is not equivalence); they are reported as a count in failure_categories and a recommendations line.
• +Bimodal score distribution — tool evaluators often produce scores at exactly 0 or 1. The analysis layer’s Shapiro-Wilk fallback routes these through Wilcoxon signed-rank automatically.
• +“no matched calls between source and target” — the tool_arguments evaluator scores a regression when the target doesn’t reuse any of the same tool names as the source. Check tool_selection first to triage.
• +“expected_tools arguments use keys absent from the tool schema” — the signature of a capture that recorded a wrapper function’s parameters instead of the arguments the model actually passes. Promotion unwraps these automatically when the capture’s own recorded toolset schema confirms the wrapping; with no resolvable sidecar the recording is left untouched.
• +“Nothing measured” on a text evaluator — a turn that ends in tool calls has no text, so semantic and llm_judge mark it skipped rather than scoring a fake tie. A comparison where every row was skipped is reported as unknown, and a pass resting on it is downgraded — it is not evidence of equivalence.

For the underlying statistics behind these comparisons, see methodology.