The suite is a JSONL file — one example per line. Default path ./golden.jsonl, overridable with --suite <path> or --suite-name <name> (a key under suites: in config, e.g. a promoted capture suite). You rarely write it by hand — the recommended workflow derives it from recorded production behaviour via evalshift capture sync.
{"id": "ex_security_01", "inputs": {"query": "User account_42 had 5 failed login attempts in the last hour"}, "tags": ["security"], "expected_tools": [{"tool_name": "notify_security_team", "match_strategy": "subset"}], "toolset_ref": "sha256:1a2b3c..."}
{"id": "ex_text_only_01", "inputs": {"query": "What is your refund policy?"}, "tags": ["text_only"], "expected_no_tools": true, "toolset_ref": "sha256:1a2b3c..."}## Example fields
| Field | Type / default | Meaning |
|---|---|---|
| id | str, required, unique | Example id |
| inputs | dict, {} | Template-variable → value; must cover the prompt's variables |
| tags | list[str], [] | Slice labels |
| expected | dict | None | Reference output (most evaluators compare source vs target directly and ignore this) |
| expected_tools | list | None | Ground-truth tool calls, in order (agent prompts) |
| expected_tool_rounds | list[list] | None | The full recorded agent loop — one list per model turn that emitted tool calls. expected_tools is always expected_tool_rounds[0]. |
| tool_result_fixtures | list[list] | None | Recorded results of the calls in expected_tool_rounds — one inner list per covered round, positionally aligned, each entry {tool_name, result, error}. Written by capture promote/sync --rounds all; when present, run replays the example teacher-forced (one round per covered round plus the answer round) and the tool evaluators score per round. null = single-shot replay. |
| expected_tool_count | int ≥ 0 | None | Pin the total tool-call count |
| expected_no_tools | bool, false | Assert the model answers without any tool call. Only meaningful when the example's toolset is non-empty. |
| expected_parallel | bool | None | Assert parallel (or strictly sequential) tool calling |
| history | list | None | Prior conversation turns for teacher-forced replay |
| conversation_id | str | None | Groups sibling turns of one conversation |
| turn_index | int ≥ 0 | None | Position within the conversation |
| generation_config | dict | None | Generation settings the SDK recorded on the capture's first model call (temperature, response_mime_type, response_schema, tool_choice, parallel_tool_calls, tool_config, …), copied verbatim by promotion. The runner applies them to both the source and target calls at dispatch, so the replay runs under the settings production used. tool_choice (OpenAI string/object or Anthropic object), Gemini's tool_config and parallel_tool_calls normalise to one OpenAI-style tool_choice + parallel_tool_calls that LiteLLM maps per provider. A tool's strict flag is not a generation setting — it rides on the toolset. Keys the runner cannot translate are dropped with a warning; delete the field to disable. |
| toolset_ref | sha256:<hex> | Content-addressed pointer to a <base>/toolsets/<hex>.json sidecar — what capture promote/sync write, carried verbatim from the source capture's first model call. |
| tools | list | Inline toolset ({name, description, input_schema}, plus an optional strict: true — function.strict in the OpenAI shape) for a hand-authored suite — the alternative to toolset_ref. [] is a real "no tools offered" value, not an absence. |
## Expected tools
Each entry in expected_tools:
{"tool_name": "issue_refund",
"arguments": {"order_id": "12345", "amount_usd": 42.5},
"match_strategy": "subset"}- +
arguments: null→ name-only check. - +
match_strategy:exact(arguments must match exactly),subset(default; expected keys must be present and equal, extras allowed),contains_per_field(per-field containment).
toolset_ref or tools (neither, or both, fails to load). expected_no_tools: true is incompatible with non-empty expected_tools or a nonzero expected_tool_count; tool-call ground truth paired with an empty toolset is rejected; tool_result_fixtures requires expected_tool_rounds, cannot cover more rounds than it has, and every covered round needs one result per expected call with a matching tool_name; duplicate ids are rejected. The loader collects all schema errors before failing, so you fix a broken suite in one pass — before any money is spent.## Multi-turn conversations
EvalShift evaluates multi-turn agents by teacher-forced replay: each turn is one suite example carrying the conversation so far in history.
{"id": "conv1_t2", "inputs": {"input": "1pm works"}, "conversation_id": "conv_9f2", "turn_index": 2,
"history": [
{"role": "system", "content": "You are a scheduling assistant."},
{"role": "user", "content": "Can we move my appointment?"},
{"role": "assistant", "content": "", "tool_calls": [{"id": "c1", "name": "get_calendar", "arguments": {"day": "tue"}}]},
{"role": "tool", "tool_call_id": "c1", "content": "{\"slots\": [\"1pm\"]}"},
{"role": "assistant", "content": "Sure — what time works?"}
]}When history is present, run sends the recorded prefix verbatim, followed by the current turn’s rendered prompt as the final user message. The candidate model never generates its own intermediate turns — both models see byte-identical context, and only the current turn’s output is compared. That keeps every turn a clean paired measurement. Rules:
- +
historymay contain at most onesystemmessage, and it must come first.history: nullmeans single-turn;history: []is a conversational example with no prefix. - +Re-driving a whole conversation (feeding the candidate’s own reply into the next turn) is deliberately unsupported — it breaks the paired-comparison contract. The same rule holds inside a turn: a suite promoted with
--rounds allreplays the agent loop teacher-forced from the recordedtool_result_fixtures, never from the candidate’s own calls — see Agent rounds. - +History carries the agent loop: an
assistantturn may havetool_calls({id, name, arguments}), and atoolmessage carries that call’s result keyed bytool_call_id— required there, forbidden elsewhere, and an unpairable result is a load-time error. They are dispatched in the OpenAI wire shape and translated per provider, so the candidate model sees the tool results production saw. - +Prefer the SDK’s messages-list convention in your instrumentation so captures recover
historyverbatim. A capture that only recorded a bare string gets history reconstructed from sibling turns — an approximation whose intermediate tool exchanges are absent.
## Prompts
prompts is the template axis and suites the dataset axis: a run renders every prompt template with every example of the suite being run, so both are always present, and prompts is required even for a capture-first project. There the single replay prompt that init writes (content: "{input}") is a passthrough — a promoted example is {"input": "<full rendered prompt>"}, and echoing it back verbatim is what makes captured inputs replayable against a second model.
Two detection modes tell EvalShift where a prompt’s body lives:
prompts:
- id: greeting
detection: manual # inline
content: "Summarise: {text}"
variables: [text]
- id: customer_routing
detection: python_string # sourced from your code
path: prompts.py
variable: AGENT_SYSTEM_PROMPT
variables: [query]- +
manual— the body is the inlinecontent, verbatim. - +
python_string— EvalShift AST-walks the.pyfile for a module-level assignment and takes the string literal. Your code is never imported or executed. Only a plain string constant is accepted — f-strings, concatenation,.format()calls, and name references are rejected with a labeled error.
variables declares the {placeholder} names the template uses; every example’s inputs must supply them. Whether a row takes the agent path is decided per example, not per prompt: an example with a non-empty toolset (toolset_ref or inline tools) dispatches on the tool-aware client path and enables the tool evaluators for that row — see Agent migrations.
## Slices
A slice collects the examples whose tags contain the slice’s filter string. Every configured evaluator is analysed once overall and once per slice, and migration-policy budgets can be tightened per slice. See Configuration.