The suite is a JSONL file — one example per line. Default path ./golden.jsonl, overridable with --suite <path> or --suite-name <name> (a key under suites: in config, e.g. a promoted capture suite). You rarely write it by hand — the recommended workflow derives it from recorded production behaviour via evalshift capture sync.

{"id": "ex_security_01", "inputs": {"query": "User account_42 had 5 failed login attempts in the last hour"}, "tags": ["security"], "expected_tools": [{"tool_name": "notify_security_team", "match_strategy": "subset"}]}
{"id": "ex_text_only_01", "inputs": {"query": "What is your refund policy?"}, "tags": ["text_only"], "expected_no_tools": true}

## Example fields

## Expected tools

Each entry in expected_tools:

{"tool_name": "issue_refund",
 "arguments": {"order_id": "12345", "amount_usd": 42.5},
 "match_strategy": "subset"}

• +arguments: null → name-only check.
• +match_strategy: exact (arguments must match exactly), subset (default; expected keys must be present and equal, extras allowed), contains_per_field (per-field containment).

## Multi-turn conversations

EvalShift evaluates multi-turn agents by teacher-forced replay: each turn is one suite example carrying the conversation so far in history.

{"id": "conv1_t2", "inputs": {"input": "1pm works"}, "conversation_id": "conv_9f2", "turn_index": 2,
 "history": [
   {"role": "system", "content": "You are a scheduling assistant."},
   {"role": "user", "content": "Can we move my appointment?"},
   {"role": "assistant", "content": "Sure — what time works?"}
 ]}

When history is present, run sends the recorded prefix verbatim, followed by the current turn’s rendered prompt as the final user message. The candidate model never generates its own intermediate turns — both models see byte-identical context, and only the current turn’s output is compared. That keeps every turn a clean paired measurement. Rules:

• +history may contain at most one system message, and it must come first. history: null means single-turn; history: [] is a conversational example with no prefix.
• +Re-driving a whole conversation (feeding the candidate’s own reply into the next turn) is deliberately unsupported — it breaks the paired-comparison contract.
• +tool-role messages in history are dropped with a warning. Prefer the SDK’s messages-list convention in your instrumentation so captures recover history verbatim.

## Prompts

Two detection modes tell EvalShift where a prompt’s body lives:

prompts:
  - id: greeting
    detection: manual            # inline
    content: "Summarise: {text}"
    variables: [text]

  - id: customer_routing
    detection: python_string     # sourced from your code
    path: prompts.py
    variable: AGENT_SYSTEM_PROMPT
    variables: [query]
    tools_path: tools.yaml       # presence makes this an agent prompt

• +manual — the body is the inline content, verbatim.
• +python_string — EvalShift AST-walks the .py file for a module-level assignment and takes the string literal. Your code is never imported or executed. Only a plain string constant is accepted — f-strings, concatenation, .format() calls, and name references are rejected with a labeled error.

variables declares the {placeholder} names the template uses; every example’s inputs must supply them. tools_path switches the prompt to the agent path — see Agent migrations.

## Slices

A slice collects the examples whose tags contain the slice’s filter string. Every configured evaluator is analysed once overall and once per slice, and migration-policy budgets can be tightened per slice. See Configuration.