Build a golden eval suite from production traffic

Every eval suite written from memory is a suite about the cases you already handle. You sit down with a blank file, imagine your users, and produce twenty prompts that look like the happy path — because the happy path is the only part of the system you have a clear mental model of. The turn where the model drops a constraint set nine turns earlier, the ticket in a language your template never anticipated, the refund request phrased as a complaint: none of those get written down, because nobody remembers them as text. They only exist as traffic.

So record the traffic. This post is the mechanics of turning real agent runs into a golden JSONL suite you can gate a migration on — what the capture SDK writes, what evalshift capture sync derives from it, and the four places the pipeline silently drops a capture on purpose.

## The shape of the thing

Two packages, one directory between them:

your agent (+ evalshift-sdk)  →  .evalshift/captures/<suite>/cap_<hex>.json
                                            │
                                  evalshift capture sync
                                            ▼
                              .evalshift/suites/<suite>/golden.jsonl

They never call each other. The SDK writes files; the CLI reads them. That is the entire contract, which is why the two live in separate virtual environments despite sharing the top-level import name evalshift — one is pip install evalshift-sdk in your app's venv, the other is pip install evalshift wherever you run evals.

## Instrumenting without risking production

The SDK's central promise is fail-open: your function call is the only statement it does not wrap in a guard. Return values and exceptions propagate exactly as if the SDK were absent, and every piece of its own bookkeeping — opening spans, serializing, writing — degrades to a dropped capture plus one debug log line rather than an exception in your request path.

from evalshift import capture, record_model_call


@capture.tool
def lookup_order(order_id: str) -> dict:
    return db.orders.get(order_id)


@capture.agent(suite="support")
def handle_ticket(query: str) -> str:
    reply = call_model(query)
    record_model_call(model_id="claude-sonnet-5", input=[{"role": "user", "content": query}],
                      output=reply)
    return reply

@capture.agent marks one invocation as one capture file and derives the agent's input by binding the call arguments against the signature. @capture.tool records each tool as a timed span that expands into a tool_call event and a tool_result event. Both auto-detect async def, and session state lives in contextvars, so tools running under asyncio.gather still get correct parentage.

Nothing is recorded until EVALSHIFT_CAPTURE is truthy — 1, true, yes, on — and that gate is read live on every call, so you can turn capture on for an hour on one host and off again without a deploy. Leaving the decorators in production is the intended end state, not a debugging phase.

Three knobs keep a long-running host from filling its disk, and they compose: dedup on (suite, input_hash) is on by default, GC caps each suite directory at the newest 200 files after every write (capture_ttl adds age-based eviction), and sample_rate — off by default — skips all bookkeeping for undrawn runs. Also worth setting on an eval-grade host: configure(require_model_call=True), which drops captures with no model_call span, since a capture with no model output carries nothing to score.

### Redact before anything hits disk

Captures record the inside of a run — tool arguments, tool results, model inputs and outputs — which in a support agent means PII on nearly every line. Redaction is opt-in, runs in process before serialization, and is deliberately fail-closed: a redactor that raises drops the whole capture rather than writing a half-masked one. Your agent is unaffected either way.

configure(redact=default_redactor)   # or redact= per agent, which wins over the global

default_redactor masks emails, sk--style and AKIA keys, and Bearer tokens. It is not a comprehensive PII scrubber and does not touch structural metadata — tool names, model_id, token counts, timestamps. Structured secrets need a domain-specific redactor. Details in /docs/sdk-redaction.

## Record conversations as conversations

A multi-turn agent evaluated one isolated turn at a time is not being evaluated. Schema 1.1.0 carries conversation_id, turn_index, and parent_capture_id on the envelope, and the decorator is the wrong tool for them: decorator kwargs are fixed at decoration time, so every call would stamp the same turn_index. Open one session per turn instead.

conversation_id = f"conv_{uuid.uuid4().hex}"

for turn_index, user_text in enumerate(user_turns):
    messages.append({"role": "user", "content": user_text})
    with capture.agent_session(suite="scheduler", agent_input=messages,
                               conversation_id=conversation_id, turn_index=turn_index):
        reply = run_model(messages)
        record_model_call(model_id="claude-sonnet-5", input=messages, output=reply)
    messages.append({"role": "assistant", "content": reply})

Passing the messages list rather than a bare string is the other decision worth making once. Captures recorded that way recover conversation history verbatim at promotion time; captures that recorded only a string get history reconstructed from sibling turns, which is an approximation — assistant replies come from final outputs and intermediate tool exchanges are simply absent. And when conversation_id is set, turn identity folds into input_hash, so short repeated turns ("yes", "1pm") stop colliding under dedup.

## Promote

evalshift capture list          # what got recorded
evalshift capture sync          # promote everything → suites + wire config

sync groups captures by conversation_id, orders them by turn_index, and builds one suite example per turn: first model input → inputs (a bare string lands under --input-var, default input), recorded tool calls → expected_tools, final output → expected, messages list → history. It writes .evalshift/suites/<suite>/golden.jsonl and rewrites the managed suites: block in evalshift.yaml, between the >>> evalshift suites markers — so the next evalshift all --suite-name <suite> finds it without further wiring.

Tool calls are grouped into agent rounds, split at each recorded model_call. Every round lands in expected_tool_rounds, but expected_tools is scoped to round one — the only round a single-shot replay can reproduce, because run does not feed tool results back. A multi-round capture prints a warning naming exactly which calls were dropped. --rounds all flattens everything instead, which is the wrong default and occasionally the right flag.

How strict the derived expectations are is yours to choose: --strict-args demands exact argument matches, --names-only ignores arguments entirely, --tool-count also pins the total number of calls. Start loose. A suite that fails on argument formatting teaches your team to ignore it.

### The four things sync throws away

None of these are bugs, and all four are the reason the resulting statistics mean anything.

Dedup is seeded from the cases already sitting in the suite directory, so it holds across repeated syncs rather than resetting each time. That is what makes "capture for a week, sync daily" work.

## Slice it while you still remember why

A slice is a named subset defined by a tag on the example, and every configured evaluator is analysed once overall and once per slice. This is how "the migration regressed" becomes "the migration regressed on the multilingual cases and is flat everywhere else."

slices:
  - name: refunds
    filter: refunds          # matched against each example's tags list

capture sync --tag refunds attaches the tag at promotion, which is the moment you actually know what a batch of captures represents. Tagging later means reading JSONL and guessing.

One behavior to know before you write six slices: slices holding exactly the same examples are collapsed to one before any test runs. Duplicate slices restate the same numbers as independent findings and skew the Benjamini-Hochberg correction anti-conservatively — extra copies of a p-value shrink every adjusted p-value in the family. all and anything named under migration_policy.slices always survive; otherwise the provenance tag captured loses to an ordinary tag, then alphabetical order decides. Drops appear on the terminal and as collapsed_slices in analysis.json.

## How much traffic is enough

Enough that the comparisons are testable, which is a smaller number than people fear and a larger one than a first sync usually produces. Deltas are grouped per (prompt_id, evaluator_name, slice_name), and each group is judged on its own: fewer than 5 paired observations and the comparison is skipped as insufficient, between 5 and 20 it runs but is flagged uncertain.

The trap is that slicing multiplies the number of groups without adding observations. Six slices over sixty examples can leave every slice below the testable threshold while the overall numbers look fine — a suite that appears to say nothing, when it is actually saying you cut it too thin. Add slices when a slice has cases to fill it, not when the category feels important.

## Then freeze it

The suite has to stop moving before the comparison starts. If cases are still being edited while two models run against them, the diff between the models is contaminated by the diff between the suites, and no amount of statistics separates those afterwards. Commit the JSONL, review changes to it like code, and treat a new capture batch as a new version of the suite rather than a patch to the running one.

The reward for that discipline is that the suite outlives the migration it was built for. The next model bump, the prompt rewrite two quarters from now, the vendor's silent alias update — all of them get measured against the same frozen set of real cases, which is the only way any of those comparisons are comparable to each other.

## Keep reading

• +How to test an LLM model migration before you ship it — what to do with the suite once it exists.
• +Evaluating agent tool calls — scoring the trace, not just the text.
• +Captures and SDK capture API — every field and flag.
• +Golden suite — the JSONL schema in full.