## Build a golden eval suite from production traffic URL: https://www.evalshift.dev/blog/build-a-golden-suite-from-production-traffic Published: 2026-08-07 Tag: suites Summary: Record real agent runs with the capture SDK, promote them into a golden JSONL suite, and understand every capture the pipeline drops on purpose. Takeaways: - Record real agent runs with the capture SDK, then promote them into a frozen golden JSONL suite — real traffic beats invented prompts. - Redaction happens inside the capture path, before anything reaches disk; redacting afterwards is not a boundary. - Captures are grouped by conversation_id and ordered by turn_index, so the case that broke in turn seven stays a case about turn seven. - capture sync drops content-duplicate captures on purpose: duplicates inflate n and corrupt the paired statistics that judge the migration. - Slice the suite while you still remember why each case matters, then freeze it — a suite that drifts cannot prove anything about a model change. Every eval suite written from memory is a suite about the cases you already handle. You sit down with a blank file, imagine your users, and produce twenty prompts that look like the happy path — because the happy path is the only part of the system you have a clear mental model of. The turn where the model drops a constraint set nine turns earlier, the ticket in a language your template never anticipated, the refund request phrased as a complaint: none of those get written down, because nobody remembers them as text. They only exist as traffic. So record the traffic. This post is the mechanics of turning real agent runs into a golden JSONL suite you can gate a migration on — what the capture SDK writes, what `evalshift capture sync` derives from it, and the four places the pipeline silently drops a capture on purpose. ## The shape of the thing Two packages, one directory between them: ``` your agent (+ evalshift-sdk) → .evalshift/captures//cap_.json │ evalshift capture sync ▼ .evalshift/suites//golden.jsonl ``` They never call each other. The SDK writes files; the CLI reads them. That is the entire contract, which is why the two live in separate virtual environments despite sharing the top-level import name `evalshift` — one is `pip install evalshift-sdk` in your app's venv, the other is `pip install evalshift` wherever you run evals. ## Instrumenting without risking production The SDK's central promise is fail-open: your function call is the only statement it does not wrap in a guard. Return values and exceptions propagate exactly as if the SDK were absent, and every piece of its own bookkeeping — opening spans, serializing, writing — degrades to a dropped capture plus one debug log line rather than an exception in your request path. ```python from evalshift import capture, record_model_call @capture.tool def lookup_order(order_id: str) -> dict: return db.orders.get(order_id) @capture.agent(suite="support") def handle_ticket(query: str) -> str: reply = call_model(query) record_model_call(model_id="claude-sonnet-5", input=[{"role": "user", "content": query}], output=reply) return reply ``` `@capture.agent` marks one invocation as one capture file and derives the agent's input by binding the call arguments against the signature. `@capture.tool` records each tool as a timed span that expands into a `tool_call` event and a `tool_result` event. Both auto-detect `async def`, and session state lives in `contextvars`, so tools running under `asyncio.gather` still get correct parentage. Nothing is recorded until `EVALSHIFT_CAPTURE` is truthy — `1`, `true`, `yes`, `on` — and that gate is read live on every call, so you can turn capture on for an hour on one host and off again without a deploy. Leaving the decorators in production is the intended end state, not a debugging phase. Three knobs keep a long-running host from filling its disk, and they compose: dedup on `(suite, input_hash)` is on by default, GC caps each suite directory at the newest 200 files after every write (`capture_ttl` adds age-based eviction), and `sample_rate` — off by default — skips all bookkeeping for undrawn runs. Also worth setting on an eval-grade host: `configure(require_model_call=True)`, which drops captures with no `model_call` span, since a capture with no model output carries nothing to score. ### Redact before anything hits disk Captures record the inside of a run — tool arguments, tool results, model inputs and outputs — which in a support agent means PII on nearly every line. Redaction is opt-in, runs in process before serialization, and is deliberately **fail-closed**: a redactor that raises drops the whole capture rather than writing a half-masked one. Your agent is unaffected either way. ```python configure(redact=default_redactor) # or redact= per agent, which wins over the global ``` `default_redactor` masks emails, `sk-`-style and `AKIA` keys, and `Bearer` tokens. It is not a comprehensive PII scrubber and does not touch structural metadata — tool names, `model_id`, token counts, timestamps. Structured secrets need a domain-specific redactor. Details in [/docs/sdk-redaction](/docs/sdk-redaction). ## Record conversations as conversations A multi-turn agent evaluated one isolated turn at a time is not being evaluated. Schema 1.1.0 carries `conversation_id`, `turn_index`, and `parent_capture_id` on the envelope, and the decorator is the wrong tool for them: decorator kwargs are fixed at decoration time, so every call would stamp the same `turn_index`. Open one session per turn instead. ```python conversation_id = f"conv_{uuid.uuid4().hex}" for turn_index, user_text in enumerate(user_turns): messages.append({"role": "user", "content": user_text}) with capture.agent_session(suite="scheduler", agent_input=messages, conversation_id=conversation_id, turn_index=turn_index): reply = run_model(messages) record_model_call(model_id="claude-sonnet-5", input=messages, output=reply) messages.append({"role": "assistant", "content": reply}) ``` > Always pass `agent_input=` to `agent_session`. It defaults to `None`, and the dedup registry keys > captures on `(suite, hash(agent_input))` — leave it unset and every session in the process hashes > identically, so dedup silently drops every capture after the first. Passing the messages list rather than a bare string is the other decision worth making once. Captures recorded that way recover conversation `history` verbatim at promotion time; captures that recorded only a string get history *reconstructed* from sibling turns, which is an approximation — assistant replies come from final outputs and intermediate tool exchanges are simply absent. And when `conversation_id` is set, turn identity folds into `input_hash`, so short repeated turns ("yes", "1pm") stop colliding under dedup. ## Promote ```bash evalshift capture list # what got recorded evalshift capture sync # promote everything → suites + wire config ``` `sync` groups captures by `conversation_id`, orders them by `turn_index`, and builds one suite example per turn: first model input → `inputs` (a bare string lands under `--input-var`, default `input`), recorded tool calls → `expected_tools`, final output → `expected`, messages list → `history`. It writes `.evalshift/suites//golden.jsonl` and rewrites the managed `suites:` block in `evalshift.yaml`, between the `>>> evalshift suites` markers — so the next `evalshift all --suite-name ` finds it without further wiring. Tool calls are grouped into agent rounds, split at each recorded `model_call`. Every round lands in `expected_tool_rounds`, but `expected_tools` is scoped to round one — the only round a single-shot replay can reproduce, because `run` does not feed tool results back. A multi-round capture prints a warning naming exactly which calls were dropped. `--rounds all` flattens everything instead, which is the wrong default and occasionally the right flag. How strict the derived expectations are is yours to choose: `--strict-args` demands exact argument matches, `--names-only` ignores arguments entirely, `--tool-count` also pins the total number of calls. Start loose. A suite that fails on argument formatting teaches your team to ignore it. ### The four things sync throws away None of these are bugs, and all four are the reason the resulting statistics mean anything. | Dropped | Why | | --- | --- | | Content-duplicate captures | Duplicates inflate `n`. Twenty recordings of "where is my order" make a comparison look twenty times more certain than it is. `--keep-duplicates` opts out. | | Turns with an `error` event | A turn that died before the agent acted is not ground truth — promoting it asserts `expected_no_tools: true` on a question that needed a tool. `--allow-errored` promotes anyway, still never asserting that. | | Captures with no `model_call` | Only with `require_model_call=True` set at capture time; nothing to score either way. | | Duplicate `(conversation_id, turn_index)` | A retried turn. This one stays a warning, not a drop — you decide which retry is canonical. | Dedup is seeded from the cases already sitting in the suite directory, so it holds across repeated syncs rather than resetting each time. That is what makes "capture for a week, sync daily" work. ## Slice it while you still remember why A slice is a named subset defined by a tag on the example, and every configured evaluator is analysed once overall and once per slice. This is how "the migration regressed" becomes "the migration regressed on the multilingual cases and is flat everywhere else." ```yaml slices: - name: refunds filter: refunds # matched against each example's tags list ``` `capture sync --tag refunds` attaches the tag at promotion, which is the moment you actually know what a batch of captures represents. Tagging later means reading JSONL and guessing. One behavior to know before you write six slices: slices holding exactly the same examples are collapsed to one before any test runs. Duplicate slices restate the same numbers as independent findings and skew the Benjamini-Hochberg correction anti-conservatively — extra copies of a p-value shrink every adjusted p-value in the family. `all` and anything named under `migration_policy.slices` always survive; otherwise the provenance tag `captured` loses to an ordinary tag, then alphabetical order decides. Drops appear on the terminal and as `collapsed_slices` in `analysis.json`. ## How much traffic is enough Enough that the comparisons are testable, which is a smaller number than people fear and a larger one than a first sync usually produces. Deltas are grouped per (`prompt_id`, `evaluator_name`, `slice_name`), and each group is judged on its own: fewer than 5 paired observations and the comparison is skipped as insufficient, between 5 and 20 it runs but is flagged uncertain. The trap is that slicing multiplies the number of groups without adding observations. Six slices over sixty examples can leave every slice below the testable threshold while the overall numbers look fine — a suite that appears to say nothing, when it is actually saying you cut it too thin. Add slices when a slice has cases to fill it, not when the category feels important. ## Then freeze it The suite has to stop moving before the comparison starts. If cases are still being edited while two models run against them, the diff between the models is contaminated by the diff between the suites, and no amount of statistics separates those afterwards. Commit the JSONL, review changes to it like code, and treat a new capture batch as a new version of the suite rather than a patch to the running one. The reward for that discipline is that the suite outlives the migration it was built for. The next model bump, the prompt rewrite two quarters from now, the vendor's silent alias update — all of them get measured against the same frozen set of real cases, which is the only way any of those comparisons are comparable to each other. ## Keep reading - [How to test an LLM model migration before you ship it](/blog/test-llm-model-migration-before-you-ship) — what to do with the suite once it exists. - [Evaluating agent tool calls](/blog/evaluating-agent-tool-calls) — scoring the trace, not just the text. - [Captures](/docs/captures) and [SDK capture API](/docs/sdk-capture) — every field and flag. - [Golden suite](/docs/golden-suite) — the JSONL schema in full.