Recommended workflow

EvalShift is two packages that meet on the capture layout (on disk, or in a bucket you own): the capture SDK records what your agent actually does in production, and the CLI replays that recorded behaviour against a candidate model and tells you what regressed. Using them together is the recommended way to work — your golden suite is built from real traffic, not hand-authored guesses, and it stays current as your agent evolves. The third piece, the GitHub Action, is optional but completes the loop for teams: it runs the same suite on every pull request and blocks merges that regress it. SDK + CLI + Action is the full setup we recommend.

You can absolutely hand-write a golden.jsonl and use the CLI alone — see Golden suite. But hand-written suites go stale, and writing realistic expected_tools ground truth by hand is tedious. Captures derive it from behaviour you have already shipped.

The loop

1–2. Instrument and capture

Install the SDK inside your agent’s environment and decorate the entry point. Capture is off by default — with the gate unset the decorators are zero-overhead pass-throughs, so they are safe to leave in production code:

# pip install evalshift already brings the SDK along;
# a production agent that only records captures installs it alone:
uv add evalshift-sdk

from evalshift import capture

@capture.agent(suite="support_agent", redact=True, tools=[])   # mask PII in-process
def handle_ticket(query: str) -> str:
    ...  # your agent: model calls, tools, retrieval

# nothing is recorded unless the gate is on
# EVALSHIFT_CAPTURE=1 python run_agent.py
# -> .evalshift/captures/support_agent/cap_<id>.json

See Instrumenting agents for tools, model calls, and async, Redaction for masking PII before anything hits disk, and Object-store sinks for writing captures to a bucket with EVALSHIFT_CAPTURE_STORE.

3. Promote captures into a golden suite

In your project directory (with evalshift init already run), turn the recorded captures into suite examples:

# wherever the CLI is installed — the agent's environment is fine
evalshift capture list                 # see what was recorded
evalshift capture sync                 # captures -> golden suite + wired config

capture sync builds one example per usable capture (a capture holding an image, document or other non-text content is skipped, with the reason) — first model input becomes inputs, recorded tool calls become expected_tools, the final output becomes expected, and multi-turn conversations recover their history. It also de-duplicates (duplicate examples would corrupt the paired statistics) and rewrites the managed suites: block in evalshift.yaml, keeping a managed entry’s hand-written evaluators.trace_invariants. Strictness knobs (--strict-args, --names-only, --tool-count) control how exacting the derived ground truth is; --rounds first|all chooses whether only the first agent round or every recorded round becomes ground truth, and all also carries the recorded tool results so run replays later rounds teacher-forced. Details in Captures → golden cases; the CLI repository’s capture-first example ↗ checks the whole loop in — instrumented agent, captures, and the derived suite.

4. Evaluate a candidate model

evalshift compare --suite-name support_agent --to <candidate-model> --open

The CLI replays every example on your current model and the candidate, scores both sides with the configured evaluators, runs paired statistics over the deltas, and renders a single-file HTML report. The canonical agent-migration failure this catches: the candidate silently stops calling a security-critical tool — tool_selection with severity_floor: high turns that into an unmissable red row. (A suite capture sync wired carries its own tool_selection, which replaces the top-level one — set managed: false on its suites: entry and add the floor to its evaluators: block — not needed for trace_invariants, which sync preserves; see Agent migrations.)

5–6. Close the loop in CI

Everything so far runs on one machine with no account. When you want the suite enforced instead of remembered, three escalating options:

• +CI gating, local only — evalshift compare --gate critical,high or --policy-gate exit non-zero on regressions; no account needed. See Command reference.
• +Team review — push runs to EvalShift Cloud for shared reports, baselines, and diffs. Opt-in, and provider keys are never uploaded. See Cloud setup.
• +PR gating — the GitHub Action runs the suite on every pull request, diffs it against the latest base-branch run, posts one self-updating PR comment, and fails the check on regression. This is the recommended end state: the suite your captures built becomes a required check nobody has to remember to run.