EvalShift is two packages that meet on disk: the capture SDK records what your agent actually does in production, and the CLI replays that recorded behaviour against a candidate model and tells you what regressed. Using them together is the recommended way to work — your golden suite is built from real traffic, not hand-authored guesses, and it stays current as your agent evolves. The third piece, the GitHub Action, is optional but completes the loop for teams: it runs the same suite on every pull request and blocks merges that regress it. SDK + CLI + Action is the full setup we recommend.

You can absolutely hand-write a golden.jsonl and use the CLI alone — see Golden suite. But hand-written suites go stale, and writing realistic expected_tools ground truth by hand is tedious. Captures derive it from behaviour you have already shipped.

## The loop

## 1–2. Instrument and capture

Install the SDK inside your agent’s environment and decorate the entry point. Capture is off by default — with the gate unset the decorators are zero-overhead pass-throughs, so they are safe to leave in production code:

# pip install evalshift already brings the SDK along;
# a production agent that only records captures installs it alone:
uv add evalshift-sdk

from evalshift import capture

@capture.agent(suite="support_agent", redact=True, tools=[])   # mask PII in-process
def handle_ticket(query: str) -> str:
    ...  # your agent: model calls, tools, retrieval

# nothing is recorded unless the gate is on
# EVALSHIFT_CAPTURE=1 python run_agent.py
# -> .evalshift/captures/support_agent/cap_<id>.json

See Instrumenting agents for tools, model calls, and async, and Redaction for masking PII before anything hits disk.

## 3. Promote captures into a golden suite

In your project directory (with evalshift init already run), turn the recorded captures into suite examples:

# wherever the CLI is installed — the agent's environment is fine
evalshift capture list                 # see what was recorded
evalshift capture sync                 # captures -> golden suite + wired config

capture sync builds one example per capture — first model input becomes inputs, recorded tool calls become expected_tools, the final output becomes expected, and multi-turn conversations recover their history. It also de-duplicates (duplicate examples would corrupt the paired statistics) and rewrites the managed suites: block in evalshift.yaml. Strictness knobs (--strict-args, --names-only, --tool-count) control how exacting the derived ground truth is; --rounds first|all chooses whether only the first agent round or every recorded round becomes ground truth, and all also carries the recorded tool results so run replays later rounds teacher-forced. Details in Captures → golden cases; the CLI repository’s capture-first example ↗ checks the whole loop in — instrumented agent, captures, and the derived suite.

## 4. Evaluate a candidate model

evalshift compare --suite-name support_agent --to <candidate-model> --open

The CLI replays every example on your current model and the candidate, scores both sides with the configured evaluators, runs paired statistics over the deltas, and renders a single-file HTML report. The canonical agent-migration failure this catches: the candidate silently stops calling a security-critical tool — tool_selection with severity_floor: high turns that into an unmissable red row.

## 5–6. Close the loop in CI

Everything so far runs on one machine with no account. When you want the suite enforced instead of remembered, three escalating options:

• +CI gating, local only — evalshift compare --gate critical,high or --policy-gate exit non-zero on regressions; no account needed. See Command reference.
• +Team review — push runs to EvalShift Cloud for shared reports, baselines, and diffs. Opt-in, and provider keys are never uploaded. See Cloud setup.
• +PR gating — the GitHub Action runs the suite on every pull request, diffs it against the latest base-branch run, posts one self-updating PR comment, and fails the check on regression. This is the recommended end state: the suite your captures built becomes a required check nobody has to remember to run.