The capture surface is small: one decorator for the agent, one for tools, and helpers for model calls. Everything is a no-op when the gate is off, so you can leave it in place.
## @capture.agent
Wrap your agent entry point. Inside the decorated call, the SDK builds a span tree and records every tool and model call made beneath it, then writes one capture file.
from evalshift import capture, record_model_call
@capture.agent(suite="support_agent", redact=True, tools=[])
def handle_ticket(query: str) -> str:
answer = call_model(query)
record_model_call(
model_id="claude-opus-4-8",
tools=None, # inherit the session's tools= above
input=query,
output=answer,
input_tokens=812,
output_tokens=140,
cost_usd=0.012,
latency_ms=940,
)
return answerrecord_model_call records an already-complete call. Its keyword args: model_id and tools (both required), plus optional input, output, input_tokens, output_tokens, cost_usd, latency_ms, generation_config, and requested_tool_calls. It is a no-op outside an active session.
requested_tool_calls is what the model asked to call in its response — distinct from tools= (what it was offered) and from the tool_call events @capture.tool records (what the app executed). The three diverge routinely — a guard rejects a requested call, the app runs a tool the model never asked for — and each divergence is a signal, so none is inferred from another. Pass a list of {name, arguments, call_id} items; the stdlib helper evalshift.capture.requested.extract_requested_tool_calls(response) builds it from an OpenAI (Chat Completions or Responses), Anthropic, or Gemini response — a dict or the response object itself. Pass [] when the model asked for nothing; omitting it records None (“not recorded”), which is a different fact. When present, the CLI promotes the case from it (promotion_source: requested) instead of from the executed calls. A malformed value is dropped with a debug log, never raised.
from evalshift.capture.requested import extract_requested_tool_calls
response = client.chat.completions.create(model="gpt-4o-mini", messages=messages, tools=TOOLS)
record_model_call(
model_id="gpt-4o-mini", tools=TOOLS, input=messages, output=text,
# what the model ASKED to call, read from the raw provider response;
# [] = it asked for nothing, None = not recorded
requested_tool_calls=extract_requested_tool_calls(response),
)redact= is a required keyword on every capture entry point — the decorator, both session forms, and the LangChain handler. True masks with default_redactor, False records verbatim on purpose, and a callable is your own; anything else raises TypeError. See Redaction.
tools= is required on the same entry points and on both model-call recorders — see Toolsets below for the full contract.
There is no separate “final output” field on the agent span — persist the agent’s answer as the last model call’s output.
## @capture.tool
Decorate tools to record a span per call, with arguments and result. Nested tools nest in the tree automatically. Use it bare or with an explicit name:
@capture.tool # name defaults to the function name def search_orders(user_id: str): ... @capture.tool(name="refund") # or name it explicitly def issue_refund(order_id: str): ...
## Toolsets — tools=
Every model call records the toolset it was offered, so a capture can tell “this agent had no tools” apart from “we don’t know what it was offered”. tools= is a required keyword — no default — on @capture.agent, both session forms, the LangChain handler, record_model_call, and capture.model_call. Omitting it is a TypeError at the call site (mypy flags it statically too).
tools = [ # what this call was actually offered
{"name": "search_orders", "description": "...", "input_schema": {...}},
]
@capture.agent(suite="support_agent", redact=True, tools=tools)
def handle_ticket(query: str) -> str:
# tools=None -> inherit the session's toolset above;
# a call's own non-None value always wins (agents can switch mid-run)
record_model_call(model_id="claude-opus-4-8", tools=None, input=query, output=...)
# a genuinely tool-less call is an asserted value, not a default
record_model_call(model_id="demo/router", tools=[], input=query, output=...)- +A real toolset — Anthropic (
{name, description, input_schema}), OpenAI ({type: "function", function: {...}}), or Geminitypes.Toolshape, or a list mixing any of those. Each is normalised to{name, description, input_schema}plus an optionalstrict: true— carried from OpenAI’sfunction.strictor Anthropic’s top-levelstrictwhen truthy, omitted entirely otherwise, so a replay re-sends the schema under the same constraint. Recorded on themodel_callevent astools_offered(a name-only list) andtoolset_ref(a content-addressedsha256:<hex>pointer to the full schema, written once per distinct toolset to<base>/toolsets/<hex>.json). - +
tools=[]— asserts this call genuinely had no tools. A real, first-class value, not a default. - +
tools=None— on the model-call recorders only: inherit the enclosing session’s owntools=instead of asserting one for this call. A call’s own non-Nonevalue always wins, because a real agent can switch toolsets mid-run.
A value matching no recognised shape is left unstamped (neither field written, logged at debug) rather than guessed at. Toolsets are config, not payload — they are never redacted, and unlike generation_config they are not allow-listed either: an input_schema is arbitrary user JSON needed in full to dispatch the tool.
## Async, streaming & concurrency
@capture.agent and @capture.tool auto-detect coroutine functions and wrap them in an async wrapper. Context propagates across await and into asyncio.gather child tasks, so concurrent tool calls land in the tree with the right parentage and a dense, collision-free ordering.
import asyncio
@capture.agent(suite="support_agent", redact=True, tools=[]) # async is auto-detected
async def handle_ticket(query: str) -> str:
# concurrent tools get correct parentage + ordering
orders, profile = await asyncio.gather(
search_orders(user_id),
load_profile(user_id),
)
return await summarize(orders, profile)### Streaming model calls
For token streams, open a recorder with capture.model_call(...), accumulate chunks with add_text, optionally record usage with set_usage and the model’s requested tool calls with set_requested_tool_calls (before or during the block; last write wins, so a streamed tool call can be set once its argument deltas have all arrived), and exactly one model-call span is recorded when the block exits. It works as a sync with or an async with. It must run inside an active agent — a @capture.agent call or an agent session — a bare capture.model_call outside one is an inert no-op.
@capture.agent(suite="support_agent", redact=True, tools=[])
def answer(query: str) -> str:
with capture.model_call(
model_id="claude-opus-4-8", tools=None, input=query # None -> inherit
) as rec:
for chunk in stream:
rec.add_text(chunk.text)
rec.set_usage(input_tokens=812, output_tokens=140, cost_usd=0.012)
return "done"
# exactly one model_call span recorded on exit (works with 'async with' too)Model calls can also record their generation config: pass generation_config= (a dict) to record_model_call / capture.model_call, or call rec.set_generation_config(...) for values that only become known mid-stream. Exactly ten keys are kept — temperature, top_p, response_mime_type, response_schema, response_format, max_output_tokens, max_tokens, tool_choice, parallel_tool_calls, tool_config (Gemini’s spelling of tool_choice) — everything else is dropped, so unlisted config (a system instruction, say) can never land in the capture unmasked. The filter is is not None, never truthiness, so parallel_tool_calls: false survives intact. Values are coerced to JSON: primitives, dicts and lists pass through, an object with a model_dump method (a Gemini ToolConfig, say — duck-typed, never imported) is dumped to a dict, and anything else degrades to its str(). evalshift capture sync uses it to replay promoted cases with the same settings, tool-use constraint included.
## Context-manager form
When a decorator doesn’t fit, capture.agent_session(...) (and capture.agent_session_async(...)) capture an inline block instead. They yield None when capture is disabled. Always pass agent_input — any JSON-able value; it is identity only, hashed into the envelope’s input_hash (the dedup key) and never stored raw. For a chat agent, pass the full messages list.
from evalshift import capture
with capture.agent_session(
suite="support_agent", agent_input=messages, redact=True, tools=[]
) as session:
... # session is None when the gate is offThe session’s scope is dynamic (contextvar-based), not lexical: once an agent wrapper or session is active, @capture.tool calls and model-call recorders attach to it from any function called inside it, however deep — recording calls don’t need to sit lexically inside the decorated function or with block. Sessions are also the recommended primitive for multi-turn conversations: one with block per turn, with a fresh turn_index.
## Failures are the best telemetry
When your agent raises, the SDK records an error event, still writes the partial capture, then re-raises the original exception unchanged. Your control flow is never altered.
cap_<id>.json: an ordered events[] trace (with concurrency metadata) plus the recorded tool_results kept as replay fixtures keyed by call id and input hash, under a versioned envelope (schema_version 2.1.0). Each model_call event carries tools_offered, toolset_ref and requested_tool_calls (null when not recorded); the full tool schemas live once per distinct toolset in <base>/toolsets/. The capture doubles as the fixture.See Redaction to control what those payloads contain, and Config, sinks & hygiene for where files land.
## Known limit
- +Tools run on a raw
threading.Thread(rather thanasyncio.to_thread/run_in_executor) start a fresh context and are silently uncaptured. Use the asyncio helpers so context copies into the worker.
