// sdk · config

Config, sinks & hygiene

where captures go, how they self-manage

Configure capture two ways: environment variables (the gate and the output root) and the programmatic configure() call (sinks, sampling, dedup, GC). Everything here is fail-open — a hygiene or sink-write fault drops a capture, never the host agent — with one deliberate exception: an unusable EVALSHIFT_CAPTURE_STORE raises at startup.

Environment

bash

EVALSHIFT_CAPTURE=1                      # gate: on for 1/true/yes/on, off otherwise (default off)
EVALSHIFT_DIR=/var/captures              # capture root; default is ./.evalshift
EVALSHIFT_CAPTURE_STORE=s3://acme-evals  # ship captures to a bucket you own instead (default: local disk)
EVALSHIFT_MAX_CAPTURES=200               # newest-N cap per suite dir (default 200; 0/off = unlimited)
EVALSHIFT_CAPTURE_TTL=604800             # evict captures older than N seconds (default off)
EVALSHIFT_DEDUP=on                       # collapse identical-input captures (default on)
EVALSHIFT_SAMPLE_RATE=0.1                # capture this fraction of runs (default off = capture all)

+EVALSHIFT_CAPTURE — the off-by-default gate. Truthy values: 1 / true / yes / on.
+EVALSHIFT_DIR — the capture root. Captures land at <base>/captures/<suite>/cap_<id>.json; toolset sidecars at <base>/toolsets/<hex>.json, written once per distinct toolset.
+EVALSHIFT_CAPTURE_STORE — ship captures to an object store you own instead of local disk: s3://<bucket>/<prefix>, gs://<bucket>/<prefix> or az://<account>/<container>/<prefix>. Needs the provider’s client library; with the gate on, an unusable value raises SinkConfigurationError rather than falling back to local disk. See ObjectStoreSink.
+EVALSHIFT_MAX_CAPTURES, EVALSHIFT_CAPTURE_TTL, EVALSHIFT_DEDUP, EVALSHIFT_SAMPLE_RATE — the hygiene knobs, for hosts that cannot call configure(). For the numeric ones, 0 / none / unlimited / off means “disabled” — for EVALSHIFT_SAMPLE_RATE that means no sampling, so every run is captured (set 0.0 to capture nothing). EVALSHIFT_DEDUP turns off with off, 0, none or unlimited; false and no are not recognised and leave it on.

Precedence is configure(...) > environment variable > built-in default, and a malformed hygiene value fails open to the default. The gate and EVALSHIFT_DIR are read live (per call, per write); the hygiene variables and EVALSHIFT_CAPTURE_STORE are read once at config construction (import or reset_config()).

configure()

configure() has merge semantics — only the keyword args you pass change. reset_config() restores defaults (useful in tests). There is no redact knob — masking is required at each capture point, not set process-wide. See Redaction.

python

from evalshift import configure, MemorySink

configure(
    sink=MemorySink(),        # where captures go (default: FileSink)
    sample_rate=0.1,          # capture 10% of runs (default: capture all)
    dedup=True,               # collapse identical-input captures (default: True)
    max_captures=500,         # cap per-suite file count (default: 200)
    capture_ttl=86400,        # evict captures older than 1 day (default: off)
    require_model_call=True,  # drop captures with no model_call span (default: False)
)

Option	Effect
`sink`	Where captures are written. Default FileSink, or the object store EVALSHIFT_CAPTURE_STORE names when that is set.
`sample_rate`	Fraction of runs to capture, decided at agent entry.
`dedup`	Collapse captures with identical input (per process).
`max_captures`	Cap a suite directory by file count; evict oldest by mtime.
`capture_ttl`	Evict captures older than this many seconds.
`require_model_call`	Drop captures that recorded no model_call span (default off).

Sinks

+FileSink (default) — one JSON file per capture. The root resolves at write time: an explicit FileSink(base=...) wins, else EVALSHIFT_DIR, else .evalshift relative to the working directory. No repo-root walk — the SDK never assumes it runs inside a checkout. On a filesystem error (read-only mount, disk full) it drops the capture, logs at debug, and returns — it does not crash.
+MemorySink — keeps captures in memory for ephemeral hosts (Lambda, tests); you drain them yourself.
+ObjectStoreSink — ships captures and their toolset sidecars to an object store you own, for hosts whose disk does not outlive them (Fargate, Lambda, pods). Selected by EVALSHIFT_CAPTURE_STORE or built in code; details below.

python

from evalshift import configure, MemorySink

sink = MemorySink()
configure(sink=sink)
# ... run agents (zero disk writes) ...
captures = sink.flush()   # drain everything captured

ObjectStoreSink

Set one env var next to the gate, or build the sink yourself. Either way the captures go to the bucket you named, with your own cloud credentials — never to EvalShift.

bash

pip install boto3    # or google-cloud-storage / azure-storage-blob azure-identity

# one env var next to the gate; local disk stays the default when it is unset
EVALSHIFT_CAPTURE=1 EVALSHIFT_CAPTURE_STORE=s3://acme-evals/support-agent python agent.py

python

from evalshift import ObjectStoreSink, configure
from evalshift.stores.s3 import S3Store   # also evalshift.stores.gcs.GCSStore, .azure.AzureBlobStore

configure(sink=ObjectStoreSink(S3Store("acme-evals", "support-agent")))

# Lambda: background threads freeze between invocations, so upload inline
configure(sink=ObjectStoreSink(S3Store("acme-evals", "support-agent"), background=False))

URI form	Store	Install	Credentials (default chain)
`s3://<bucket>/<prefix>`	Amazon S3; also MinIO, Cloudflare R2, Backblaze B2 via boto3’s `AWS_ENDPOINT_URL`	`pip install boto3`	IAM task role / instance profile; `aws sso login` locally
`gs://<bucket>/<prefix>`	Google Cloud Storage	`pip install google-cloud-storage`	Workload Identity / Cloud Run service account; `gcloud auth application-default login` locally
`az://<account>/<container>/<prefix>`	Azure Blob Storage	`pip install azure-storage-blob azure-identity`	Managed Identity; `az login` locally

<prefix> is optional. Credentials never go in the URI — a value containing @ or ? is rejected, and the error does not echo it back. EVALSHIFT_SINK, the variable’s name before SDK 0.6.0, is still read when EVALSHIFT_CAPTURE_STORE is unset.

An unusable store raises; it never falls back to disk. Since SDK 0.6.0, if the variable is invalid, its client library is missing, or the store cannot be opened, the SDK raises SinkConfigurationError (exported from evalshift) while EVALSHIFT_CAPTURE is on — the first time your code asks it to capture: applying a capture.* decorator, wrapping a client, building EvalShiftCallbackHandler, or calling configure() without a sink. In a real agent that is process start. Bare import evalshift never raises, and with capture off nothing happens. If the gate turns on only after that, captures are dropped with one warning: a store you asked for is never silently replaced by local disk. The message names the variable you set and either the scheme and package to install or the accepted URI forms — never the value. An explicit configure(sink=...) overrides the variable. Building a store in code (S3Store(...)) without its library raises MissingStoreDependencyError (an ImportError) at construction.

+The keys are the local layout. Under your prefix the sink writes exactly what FileSink writes to disk — captures/<suite>/cap_<hex>.json and toolsets/<hex>.json — so the CLI mirrors the bucket into .evalshift/ and reads it unchanged once captures.store names it in evalshift.yaml.
+Toolset sidecars follow the sink. With an ObjectStoreSink they go to the same store, once per process per toolset; FileSink, MemorySink and custom sinks keep the on-disk sidecar.
+Background, bounded, fail-open. write enqueues and returns; one daemon thread uploads. The queue holds 1000 items by default and drops the newest when full. A failing upload never raises into your agent: the first failure per sink logs a warning with the store URI, later ones at debug.
+Retention is a lifecycle rule. Dedup still applies, but max_captures and capture_ttl do not — GC only ever runs after a real disk write. Bound the bucket with a lifecycle rule instead.

shutdown: flush before the process dies

The sink registers a flush with atexit and waits up to flush_timeout (10 s by default), logging one warning if uploads were still queued when it gave up. Python’s default SIGTERM handling does not run atexit, so on ECS/Fargate, Kubernetes or anything that stops you by signal, have the handler call sys.exit(0) (so atexit runs), or call evalshift.flush_captures(timeout=10) from your own shutdown hook — not from inside the signal handler itself, where it can deadlock on the sink’s lock. On Lambda, call flush_captures() before the handler returns, or pass background=False as above.

Hygiene

Capture can be a firehose, so it self-manages — and two of the three mechanisms are on by default, so a host that enables the gate and never calls configure() still cannot grow .evalshift/captures/ without limit. Whenever dedup or GC is active the sink is transparently wrapped; with both disabled your sink is used bare. Sampling never wraps it — it decides at agent entry.

+Sampling (sample_rate) — decided once at agent entry. 0.0 captures nothing; 1.0 / unset captures every run. A fault in the draw fails open (captures).
+Dedup (dedup, on by default) — per-process, best-effort collapse of identical-input captures, keyed on (suite, input_hash). The registry resets with the process.
+GC (max_captures, 200 per suite by default, and capture_ttl, off) — evicts the oldest captures by filesystem mtime once a suite exceeds the cap or ages past the TTL. GC only ever scans <base>/captures/<suite>/ — toolset sidecars under <base>/toolsets/ are never touched (the CLI prunes those, refcounted, via capture clean).

bash

# opt back out of hygiene entirely (write every capture, keep them all)
EVALSHIFT_MAX_CAPTURES=0 EVALSHIFT_DEDUP=off

One more persistence gate, off by default: configure(require_model_call=True) drops captures that recorded no model_call event. A capture with no model output carries nothing scoreable, so eval-grade hosts that only want promotable captures turn it on.

fail-open is sacred

Every SDK boundary swallows and logs its own faults — a failing sink write, redactor, sampler, dedup, or GC step drops at most one capture. The only statement not wrapped is your own function call, whose return value and exceptions always propagate. Configuration errors are the deliberate exception: a bad redact= raises TypeError where you pass it, and an unusable store raises SinkConfigurationError while capture is on.