Configuration

Every EvalShift run is driven by a single evalshift.yaml file. This page documents every field — types, defaults, and what they do.

evalshift init writes a heavily-commented starter you can edit to your needs. Below is the canonical reference.

Top-level shape

version: 1                # optional; defaults to 1, must be 1
project: org/slug         # optional; Cloud push needs it or --project
prompts: [...]            # required, at least one
defaults: {...}           # optional
evaluators: {...}         # optional (but at least one is needed for evaluate)
suites: {...}             # optional; managed block, rewritten by capture sync
captures: {...}           # optional; the object store the SDK ships captures to
migration_policy: {...}   # optional; regression budgets and the verdict
retention: {...}          # optional; how much run history to keep

project (Cloud)

Optional. Identifies which EvalShift Cloud project a run belongs to when you push. Format is org-slug/project-slug (regex ^[a-z0-9-]+/[a-z0-9-]+$). On first push, EvalShift auto-creates the project if your token has owner or admin access to the org and you didn’t pass --no-create-project.

Leave it unset for local-only use; push needs it or --project. See Getting started for the full Cloud walkthrough.

thresholds was removed

evalshift.yaml used to accept a free-form thresholds: block next to project:. It was never read by anything that gates: the migration verdict comes from migration_policy, and the Cloud gate reads the resolved policy the bundle carries. Nothing replaced it. Since CLI 1.1.0 a config that still sets the key fails to load, and push no longer syncs it. Delete the block; if you were using it to express a gate, encode that as a migration_policy budget instead — migration_policy is the only gating block.

top-level slices was removed

evalshift.yaml also used to accept a top-level slices: list (name, filter, applies_to). It was validated and recorded in the run bundle, but analysis never read it: slices come from example tags (see Slices), so none of its fields renamed, filtered or scoped anything. Since CLI 1.2.0 a config that still sets the key fails to load, with an error saying so. Delete the block — a run reports the same slices without it — and put per-slice budgets under migration_policy.slices, keyed by tag.

Cloud baselines are unaffected for any config that never set the key (or set slices: []): the run’s config hash does not move. Deleting a non-empty block does change that hash, so runs pushed afterwards are not comparable to earlier baselines until the base branch pushes a run with the edited config.

prompts

A list of prompt definitions. Each entry has:

Two prompt-detection modes

manual — write the prompt body inline:

- id: greet
  detection: manual
  content: "Hello {name}"
  variables: [name]

python_string — point at an existing module-level string in your codebase:

- id: greet
  detection: python_string
  path: src/prompts/greet.py
  variable: GREET_PROMPT
  variables: [name]

EvalShift AST-walks the file and extracts the string literal. It does not run user code. F-strings, concatenations, .format() calls, and other dynamic forms are explicitly rejected.

defaults

evaluators

Eight sub-keys, all optional: structural, semantic, llm_judge, tool_selection, tool_arguments, tool_trace_structure, agent_trace, and trace_invariants. At least one evaluator must be configured for evalshift evaluate to do anything.

evaluators.structural

A list. Each entry has a type and the fields that type needs.

Optional applies_to: ["prompt-id-glob", ...] (default ["*"]) for future per-prompt scoping.

evaluators.semantic

A single object (not a list).

When both outputs are empty — a turn that ended in tool calls — the evaluator writes no record and makes no provider call: nothing was measured, so nothing is scored. One empty side still scores — 0.0 similarity by definition, with no embedding call, gated by min_similarity as usual — because a silent target is a real regression.

The semantic evaluator scores the target’s similarity to the source: target_score = cosine(source, target), source_score = 1.0. A negative delta means the target drifted from the source’s meaning.

evaluators.tool_selection

The two axes answer different questions — “did each model do what production did?” and “did the candidate change behaviour?” — and render as separate, separately-labelled rows in report.html. (The old single mode: key was removed; a stale one fails validation.)

evaluators.tool_arguments

evaluators.tool_trace_structure

evaluators.agent_trace

A list. These evaluators score timelines imported with evalshift traces import; they do not run your agent and do not replace the normal raw.jsonl model-call artifact.

evaluators.trace_invariants

A list of rule sets: hand-written, deterministic rules over the tool calls in each trace, checked on both sides of every pair. Put them in the top-level evaluators: block, outside the region capture sync manages, or in a suite’s own evaluators override. Why and when to write one is on Evaluators. Since CLI 1.3.0 the CLI reads the block; an older CLI fails to load a config that sets it.

evaluators:
  trace_invariants:
    - name: payments_contract
      owner: "@payments-team"     # optional; echoed onto every record and the report
      applies_to: ["checkout-*"]  # prompt-id globs; default ["*"]
      blocking: true              # default; false = reported, never gates
      traces: replayed            # default; `imported` checks `evalshift traces import` traces
      rules:
        - id: auth-before-charge
          type: order
          before: authenticate
          after: charge_card
        - id: no-legacy-refund
          type: forbidden
          tools: [refund_v1]
        - id: one-charge
          type: call_count
          tool: charge_card
          max_calls: 1              # min_calls, max_calls or both; both equal = exact
        - id: must-book
          type: required
          tools: [create_booking]
        - id: sane-amount
          type: arguments
          tool: charge_card
          json_schema:
            type: object
            required: [amount]
            properties:
              amount: {type: number, minimum: 0.01, maximum: 5000}

migration_policy:
  max_invariant_violations: 0     # default; examples where the target broke any blocking rule

The unit of checking is a response: one model turn, together with the tool calls already in its context before it answered. With traces: replayed there is one response per replayed round, and round k’s context is the tool calls on the example’s history assistant turns plus the recorded calls of rounds 0..k-1 that the replay fed back. The model’s own earlier rounds are not context, because a teacher-forced replay never feeds them back. With traces: imported the whole trace is one response with an empty context: every call in it was the agent’s own.

Context calls can never be violations — the model under test did not make them — but they can satisfy a rule or count toward it:

“Earlier in the same response” means emitted earlier. Most harnesses execute one response’s calls in order, and imported traces are ordered by sequence_index, so the two modes agree.

required and min_calls are not vacuous. order, arguments and a call_count with only max_calls say nothing about a trace that never calls the tool. required and call_count with min_calls fail on every in-scope trace that lacks the calls. On a single-shot replay they can only see the first response, so scope them with applies_to, or use them with a suite promoted by capture sync --rounds all or imported traces.

required asks about the model, call_count about the conversation. required asks whether the model under test made the call; call_count asks whether the conversation contains it. A recorded history that already called charge_card satisfies min_calls: 1 and leaves max_calls: 1 no room.

Teacher-forced rounds see recorded context. min_calls is judged on the last response’s context plus that response’s own calls. The model’s own earlier rounds were never fed back, so they do not count.

Malformed arguments fail a schema that requires a field. Arguments that did not parse are checked as recorded ({"_parse_error": true}), with no special case, so the field the schema requires is missing.

Each side’s score is the share of rules it broke zero times, (rules − rules_broken) / rules, and delta = target − source. The record’s kind is trace_invariants; its metadata carries rules_checked, source_violations and target_violations (each {rule_id, rule_type, tool, round_index, detail}), owner when set, and failure_categories: ["INVARIANT_VIOLATION"] when the target broke anything. A row where both sides broke a rule still counts against the target, and adds a recommendation line: either both models break the rule, or the rule is stale and needs review with its owner. The HTML report’s Trace rules broken panel lists every rule the target broke whatever the delta, with its owner, whether the source broke it too, and advisory rows tagged “advisory”; report.json carries the same rows top-level as invariant_violations (advisory included, flagged by blocking), and Top regression causes counts INVARIANT_VIOLATION as distinct examples with a blocking target violation — the same number the budget sees. Blocking rule sets gate through max_invariant_violations.

evaluators.llm_judge

A list of pairwise judges. Each entry has:

Judge family. LLM judges tend to prefer output from their own relatives (self-preference bias), and nothing in the scoring can remove that. When a judge_model resolves to the same provider as defaults.source_model or defaults.target_model, evalshift doctor prints a warn-level judge family row and evalshift validate a matching line — never a failure, because init deliberately scaffolds a same-provider judge so a first run needs one API key. The report repeats the note above the verdict whenever a judge that actually contributed llm_judge rows shares a family with an arm, and report.json carries it as judge_family_overlap. “Family” is the provider the model id resolves to (anthropic, deepseek, google, openai); ids the registry cannot place never match. defaults.judge_model is exempt: it only seeds the run-narrative model, never a pairwise verdict.

The judge sees both outputs (with random A/B order to defang positional bias) and produces strict-JSON {"winner": "A"|"B"|"tie", "reason": "..."}. Target wins → (0.0, 1.0); tie → (0.5, 0.5); source wins → (1.0, 0.0). Malformed responses degrade to (0.5, 0.5) with the error preserved.

Slices

Slices come from example tags automatically; there is nothing to configure. Every distinct tag in the suite becomes a slice under its own name, plus the implicit "all" slice, and each is analysed separately. Per-slice budgets go under migration_policy.slices, keyed by the tag. There is no top-level slices: key — it was removed.

overall is reserved and cannot be used as an example tag or as a migration_policy.slices key. It names the run-level scope in the run bundle, so a slice by that name would shadow the whole-run numbers wherever the two are rendered together. Both spellings are rejected when the config or suite loads.

Tag-derived slices with identical membership are collapsed to one before analysis, so a duplicate never enters the Benjamini–Hochberg family twice; all and any slice named under migration_policy.slices always survive.

suites

A managed block mapping a suite name to the file behind it, so evalshift run --suite-name <name> can select one without a path. evalshift capture sync rewrites everything between its markers except a suite’s hand-written trace_invariants family — edit around them, not inside.

# >>> evalshift suites (managed by `evalshift capture sync`) >>>
suites:
  support_agent:
    source: captured
    path: .evalshift/suites/support_agent/golden.jsonl
# <<< evalshift suites <<<

capture sync regenerates a managed entry’s path and evaluators from what the suite’s captures contain. A suite whose rows were offered a toolset gets a tool_selection evaluator named routing (conformance: expected, divergence: set), plus a tool_arguments evaluator named routing_args (against: expected) when rows recorded arguments; a tool-free suite gets no block and inherits the top level; a tool evaluator it inherits (including trace_invariants) skips every pair that was offered no tools, and evaluate warns that it checked nothing. The per-suite block is only used when the suite is selected by name (--suite-name); a raw --suite <path> resolves to the top-level block.

Regenerating a managed entry keeps a hand-written trace_invariants family in its evaluators block.

captures

Optional. Where the capture SDK’s recordings live when they are not on this machine. Omit the block for the default: captures are read from .evalshift/captures/ (or EVALSHIFT_DIR) on local disk and nothing here is needed.

Production agents on Fargate, Lambda or Kubernetes lose their disk when they stop, so the SDK can ship captures to an object store you own instead (EVALSHIFT_CAPTURE_STORE=<uri> in the agent’s environment; the older name EVALSHIFT_SINK still works — see Config, sinks & hygiene). Name the same store here and the CLI pulls new captures down before it promotes them, with no flags to remember:

captures:
  store: s3://acme-evals/support-agent   # or gs://<bucket>/<prefix>, az://<account>/<container>/<prefix>

Accepted URI forms — the same grammar the SDK uses:

<prefix> is optional. Credentials never go in the URI: a value containing @ or ? fails to load, naming the accepted forms, and a rejected value is never echoed back (it may be a pasted key or connection string) — the error names at most its scheme. Under the prefix the layout is exactly the local one — captures/<suite>/cap_<hex>.json and toolsets/<hex>.json — so the bucket is a mirror of .evalshift/, not a different format.

What the fetch does: it lists the bucket, downloads only objects missing locally (captures are immutable and toolsets content-addressed, so “exists” means “current”), skips captures already promoted into a suite (so capture clean never causes a re-download), and writes each file atomically. --since 24h (or 7d, 30m, an ISO date) limits a fetch to recently written captures; --offline skips it and works with what is already local. Retention in the bucket is yours to set with a lifecycle rule; the SDK’s max_captures does not apply there. Reads use your own cloud credentials; nothing from the bucket goes to EvalShift Cloud. See capture fetch.

migration_policy

Optional. Turns the statistics into a decision: analyze writes migration_decision.json with a verdict, and --policy-gate exits non-zero when that verdict is fail or conditional_pass (or no policy is configured). Rate budgets are fractions, not percents; max_critical_regressions and max_invariant_violations are counts.

evalshift init --profile writes pre-tuned budgets for a migration shape: model-upgrade (the default), cost-reduction, local-model, quantization, provider-switch. Since CLI 1.3.0 every profile block includes max_invariant_violations: 0, which older CLIs reject — bump the GitHub Action’s evalshift-version pin to 1.3.0 or later before committing it. The rate budgets carry a 95% Wilson interval, so a breach only fails when the sample can confirm it — a thin suite returns inconclusive instead of a confident fail, while a budget the observation held stays conclusive however wide its interval. See Migration policy.

retention

Optional. Run history under .evalshift/runs/ is pruned per suite after every completed run: max_runs_per_suite (default 20; 0 disables) keeps the newest N, and optional run_ttl_days evicts anything older. In-progress runs and the run just finished are never pruned. Override the count with EVALSHIFT_MAX_RUNS, or prune on demand with evalshift runs clean.

Suite (golden.jsonl) shape

The suite is JSON Lines — one example per non-blank line. Every example must also carry a toolset — exactly one of toolset_ref or inline tools. The core fields are below; the toolset, the agent ground truth (expected_tools, expected_tool_rounds, …), generation_config, and the multi-turn fields are documented in full on Golden suite.

Unknown keys are rejected (typos fail fast).

Want to see how all of this plays together for an agent migration? Read the agents page.