Every EvalShift run is driven by a single evalshift.yaml file. This page documents every field — types, defaults, and what they do.
evalshift init writes a heavily-commented starter you can edit to your needs. Below is the canonical reference.
## Top-level shape
version: 1 # required, must be 1
project: org/slug # optional; required for Cloud push
thresholds: {...} # optional; synced to EvalShift Cloud when project is set
prompts: [...] # required, at least one
defaults: {...} # optional
evaluators: {...} # optional (but at least one is needed for evaluate)
slices: [...] # optional
suites: {...} # optional; managed block, rewritten by capture sync
migration_policy: {...} # optional; regression budgets and the verdict
retention: {...} # optional; how much run history to keepextra: forbid everywhere) so typos fail fast instead of silently dropping.## project (Cloud)
Optional. Identifies which EvalShift Cloud project a run belongs to when you push. Format is org-slug/project-slug (regex ^[a-z0-9-]+/[a-z0-9-]+$). On first push, EvalShift auto-creates the project if your token has owner access to the org and you didn’t pass --no-create-project.
Leave it unset for local-only use; push won’t succeed without it. See Getting started for the full Cloud walkthrough.
## thresholds (Cloud)
Optional. A free-form key/value block that travels with the run on push. EvalShift Cloud treats these as the canonical thresholds for the project: if the pusher has write permission on the project they replace the server-side copy; otherwise the server’s canonical thresholds are returned and the CLI prints a drift warning so you notice the divergence.
project: acme/model-migration thresholds: pass_rate_min: 0.95 regression_count_max: 0
The GitHub Action’s fail-on: regression mode and the evalshift/regression commit status read these thresholds when computing pass/fail on a PR.
## prompts
A list of prompt definitions. Each entry has:
| Field | Type | Required | Description |
|---|---|---|---|
id | string | yes | Stable identifier surfaced in reports. Must be unique within the file. |
detection | enum | yes | manual or python_string. |
content | string | when detection: manual | Inline prompt body. Forbidden when detection: python_string. |
path | string | when detection: python_string | Relative or absolute path to a .py file. Resolved against the directory containing evalshift.yaml. |
variable | string | when detection: python_string | Module-level variable name holding the prompt string. |
variables | list | optional | Names of {template} placeholders the prompt expects. Used by the pre-flight compatibility check. |
max_tokens | int | optional | Per-prompt override of defaults.max_tokens. |
### Two prompt-detection modes
manual — write the prompt body inline:
- id: greet
detection: manual
content: "Hello {name}"
variables: [name]python_string — point at an existing module-level string in your codebase:
- id: greet detection: python_string path: src/prompts/greet.py variable: GREET_PROMPT variables: [name]
EvalShift AST-walks the file and extracts the string literal. It does not run user code. F-strings, concatenations, .format() calls, and other dynamic forms are explicitly rejected.
toolset_ref or inline tools), not from the prompt — so one prompt can dispatch some examples plainly and others with tools in the same run. See Agent migrations.## defaults
| Field | Type | Default | Description |
|---|---|---|---|
source_model | string | (none) | Default --from model id (or alias). |
target_model | string | (none) | Default --to model id (or alias). |
judge_model | string | gemini-3.1-flash-lite-preview | Default LLM-as-judge model. Note that evaluators.llm_judge[*].judge_model has its own default and does not read this field. |
insights_model | string | (falls back to judge_model) | Model that writes the run narrative rendered at the top of report.html. |
concurrency | int | 10 (1 ≤ x ≤ 64) | Max in-flight LLM calls — applies to run and evaluate. |
max_tokens | int | 4096 | Completion cap on every call. A call cut off at the cap is marked truncated and excluded from the statistics — a truncated output is a broken measurement, not a model verdict. |
cache | bool | true | Read/write the local SQLite cache at ~/.evalshift/cache.db. |
max_cost_usd | float | 50.0 | Soft ceiling reserved for future enforcement. The pre-flight cost prompt currently triggers above $10 (skip with --yes). |
samples_per_example | int | 1 (1 ≤ x ≤ 20) | How many times each (prompt, example) is sent to each model. Above 1, every sample is its own live call (raw.jsonl rows carry sample_index, and the cache keys on it), sample i of the source is scored against sample i of the target, and the example’s scores.jsonl row becomes the mean source_score, target_score and delta over its samples, with the per-sample lists and the population delta_variance under metadata.samples. The paired tests still run over examples, so n is unchanged. Cost and the pre-flight call count multiply by it. See Methodology. |
## evaluators
Seven sub-keys, all optional: structural, semantic, llm_judge, tool_selection, tool_arguments, tool_trace_structure, and agent_trace. At least one evaluator must be configured for evalshift evaluate to do anything.
blocking: bool (default true). Advisory evaluators (blocking: false, what a fresh evalshift init writes) still score and still appear in reports, but never move the migration verdict.### evaluators.structural
A list. Each entry has a type and the fields that type needs.
| type | Required fields | Behaviour |
|---|---|---|
json_schema | schema_path | Output is parsed as JSON; score 1.0 if it validates against the schema, 0.0 otherwise. |
regex | pattern | Score 1.0 if the regex matches anywhere in the output, 0.0 otherwise. |
length | min_chars and/or max_chars | Score 1.0 inside the bounds, distance-decayed outside. |
Optional applies_to: ["prompt-id-glob", ...] (default ["*"]) for future per-prompt scoping.
### evaluators.semantic
A single object (not a list).
| Field | Type | Default | Description |
|---|---|---|---|
embedding_model | string | text-embedding-3-small | LiteLLM-compatible embedding model id. Use a Gemini one (e.g. gemini/text-embedding-004) if you don’t have an OpenAI key. |
min_similarity | float | 0.9 | Cosine similarity below which the target is flagged SEMANTIC_REGRESSION. Raise to 1.0 to flag any deviation from byte-identical. |
When both outputs are empty — a turn that ended in tool calls — the row is marked skipped (1.0/1.0, no provider call) and excluded from the statistics rather than counted as a perfect match. One empty side still scores: a silent target is a real regression.
The semantic evaluator scores the target’s similarity to the source: target_score = cosine(source, target), source_score = 1.0. A negative delta means the target drifted from the source’s meaning.
### evaluators.tool_selection
| Field | Type | Default | Description |
|---|---|---|---|
name | string | (required) | Identifier surfaced in reports. |
conformance | enum | expected | Grades each side absolutely against the example’s ground truth (one record, kind: tool_selection.conformance). expected matches example.expected_tools in order; expected_set is the same, order-insensitive multiset recall — for parallel fan-outs; off disables the axis. A row both sides missed is tagged TOOL_GROUND_TRUTH_MISS and excluded from the policy rates. |
divergence | enum | set | Grades the target against the source (one record, kind: tool_selection.divergence; the source is its own baseline at 1.0). set is Jaccard on tool names — reordered identical calls don’t read as drift; exact is sequence equality; first compares the first call only; off disables the axis. Both axes off is a config error. |
applies_to | list | ["*"] | Glob list of prompt ids. |
severity_floor | enum | null | low | medium | high | critical — a regression on either axis is never classified below the floor. |
The two axes answer different questions — “did each model do what production did?” and “did the candidate change behaviour?” — and render as separate, separately-labelled rows in report.html. (The old single mode: key was removed; a stale one fails validation.)
### evaluators.tool_arguments
| Field | Type | Default | Description |
|---|---|---|---|
name | string | (required) | Identifier. |
applies_to | list | ["*"] | Glob list. |
strategies | dict | {} | Per-field strategy overrides (exact/subset/numeric/semantic). Keys are bare field names, matched across every tool. semantic borrows the embedding model from evaluators.semantic; without one it degrades to exact. |
numeric_tolerance | float | 0.05 | Relative-error tolerance for numeric. |
against | enum | source | source scores drift from the current model, which pins its own score at 1.0 by construction — it measures change, not correctness. expected scores both models against expected_tools[].arguments; each expectation’s match_strategy picks the compared keys, an expected call the model never made scores 0, and an example with no recorded arguments is skipped. |
optional_fields_scored | enum | lenient | How a field present on one side only is scored: lenient gives it 0.5 (omitting an optional parameter is a difference, not a wrong value), strict gives it 0.0. |
use_llm_judge_fallback | bool | false | Reserved; currently ignored. |
### evaluators.tool_trace_structure
| Field | Type | Default | Description |
|---|---|---|---|
name | string | (required) | Identifier. |
applies_to | list | ["*"] | Glob list. |
check_call_count | bool | true | Score the number of tool calls. |
check_parallelism | bool | true | Score parallel-vs-sequential alignment. |
check_refusals | bool | true | Score refusal alignment; mismatches force severity_floor: high. |
call_count_tolerance | int | 1 | +/- N calls considered equivalent. |
### evaluators.llm_judge
A list of pairwise judges. Each entry has:
| Field | Type | Required | Description |
|---|---|---|---|
criterion_name | string | yes | Short id surfaced in reports. |
criterion_prompt | string | yes | Free-form criterion the judge applies (e.g. "which output preserves more factual detail?"). |
judge_model | string | optional | Model used as the judge. Prefer a judge from a third model family so it isn't grading its own relatives. |
Judge family. LLM judges tend to prefer output from their own relatives (self-preference bias), and nothing in the scoring can remove that. When a judge_model resolves to the same provider as defaults.source_model or defaults.target_model, evalshift doctor prints a warn-level judge family row and evalshift validate a matching line — never a failure, because init deliberately scaffolds a same-provider judge so a first run needs one API key. The report repeats the note above the verdict whenever a judge that actually contributed llm_judge rows shares a family with an arm, and report.json carries it as judge_family_overlap. “Family” is the provider the model id resolves to (anthropic, openai, google); ids the registry cannot place never match. defaults.judge_model is exempt: it only seeds the run-narrative model, never a pairwise verdict.
The judge sees both outputs (with random A/B order to defang positional bias) and produces strict-JSON {"winner": "A"|"B"|"tie", "reason": "..."}. Target wins → (0.0, 1.0); tie → (0.5, 0.5); source wins → (1.0, 0.0). Malformed responses degrade to (0.5, 0.5) with the error preserved.
## slices
A list of named subsets used for slice-level statistical analysis. The implicit "all" slice always exists.
| Field | Type | Required | Description |
|---|---|---|---|
name | string | yes | Slice name surfaced in reports. |
filter | string | yes | A tag string. The filter is a literal tag — examples whose tags list contains the value land in this slice. Slices covering exactly the same examples are collapsed before analysis, so a duplicate never enters the Benjamini–Hochberg family twice. |
applies_to | list | optional | Glob list of prompt ids this slice applies to (default ["*"]). |
## suites
A managed block mapping a suite name to the file behind it, so evalshift run --suite-name <name> can select one without a path. evalshift capture sync rewrites everything between its markers — edit around them, not inside.
# >>> evalshift suites (managed by `evalshift capture sync`) >>>
suites:
support_agent:
source: captured
path: .evalshift/suites/support_agent/golden.jsonl
# <<< evalshift suites <<<Each entry takes source (captured or jsonl) and path.
## migration_policy
Optional. Turns the statistics into a decision: analyze writes migration_decision.json with a verdict, and --policy-gate exits non-zero on it. Budgets are fractions, not percents.
| Field | Default | Meaning |
|---|---|---|
max_overall_regression_rate | 0.30 | Share of blocking records allowed to regress. |
max_critical_regressions | 1 | Count of critical-severity regressions. An integer, not a ratio. |
min_equivalence_rate | 0.75 | Floor on the non-regression rate — equivalent or improved both count. |
max_tool_argument_drift | 0.20 | Share of tool-argument rows allowed to drift. |
max_tool_divergence | 0.20 | Share of tool_selection.divergence rows where the target called different tools than the source. |
tool_argument_drift_floor | 0.9 | Target argument score below which a call counts as drifted. Argument scores are continuous — without a floor every non-identical call counts and the drift budget is unreachable. |
max_cost_increase | 0.30 | Relative increase in mean per-call cost, target vs source. |
max_latency_increase | 0.30 | Relative increase in mean per-call latency. |
fail_on_dropped_params | false | Fail the verdict when state.json → dropped_params is non-empty — an arm could not honour a generation parameter the source capture recorded (the report shows a Constraints not honoured banner either way). Top level only: a model either accepts a parameter or does not, which no slice can vary. Runs recorded before this field existed are never failed by it. |
slices | {} | Per-slice overrides; unset fields inherit the top level. A slice budget blocks on the same terms as an overall one — breach it conclusively and the run fails. |
evalshift init --profile writes pre-tuned budgets for a migration shape: model-upgrade (the default), cost-reduction, local-model, quantization, provider-switch. The rate budgets carry a 95% Wilson interval, so a breach only fails when the sample can confirm it — a thin suite returns inconclusive instead of a confident fail, while a budget the observation held stays conclusive however wide its interval. See Migration policy.
## retention
Optional. Run history under .evalshift/runs/ is pruned per suite after every completed run: max_runs_per_suite (default 20; 0 disables) keeps the newest N, and optional run_ttl_days evicts anything older. In-progress runs and the run just finished are never pruned. Override the count with EVALSHIFT_MAX_RUNS, or prune on demand with evalshift runs clean.
## Suite (golden.jsonl) shape
The suite is JSON Lines — one example per non-blank line. Every example must also carry a toolset — exactly one of toolset_ref or inline tools. The core fields are below; the toolset, the agent ground truth (expected_tools, expected_tool_rounds, …), generation_config, and the multi-turn fields are documented in full on Golden suite.
| Field | Type | Required | Description |
|---|---|---|---|
id | string | yes | Unique within the suite. |
inputs | object | yes | Mapping of template-variable name to value. |
tags | list | optional | Slice tags. |
expected | object | optional | Reference output (unused by most evaluators). |
Unknown keys are rejected (typos fail fast).
Want to see how all of this plays together for an agent migration? Read the agents page.