Evaluators

An evaluator scores the (source_output, target_output) pair for one example. Every evaluator returns a PairedScore with both halves and a delta = target_score − source_score. Negative deltas mean the target regressed; positive deltas mean it improved.

EvalShift ships eight evaluator keys — structural, semantic, llm_judge, tool_selection, tool_arguments, tool_trace_structure, agent_trace and trace_invariants — grouped here by what they measure:

Structural (deterministic, free)

Fast checks that depend only on the output shape — no API calls.

• +json_schema — does the output parse as JSON and validate against a schema you provide? 1.0 / 0.0.
• +regex — does the output match a pattern? 1.0 / 0.0.
• +length — is the output within [min_chars, max_chars]? 1.0 inside, distance-decayed outside.

Use these whenever you can — they’re free, deterministic, and cover most real regressions.

Semantic (cheap)

• +semantic.cosine — embed both outputs with a configurable embedding model, compute cosine similarity, then frame the result as a target-preservation score: source = 1.0, target = similarity. Delta < 0 means the target drifted in meaning from the source; a target below min_similarity (default 0.9) is what counts as a regression in the policy rates.

Drift inside min_similarity is no change. Because the source is pinned at 1.0, any rewording is a small negative delta, so a row within the threshold counts as equivalent in the budgets and enters the paired significance test as a delta of 0; only a row below it carries its full drop into severity. delta_avg_score still shows the raw drift. See methodology.

Formatting is not drift either. Both outputs are normalized before embedding: a code fence wrapping the whole output is dropped, and a JSON object or array is re-serialized canonically (sorted keys), so a fenced, pretty-printed object and the same object returned compact score 1.0. A fence inside prose is part of the answer and kept.

The default embedding_model is text-embedding-3-small, which is OpenAI’s and needs OPENAI_API_KEY. evalshift init writes this evaluator active but advisory for every provider; Anthropic and DeepSeek have no embeddings endpoint, so it borrows OpenAI’s or Gemini’s, whichever key you already have. What happens when the key is missing: see Blocking vs advisory.

Use when:

• +You don’t have a clean structural check.
• +You want to detect “wandered off” outputs that still look fine syntactically.

Don’t use when:

• +The target is intentionally meant to differ from the source (e.g. you’re migrating from a verbose model to a terse one). The similarity will look low and you’ll get a confusing “regression” signal.

LLM-as-judge (most expensive)

• +llm_judge.<criterion> — ask a strong model “which output better satisfies this criterion?” with random A/B ordering to reduce positional bias. Verdict maps cleanly to (source, target) scores.

Use when you can articulate the difference you care about as a sentence (“which output preserves more factual detail?”). Multiple llm_judge entries are allowed — each becomes its own evaluator. Tool-only turns (both outputs empty) are skipped without spending a judge call.

Pick the judge from a third model family. A judge prefers output that reads like its own (self-preference bias), and A/B randomisation does nothing against that. doctor and validate warn when an llm_judge judge_model resolves to the same provider as defaults.source_model or target_model; the report repeats the note above the verdict for judges that actually contributed rows, and report.json carries it as judge_family_overlap. Advisory only, never a failure — init scaffolds a same-provider judge on purpose so a first run needs only your provider’s key (on Anthropic and DeepSeek, semantic borrows another provider’s embeddings and is skipped until that key is set — see Blocking vs advisory). defaults.judge_model is exempt: it only seeds the run-insights model, never a pairwise verdict.

Tool-call evaluators (agent migrations)

For examples whose toolset is non-empty (toolset_ref or inline tools), EvalShift parses each model’s response into a provider-agnostic ToolTrace and scores three orthogonal dimensions:

• +tool_selection — which tools fire? Two independent axes, one record each. conformance grades each side absolutely against the example’s ground truth: expected (default; expected_tools matched in order), expected_set (order-insensitive multiset recall, for parallel fan-outs), off. divergence grades the target against the source: set (default; Jaccard on tool names, so reordered identical calls don’t read as drift), exact, first, off. A conformance row both sides missed is tagged TOOL_GROUND_TRUTH_MISS — a broken harness, not a migration finding — and leaves the policy rates. Configure severity_floor: high so a regression here can never be downgraded.
• +tool_arguments — what did the model pass? Greedy match by (tool_name, sequence_index), then per-field strategies (exact / subset / numeric / semantic / auto). Fields no strategy names use default_strategy, which defaults to auto: normalized exact first, then the field’s schema type, then graded similarity for free text. Use when arg drift matters (e.g. the model still calls issue_refund but the amount is wrong). against: source (default) measures drift from the current model, which pins its own score at 1.0 by construction; against: expected scores both models against the recorded ground truth instead.
• +tool_trace_structure — how did it sequence them? Sub-scores: call count, parallelism, refusal alignment, expected count. Refusal mismatches force severity_floor: high.

Per-round scoring. When an example carries tool_result_fixtures (a suite promoted with --rounds all), run replays it teacher-forced and each of the three scores per round: tool_selection conformance grades round k against expected_tool_rounds[k] (“called nothing” for the answer round) and divergence compares round k of the target to round k of the source; tool_arguments pairs calls within a round, so a right call in the wrong round is a miss; tool_trace_structure counts calls over the whole trace and compares parallelism round by round, recording details.rounds_replayed. Each record is the mean over replayed rounds with the per-round names and scores under metadata.rounds, so max_tool_divergence counts an example as diverged when any round diverged; a round with no ground truth in which neither side called anything does not enter the mean. Single-shot pairs score exactly as before and carry no rounds key. See Agent rounds.

Each agent failure mode maps to one of the three: dropped tool / wrong tool → tool_selection; argument drift → tool_arguments; parallel↔serial flip, loop divergence, refusal regression → tool_trace_structure. Start with tool_selection; turn the others on once you’ve decided you care.

Imported agent traces

If your agent runs its loop outside EvalShift — LangChain, a custom orchestrator, another language — attach the timelines to a completed run with evalshift traces import and score them with the agent_trace evaluator: tool-order similarity (LCS normalised), per-field argument equality, and missing-verification checks, with verification_tools and dangerous_tools naming the calls that matter. Extra dangerous calls on the target flag DANGEROUS_ACTION_DRIFT; a dangerous call with no preceding verification flags MISSING_VERIFICATION_STEP. Or hold both sides to rules of your own with trace_invariants and traces: imported.

Trace invariants

The relative tool checks — divergence, argument drift with against: source, and dangerous_tools — treat “the source did it” as correct: they measure the target against the source. trace_invariants judges both sides against rules your team wrote — forbidden, required, order, call_count and arguments — on replayed traces or, with traces: imported, on traces attached with evalshift traces import. Each side scores the share of rules it broke zero times, and a rule the target breaks counts against it whatever the source did. On a blocking rule set those breaches feed the max_invariant_violations budget, which fails the run even when the suite is too small for statistics.

Write rules only for invariants where a silent miss is expensive: authenticate before a write, never call a deprecated endpoint, charge at most once, keep an amount inside its bounds. Production traces remain the broad corpus.

Keep the recording and the rules apart. Captured expected_tools are the recording, and conformance: expected grades their order as recorded. An ordering that matters belongs in an order rule, not in the recording. The fields, rule types and their exact semantics are in evaluators.trace_invariants.

Blocking vs advisory

Every evaluator takes blocking: bool (default true). Advisory evaluators still score and still appear in reports, but never move the migration verdict — which is what a fresh evalshift init writes for semantic and llm_judge, so a new suite cannot fail a build on evaluators nobody has calibrated yet. If every evaluator is advisory the verdict is inconclusive, with the exception of the cost and latency budgets: those read the run’s calls rather than evaluator records, so a conclusive breach still fails.

When nothing gates the verdict, the recommendations say whether each advisory judge is ready to gate: once it has scored at least 20 pairs on every prompt they tell you to set blocking: true on it; until then they name the thinnest prompt’s count. A judge that measured nothing on a prompt (every call errored, say) is told so instead, since more examples would not help. semantic is never suggested — it measures drift, not correctness.

Missing API keys. semantic and llm_judge are the evaluators that call a model of their own. Since CLI 1.4.0, compare, run and evaluate check each one’s API key before the first model call — including the defaults a hand-written entry falls back to: text-embedding-3-small (OpenAI) for semantic, and gemini-3.1-flash-lite-preview for a judge with no judge_model.

• +An advisory entry without a key is skipped, with a line naming the env var to export. The run records it in state.json as skipped_evaluators and repeats it in the verdict’s recommendations, the report and the bundle. A skipped semantic also degrades tool_arguments: free-text arguments are compared with difflib instead of embeddings, and semantic-strategy fields fall back to exact matching.
• +A blocking entry — any semantic or llm_judge that does not say blocking: false — stops the run with exit 1. So does a semantic marked advisory whose embedder a blocking tool_arguments entry uses for a semantic-strategy field; the error names that entry.
• +A config whose every evaluator would be skipped exits 1 as well: nothing would score the run.
• +doctor gives the same answer in its evaluator keys row — fail for a blocking gap, warn for an advisory one.

Before CLI 1.4.0, a keyless judge or embedder failed on every pair, the run landed on inconclusive and passed --policy-gate, so a CI job that was green can turn red on upgrade. Export the key, or set blocking: false to have the evaluator skipped with a warning. A model on a provider EvalShift does not recognise is not checked.

A turn that ends in tool calls has no text. semantic and llm_judge mark such rows skipped rather than scoring them a 0.5/0.5 tie, and skipped rows are excluded from the paired test, the slice aggregates and the policy metrics. A comparison with nothing left reports as unknown (“Nothing measured”), never as equivalent.

Mixing evaluators

You can configure several at once. They all run on every (prompt, example) pair (trace_invariants only on prompts its applies_to matches), and the analysis layer treats each as a separate comparison (so BH correction adjusts for the multiple-test count correctly).

A typical migration uses:

• +1–2 structural evaluators (cheap baseline checks)
• +1 semantic.cosine (catches semantic drift)
• +1 llm_judge per criterion the team cares about

Cost considerations

Per (prompt, example) pair, each evaluator means:

A 100-example suite with 1 prompt and 4 evaluators (2 structural + 1 semantic + 1 judge) is:

• +Run: 200 model calls (100 × 2 models)
• +Evaluate: 200 embedding calls + 100 judge calls

LiteLLM’s pricing data drives the pre-flight estimate; the local SQLite cache absorbs identical re-runs, examples with a toolset included, and the embedding and judge calls are cached too.

Want the statistical machinery behind delta-aggregation? See methodology.