Methodology

How EvalShift decides whether a model swap caused a regression.

The shape of the problem

For each (prompt × example) pair, EvalShift produces two outputs: source_output (from the old model) and target_output (from the new one). Each evaluator scores both halves on a 0–1 scale, and the quantity we care about is the per-example delta:

delta_i = target_score_i − source_score_i

Negative deltas mean the target lost ground; positive deltas mean it gained. A “regression” is a slice where the deltas are systematically negative across many examples.

Per-comparison testing

For each (prompt, evaluator, slice) triple we run one paired test on the deltas (tool_selection gets two, one per axis: conformance and divergence are tested separately):

1. Sample-size guards.
   • +Rows an evaluator marked non-applicable are dropped first, and noted: a tool-only turn has no text for semantic or llm_judge to compare, and so are calls that errored upstream or were truncated at the token cap. They are excluded from the test, the slice aggregates, and the policy metrics — a broken measurement is not a zero delta.
   • +n < 5: no test runs; severity is insufficient.
   • +5 ≤ n < 20: test runs but the result carries an “uncertain” note in the report.
   • +No applicable rows left: severity is insufficient with a nothing measured note — never none. The report gives it its own headline, Nothing measured, and a run in which any blocking evaluator landed there cannot return pass: it is downgraded to conditional_pass, with those evaluators named — or, when every blocking evaluator did, is inconclusive unless a cost or latency budget was conclusively breached. An evaluator that compared nothing is unknown, not equivalent.
2. Normality screen. Shapiro-Wilk on the deltas. If the screen passes (p > 0.05), we run a paired t-test. Otherwise we fall back to a Wilcoxon signed-rank test, which only assumes symmetric distributions. (For n > 5000, Shapiro-Wilk is over-powered, so we skip the screen and use the t-test by CLT grounds.)
3. Effect size. Cohen’s d for paired samples:
   text

   d = mean(deltas) / std(deltas, ddof=1)

   Sign of d tracks the sign of mean(deltas) (negative = regression).
4. 95% CI on the effect size. Analytical (d ± 1.96 × SE) when the t-test ran; percentile bootstrap (2000 resamples) when Wilcoxon ran.

Multi-test correction

The whole point of slicing is to find regressions in some subset of the data, but every additional comparison inflates the family-wise false-discovery rate. We control it with Benjamini–Hochberg at FDR α = 0.05 across every p-value in the run — every comparison above that ran a test. Comparisons whose test was skipped keep p = 1 and stay out of the correction.

We pick BH over Bonferroni because BH controls the same error rate with substantially more power, especially when several comparisons are real (the typical case in a real eval suite).

Severity classification

After BH correction, each comparison gets a severity tag based on the adjusted p-value and the absolute effect size:

A uniform shift — every example losing exactly the same amount — has zero variance, so it reads none here with the note zero variance — no test. It is not waved through: the migration policy counts regressions per record, so on a blocking evaluator those examples still count against max_overall_regression_rate (on semantic, only when the shift takes the target below min_similarity).

semantic rows enter the test tolerance-adjusted. The source is pinned at 1.0, so a negative delta is the evaluator’s normal output, not evidence of a regression: a row within min_similarity contributes a delta of 0, a flagged row its full drop. That is the line the budgets already draw, so nine rows at 0.90–0.98 can no longer test as critical while every budget passes. delta_avg_score stays the raw mean, and the comparison notes how many rows were within the threshold.

A critical or high comparison from a blocking evaluator fails the run on its own, even when every budget holds; medium and low downgrade a pass to conditional_pass. When severity alone failed the run, the decision’s reason names each such comparison.

What we always report

Every comparison row in analysis.json and report.html carries:

• +raw p-value
• +BH-adjusted p-value
• +Cohen’s d and its 95% CI
• +sample size
• +test used (paired_t, wilcoxon, or skipped)
• +severity classification

You will never see a naked “regression detected” without the numbers backing it up.

Limitations to be aware of

• +Score scales must be comparable across (source, target). Most evaluators are inherently per-output (structural, judge), so this is fine. The semantic evaluator deliberately frames source as 1.0 to make deltas interpretable, but that means the absolute similarity is the signal — interpret with care.
• +Wilcoxon’s assumption is symmetry. If your delta distribution is asymmetric (e.g. small wins, occasional huge losses), Wilcoxon’s null can be off. The test still flags real shifts but the p-values may be approximate.
• +BH controls expected FDR, not per-test type-I error. A few false positives are expected when many tests run. Use the effect sizes and CIs to triage.
• +Per-call sampling variance. By default EvalShift makes one call per (prompt, example) per model, at temperature=0, so the only intended difference between the two arms is the model. If a model samples freely anyway (the report says so in a banner), set defaults.samples_per_example above 1. Every example is then sent to each model N times; sample i of the source is scored against sample i of the target; and the example’s row in scores.jsonl is the mean of its samples’ source_score, target_score and delta, with the per-sample values and the population delta_variance recorded under metadata.samples. The paired tests above run over those per-example rows, so n stays the number of examples: repeated sampling reduces the noise in each delta, it does not manufacture power. Read a large delta_variance next to a small mean delta as “this example is noisy”, not “this example regressed”.