How many eval cases do you need?

Everyone asks the question the same way: how many examples does a golden suite need? Fifty? Two hundred? The number that actually decides whether your run can say anything is not the size of the suite. It is n inside one comparison — and the suite is chopped into comparisons before a single statistic is computed.

This is where most first suites disappoint. Two hundred examples feel serious, come back inconclusive, and nobody can explain why. Here is how n is counted, what is dropped before counting, and how to size a suite so the verdict is earned rather than lucky.

## n is per comparison, not per suite

The analysis groups paired deltas — target_score - source_score, per example — by the triple (prompt_id, evaluator_name, slice_name). Every one of those groups is tested on its own, and every one has its own n.

So a 200-example suite run across 3 prompts, scored by 4 evaluators, sliced 3 ways is not one comparison with n=200. It is up to 36 comparisons, each drawing on the rows that match its slice. Give the smallest slice 15 examples and that column of the matrix is stuck at n=15 no matter how impressive the suite total looks.

Three constants decide what happens next:

• +MIN_N_FOR_TEST = 5 — below five paired observations the comparison is skipped entirely, with severity insufficient. It is not scored as "no change"; it is scored as "not measured".
• +MIN_N_RELIABLE = 20 — between 5 and 20 the comparison is tested but flagged uncertain.
• +Zero variance (std < 1e-9) — skipped with severity none, because every delta was identical.

Twenty per group is the line where a result stops carrying an asterisk. Work backwards from the groups you care about, not forwards from a round suite size.

## Rows evaporate before n is counted

The n a comparison is tested at is smaller than the number of examples you wrote, and the gap is deliberate:

• +Rows an evaluator marked non-applicable (metadata.skipped — a tool-only turn handed to a text evaluator, say) are dropped before n is computed, and noted as "K of N rows not applicable". They also leave the slice aggregates and the policy metrics.
• +Truncated calls — the ones that hit max_tokens — are excluded from statistics, because a cut-off answer scores badly for a reason that has nothing to do with the model's quality.
• +Evaluator-side failures (a judge call that broke, an embedding request that errored) are stored as errored and excluded. A flaky judge shrinks n and drifts a comparison toward insufficient, which is visible, instead of poisoning the mean, which is not.

## Slices are not free

Slicing is how you find the regression that only hits Spanish, or only hits tool-heavy turns. It costs twice.

First, it splits the same rows into more, smaller groups — the fastest way to convert a healthy comparison into three uncertain ones. Second, every testable comparison goes through a Benjamini-Hochberg FDR correction at α=0.05 across all comparisons in the run. More comparisons means each one clears a stricter bar to stay significant.

Slice because a subgroup can move independently and you would act on it, not because the tag existed in your data. (Slices holding identical (prompt, evaluator, example) triples collapse to one, so duplicating a slice under two names buys nothing.)

## Effect size, not just p, sets severity

A statistically significant result is not automatically a blocking one. Severity comes from the FDR-corrected p-value together with paired Cohen's d — mean(deltas) / std(deltas, ddof=1), with a 95% CI that is analytical after a t-test and a seeded percentile bootstrap after Wilcoxon:

• +critical — a regression with p < .01 and |d| > 0.8
• +high — significant, |d| > 0.5
• +medium — significant, |d| > 0.2
• +low — significant, small effect
• +improved — significant and positive

That ladder is the practical sizing question restated: a suite sized to catch only |d| > 0.8 will sail past the medium drift that annoys users every day. Small effects need more pairs, and no amount of confidence in the config substitutes for them.

Which test runs is decided for you: Shapiro-Wilk on the deltas at α=0.05 picks a paired t-test when they look normal and a Wilcoxon signed-rank test when they do not — skipped above n=5000, where the CLT justifies a t-test outright. Judge deltas confined to +1/0/-1 usually fail that screen and land on Wilcoxon.

## Rate budgets need a wider suite than tests do

migration_policy rate budgets — max_overall_regression_rate, min_equivalence_rate — are Wilson-confidence-interval aware at 95%. A breach fails the run only when the interval confirms it. Breach with the interval still spanning the budget returns inconclusive, and the reason is the suite, not the model.

Put concretely: a 3% regression-rate budget cannot be confirmed breached by a 30-example suite. One bad example is 3.3%, and the interval around it is enormous. Cost and latency budgets are exact and always conclusive, because they are ratios over measured calls rather than rates over sampled outcomes.

## A sizing rule that survives contact

1. +List the comparisons you would actually act on — the (prompt, evaluator, slice) triples where a regression would change your decision. That count, not the example count, is the shape of the run.
2. +Target 20 paired observations in each of them. Below 20 you get an answer with a flag on it; below 5 you get no answer at all.
3. +Add headroom for what gets dropped — non-applicable rows, truncations, evaluator errors. A 25% cushion is not paranoid on agent suites.
4. +Start with fewer slices than you think you want. You can always split a slice later; you cannot un-spend the FDR budget on slices nobody read.
5. +Check the bill before the run. A paired run is prompts × examples × 2 models, and the CLI asks for confirmation above a $10 estimate (--yes or EVALSHIFT_NONINTERACTIVE skips it). Iterate on suite structure offline with run --offline --fixtures, and let the 7-day response cache absorb the reruns where structure did not change.

The honest version of "how many cases do I need" is: enough that the comparisons you would act on each hold twenty pairs after attrition. For most teams that is a smaller, sharper suite than the one they were planning — and one that comes back with a verdict instead of a shrug.

## Keep reading

• +How to test an LLM model migration before you ship it — the paired run these statistics summarize.
• +Build a golden eval suite from production traffic — where the examples come from in the first place.
• +When to trust an LLM judge — the evaluator most likely to shrink your n without telling you.
• +Methodology — the statistics contract in full.