How many eval cases do you need?

Everyone asks the question the same way: how many examples does a golden suite need? Fifty? Two hundred? The number that actually decides whether your run can say anything is not the size of the suite. It is n inside one comparison — and the suite is chopped into comparisons before a single statistic is computed.

This is where most first suites disappoint. Two hundred examples feel serious, come back inconclusive, and nobody can explain why. Here is how n is counted, what is dropped before counting, and how to size a suite so the verdict is earned rather than lucky.

n is per comparison, not per suite

The analysis groups paired deltas — target_score - source_score, per example — by the triple (prompt_id, evaluator_name, slice_name). Every one of those groups is tested on its own, and every one has its own n.

So a 200-example suite run across 3 prompts, scored by 4 evaluators, sliced 3 ways is not one comparison with n=200. It is up to 36 comparisons, each drawing on the rows that match its slice. Give the smallest slice 15 examples and that column of the matrix is stuck at n=15 no matter how impressive the suite total looks.

Three constants decide what happens next:

• +MIN_N_FOR_TEST = 5 — below five paired observations the comparison is skipped entirely, with severity insufficient. It is not scored as "no change"; it is scored as "not measured".
• +MIN_N_RELIABLE = 20 — between 5 and 20 the comparison is tested but flagged uncertain.
• +Zero variance (std < 1e-9) — skipped with severity none, because every delta was identical.

Twenty per group is the line where a result stops carrying an asterisk. Work backwards from the groups you care about, not forwards from a round suite size.

Rows evaporate before n is counted

The n a comparison is tested at is smaller than the number of examples you wrote, and the gap is deliberate:

• +Rows an evaluator measured nothing on — a tool-only turn handed to a text evaluator, say — write no score at all, so they are absent from n before it is computed, and noted as "K of N rows not applicable". They also leave the slice aggregates and the policy metrics.
• +Truncated calls — the ones that hit max_tokens — are excluded from statistics, because a cut-off answer scores badly for a reason that has nothing to do with the model's quality.
• +Evaluator-side failures (a judge call that broke, an embedding request that errored) are stored as errored and excluded. A flaky judge shrinks n and drifts a comparison toward insufficient, which is visible, instead of poisoning the mean, which is not.

Slices are not free

Slicing is how you find the regression that only hits Spanish, or only hits tool-heavy turns. It costs twice.

First, it splits the same rows into more, smaller groups — the fastest way to convert a healthy comparison into three uncertain ones. Second, every testable comparison goes through a Benjamini-Hochberg FDR correction at α=0.05 across all comparisons in the run. More comparisons means each one clears a stricter bar to stay significant.

Slice because a subgroup can move independently and you would act on it, not because the tag existed in your data. (Slices holding identical (prompt, evaluator, example) triples collapse to one, so duplicating a slice under two names buys nothing.)

Effect size, not just p, sets severity

A statistically significant result is not automatically a blocking one. Severity comes from the FDR-corrected p-value together with paired Cohen's d — mean(deltas) / std(deltas, ddof=1), with a 95% CI that is analytical after a t-test and a seeded percentile bootstrap after Wilcoxon:

• +critical — a regression with p < .01 and |d| > 0.8
• +high — significant, |d| > 0.5
• +medium — significant, |d| > 0.2
• +low — significant, small effect
• +improved — significant and positive

That ladder is the practical sizing question restated: a suite sized to catch only |d| > 0.8 will sail past the medium drift that annoys users every day. Small effects need more pairs, and no amount of confidence in the config substitutes for them.

Which test runs is decided for you: Shapiro-Wilk on the deltas at α=0.05 picks a paired t-test when they look normal and a Wilcoxon signed-rank test when they do not — skipped above n=5000, where the CLT justifies a t-test outright. Judge deltas confined to +1/0/-1 usually fail that screen and land on Wilcoxon.

Rate budgets need a wider suite than tests do

migration_policy rate budgets — max_overall_regression_rate, min_equivalence_rate — are Wilson-confidence-interval aware at 95%. A breach fails the run only when the interval confirms it. Breach with the interval still spanning the budget returns inconclusive, and the reason is the suite, not the model.

Put concretely: a 3% regression-rate budget cannot be confirmed breached by a 30-example suite. One bad example is 3.3%, and the interval around it is enormous. Cost and latency budgets are exact and always conclusive, because they are ratios over measured calls rather than rates over sampled outcomes.

A sizing rule that survives contact

1. +List the comparisons you would actually act on — the (prompt, evaluator, slice) triples where a regression would change your decision. That count, not the example count, is the shape of the run.
2. +Target 20 paired observations in each of them. Below 20 you get an answer with a flag on it; below 5 you get no answer at all.
3. +Add headroom for what gets dropped — non-applicable rows, truncations, evaluator errors. A 25% cushion is not paranoid on agent suites.
4. +Start with fewer slices than you think you want. You can always split a slice later; you cannot un-spend the FDR budget on slices nobody read.
5. +Check the bill before the run. A paired run is prompts × examples × 2 models, and the CLI asks for confirmation above a $10 estimate (--yes or EVALSHIFT_NONINTERACTIVE skips it). Iterate on suite structure with evalshift validate — it loads config, suite, and prompts and cross-checks them without a single model call — and let the 7-day response cache absorb the reruns where structure did not change.

The honest version of "how many cases do I need" is: enough that the comparisons you would act on each hold twenty pairs after attrition. For most teams that is a smaller, sharper suite than the one they were planning — and one that comes back with a verdict instead of a shrug.

FAQ

How many examples does an LLM eval suite need? Enough that every comparison you would act on holds about 20 paired observations after attrition. Because deltas are grouped per prompt, evaluator and slice, the suite total matters far less than the size of the smallest group — below 5 pairs a comparison is skipped as insufficient, and between 5 and 20 it is tested but flagged uncertain.

Why did my eval run come back inconclusive? Either every comparison was insufficient, or a rate budget was breached without the 95% Wilson confidence interval confirming it. Both are suite-size answers: the run is saying it cannot distinguish the result from noise, not that the models are equivalent.

Do more slices make an eval run more sensitive? No — they make it less sensitive per comparison. Slices split the same rows into smaller groups, and every testable comparison goes through a Benjamini-Hochberg FDR correction at alpha=0.05 across the whole run, so adding comparisons raises the bar each one has to clear.

Which rows are excluded from eval statistics? Rows an evaluator marked non-applicable are dropped before n is computed and noted in the report; calls truncated at max_tokens are excluded; and records where the evaluator itself errored are stored as errored and excluded. When nothing survives, the comparison reports a nothing-measured note rather than severity none.

Keep reading

• +How to test an LLM model migration before you ship it — the paired run these statistics summarize.
• +Build a golden eval suite from production traffic — where the examples come from in the first place.
• +When to trust an LLM judge — the evaluator most likely to shrink your n without telling you.
• +Methodology — the statistics contract in full.