Baselines & diffs — EvalShift docs

// hosted · baselines

Baselines & diffs

pin once, compare every run

A baseline is the run every new run is measured against. Pin one per suite and the hosted app turns from a run archive into a migration-safety dashboard: each push gets an automatic diff and a verdict relative to the run you decided was good.

## Pinning a baseline

+Pick a run you trust as the reference point — usually the last known-good run on main.
+Pin it: Pin as baseline in the run's row menu on the timeline, or Promote to baseline in the run detail action strip. One baseline is active per suite; pinning replaces that suite's previous baseline and leaves your other suites alone.
+Every later run in that suite is automatically diffed against it at ingest, so the comparison is ready the moment you open it.

The project’s Baselines page lists one row per suite: the pinned run, its verdict and models, when it was pinned, and how many runs the suite has. Each row’s actions menu offers Compare with newest run, and — for project owners — Unpin baseline or Pin a run. Suites with runs but no baseline are listed too — runs in those suites are not compared to anything until you pin one. The run timeline keeps a one-line summary that links to the page.

re-pin invalidates nothing

Diffs are keyed by the pair of runs being compared and are deterministic forever, so changing the baseline never invalidates a cached diff — the new baseline simply warms lazily on first read. The headline “vs current baseline” is always computed against whatever is pinned right now.

## What the diff shows

The diff view (linked from a run, and from the PR comment when a compatible baseline exists) layers four things:

+Verdict strip — the baseline verdict moving to the candidate verdict. See Verdicts & gating.
+Cost / latency strip — aggregate spend and p50/p95 latency deltas. A negative (cheaper / faster) delta is shown in the accent color; a positive one in the regression color.
+Aggregate + slice table — pass-rate deltas overall and per slice, sorted so the biggest movers surface first, with newly-regressed / newly-improved pills on slices that flipped sign.
+Per-example deltas — the individual outputs that drove the change.

## Compatibility

A diff is only meaningful when both runs evaluated the same thing. Runs with a different evaluation config hash are marked incompatible and the per-example deltas are suppressed; the verdict and cost/latency context still render. This is the same compatibility rule the GitHub Action uses to decide whether to link a diff at all.

## Existing runs

Runs that predate a freshly-pinned baseline get their diffs filled in by a one-shot backfill job the operator runs after pinning. Until then, opening a diff computes it lazily on read — there is never a stale result, only a cold one.

## Pinning a baseline

+Pick a run you trust as the reference point — usually the last known-good run on main.

+Pin it: Pin as baseline in the run's row menu on the timeline, or Promote to baseline in the run detail action strip. One baseline is active per suite; pinning replaces that suite's previous baseline and leaves your other suites alone.

+Every later run in that suite is automatically diffed against it at ingest, so the comparison is ready the moment you open it.

re-pin invalidates nothing

## What the diff shows

The diff view (linked from a run, and from the PR comment when a compatible baseline exists) layers four things:

+Verdict strip — the baseline verdict moving to the candidate verdict. See Verdicts & gating.

+Cost / latency strip — aggregate spend and p50/p95 latency deltas. A negative (cheaper / faster) delta is shown in the accent color; a positive one in the regression color.

+Aggregate + slice table — pass-rate deltas overall and per slice, sorted so the biggest movers surface first, with newly-regressed / newly-improved pills on slices that flipped sign.

+Per-example deltas — the individual outputs that drove the change.

## Compatibility