EvalShift
Login
// cloud · verdicts

Verdicts & gating

how a run is judged

Every Cloud run carries a single verdict — the headline answer to “is this migration safe?” The verdict is computed by the CLI from the run’s statistical analysis and the migration policy, stamped into the immutable bundle, and stored as an indexed column so EvalShift Cloud never has to crack the blob open to show it.

The four verdicts

BadgeValueMeaning
PASSpassNo budget exceeded and no blocking regression. Safe to ship the target model.
CONDITIONALconditional_passEvery budget held and nothing critical or high regressed over the whole run, but there are caveats: medium/low regressions, a blocking evaluator that scored no pairs, or a critical/high regression confined to one slice. A passing state — it never blocks a PR in the Cloud gate or the GitHub Action (the local --policy-gate flag is stricter; see below).
FAILfailA budget was conclusively exceeded, overall or on a slice, or a blocking evaluator regressed at severity critical/high over the whole run. This is the only verdict that gates.
INCONCLUSIVEinconclusiveNo decision could be made: no policy is configured, nothing was measured (no blocking evaluator scored a record), every comparison scored severity insufficient, or a rate budget was breached on too small a sample to confirm it. The reason string says which.

The gate is binary

Badges are four-valued, but enforcement is two-valued. A run is blocked if and only if its verdict is fail — in the common case, a policy budget conclusively exceeded (overall or on a slice, the two block alike) or a blocking regression of severity critical or high over the whole run (its reason then names each such comparison). A rate-budget breach the sample cannot confirm is inconclusive, not a fail; the other edge cases are below. Everything else passes the gate:

  • +PASS and CONDITIONAL both clear the gate — conditional_pass is informational only.
  • +FAIL blocks (PR comment + commit status, or the Blocked marker in the Pull Requests view).
  • +INCONCLUSIVE never blocks; it means there was nothing to decide against. It is not a quiet pass — the app renders it in a neutral tone rather than the amber of CONDITIONAL, because “held with caveats” and “could not be judged” are different answers. A run pushed with no migration policy, and a run whose blocking evaluators scored no records, both land here. A blocking semantic or llm_judge whose model has no API key no longer gets this far: since CLI 1.4.0 compare, run and evaluate exit 1 before any call — see Blocking vs advisory.
−
the local --policy-gate is stricter
The rule above is the Cloud gate’s — what the Pull Requests view and the GitHub Action’s fail-on: policy enforce. The CLI’s own analyze --policy-gate / compare --policy-gate exits 1 on fail and on conditional_pass, and also when no migration_policy is configured; it passes pass and inconclusive. A run the Action lets merge can therefore still fail a local --policy-gate step.
+
server-side enforcement
EvalShift Cloud answers for a run under the policy that was pushed with it: the verdict the CLI computed, stored at finalize alongside the budgets it was computed against. Tightening a budget in evalshift.yaml changes the next run, never a stored one. See Migration policy.

Edge cases

  • +A regression confined to one slice. Severity gates on the whole-run comparisons. A critical or high regression that shows up only in one slice makes that slice fail and the run conditional_pass, which does not block. A slice budget is different: breach it conclusively and the run fails.
  • +Every comparison insufficient. When every whole-run comparison, across all blocking evaluators, has fewer than 5 scored pairs, the verdict is inconclusive before any budget is read — even a conclusively breached one. A run whose blocking evaluators scored no records at all is handled differently: its cost and latency budgets, which come from the calls, still fail on a conclusive breach.
  • +Two overrides. A max_invariant_violations breach fails the run however small the sample, even when every comparison is insufficient, and on a slice exactly as overall. fail_on_dropped_params: true fails a run whose arm could not honour a recorded generation parameter, whatever the verdict would otherwise have been.
  • +A slice key no tag matches. The keys under migration_policy.slices are matched against your suite’s tags. A key no example carries is never measured and never flagged — it adds no budget rows and leaves the verdict alone, so a typo there passes silently. Check the keys against your tags. Runs judged by a legacy web-configured policy are the exception: there, a policy slice the run never measured, or a run that recorded no metrics, is inconclusive.

Where the verdict shows up

  • +Run history — a verdict column on the runs table, with a verdict filter.
  • +Run detail — the header strip, alongside the regression count and cost/latency deltas.
  • +Diff view — a verdict-delta strip showing the baseline verdict moving to the candidate verdict. See Baselines & diffs.
  • +Trends — a verdict-history band over time. See Trends & slices.
  • +Pull Requests — the latest verdict per PR plus a policy status badge.