Prompt edits deserve the same gate as a model swap

A model swap gets a meeting. A prompt edit gets a commit. Both change what your product says to users, and only one of them is reviewed like it matters — usually because a prompt diff is one paragraph of English that reads fine and proves nothing.

The gate you already built for model migrations works on prompt edits unchanged: a golden suite, a paired run, a diff against the base branch. But whether a prompt edit actually reaches that gate depends on something most teams never look at — where the prompt text lives.

## The gate, briefly

On a pull request the EvalShift action runs the suite, uploads the run, asks the hosted API for a compatible baseline run on the base branch, fetches the server-side diff, updates one PR comment, and sets the evalshift/regression commit status. Under fail-on: regression the check goes red when the diff shows regressions.

The load-bearing word is compatible. The lookup scans recent available runs on the base branch for the same suite, and skips every one whose eval_config_hash differs from the candidate's. No compatible baseline means no diff — and no diff means the check passes.

## Where the prompt lives decides whether it is gated

eval_config_hash is a SHA-256 over the canonical JSON of your config snapshot: version, prompts, defaults, evaluators, slices. The prompts block is in the hash. What is inside that block depends on the detection mode:

# detection: manual — the prompt body IS the config
prompts:
  - id: replay
    detection: manual
    content: "You are a support agent. Answer in at most three sentences.\n\n{input}"
    variables: [input]

# detection: python_string — the config only points at the body
prompts:
  - id: customer_routing
    detection: python_string
    path: prompts.py
    variable: AGENT_SYSTEM_PROMPT
    variables: [query]

With detection: manual, editing one word of the prompt changes prompts[].content, which changes the config snapshot, which changes eval_config_hash. Your PR's run is now incomparable with every run on main. The action finds no compatible baseline, reports exactly that in the comment, and exits green.

With detection: python_string, content must be null — the config carries only path and variable, and the body is AST-extracted from the Python file at run time. Edit the prompt and the hash does not move: the baseline lookup matches, the server diffs the runs directly, and the regression shows up as a regression.

The same rule governs everything else in the hash: retuning defaults, adding an evaluator, or renaming a slice all re-baseline you on purpose. Prompt wording is the one that gets edited weekly and the one nobody expects to sever the comparison.

## What still changes when the text changes

Keeping the prompt out of the config hash does not make edits invisible to the pipeline:

• +The response cache is keyed on {model, prompt, inputs, temperature, max_tokens[, history]}, so an edited prompt is a cache miss. You cannot accidentally score new wording with old outputs. (Offline projects should still set defaults.cache: false so fixture edits are not shadowed.)
• +Every example is validated against every prompt — template variables covered — before any model call is dispatched. Dropping {order_id} from the template fails the run in seconds rather than after $9 of calls.
• +python_string extraction never imports or executes your code. It AST-walks for a module-level string assignment and takes the last one; f-strings, concatenation, .format(), and function calls are rejected rather than evaluated.

That last rule has a sharp edge. A prompt assembled at runtime cannot be extracted, and the documented workaround is to paste it into the config as detection: manual — which puts the body back inside the hash and back into re-baselining on every edit. Prefer refactoring the template so the literal is the literal and the variables are {placeholders} the suite fills in.

## Wire it once

Keep one copy of the prompt, in the application code that ships it, and point the config at it:

prompts:
  - id: customer_routing
    detection: python_string
    path: app/prompts.py
    variable: AGENT_SYSTEM_PROMPT
    variables: [query]
    tools_path: .evalshift/tools.json

Your app imports AGENT_SYSTEM_PROMPT; EvalShift reads the same file. There is no second copy to drift, and a reviewer looking at the PR diff sees the prompt change and the eval result side by side. For agent prompts, evalshift tools sync app/tool_definitions.py extracts the tool schemas the same way and writes tools_path into the config in place, preserving comments.

Then let the run answer the question the prose cannot. Both sides score paired per example, so "the new prompt is more concise" becomes a delta per evaluator with a severity attached, and the migration policy turns it into pass, conditional_pass, fail, or inconclusive without a meeting. Roll out with fail-on: never for a week to collect baselines, then switch to fail-on: regression once the comments match your judgement.

## The review question worth adopting

For every PR that touches a prompt: did this run diff against a baseline, or against nothing? The comment answers it in one line. A prompt edit that produced no comparison has not been tested — it has only been observed to compile.

## Keep reading

• +LLM regression testing in CI — the gate itself, end to end.
• +How many eval cases do you need? — sizing the suite that gate reads.
• +Evaluating agent tool calls — what changes in the trace when a system prompt changes.
• +Configuration and Baselines — the config fields and the baseline model in full.