## Prompt edits deserve the same gate as a model swap

URL: https://www.evalshift.dev/blog/prompt-regression-testing
Published: 2026-08-13
Tag: prompts
Summary: A prompt edit changes behavior the way a model swap does. Where the prompt text lives decides whether CI diffs it or passes on silence.

Takeaways:
- A prompt edit is a behavior change with no diff to review — the same golden suite that gates a model swap gates it, unchanged.
- eval_config_hash covers version, prompts, defaults, evaluators and slices, and the baseline lookup skips every run whose hash differs.
- detection: manual puts the prompt body inside that hash, so every wording edit re-baselines the project and the PR check passes with no comparison at all.
- detection: python_string keeps the body in your application code — the hash holds still, the diff runs, and a regression reads as a regression.
- A permanently green EvalShift check usually means no comparable baseline was found, not that the suite passed.

A model swap gets a meeting. A prompt edit gets a commit. Both change what your product says to
users, and only one of them is reviewed like it matters — usually because a prompt diff is one
paragraph of English that reads fine and proves nothing.

The gate you already built for model migrations works on prompt edits unchanged: a golden suite, a
paired run, a diff against the base branch. But whether a prompt edit actually reaches that gate
depends on something most teams never look at — where the prompt text lives.

## The gate, briefly

On a pull request the EvalShift action runs the suite, uploads the run, asks the hosted API for a
compatible baseline run on the base branch, fetches the server-side diff, updates one PR comment,
and sets the `evalshift/regression` commit status. Under `fail-on: regression` the check goes red
when the diff shows regressions.

The load-bearing word is *compatible*. The lookup scans recent `available` runs on the base branch
for the same suite, and skips every one whose `eval_config_hash` differs from the candidate's.
No compatible baseline means no diff — and no diff means the check passes.

## Where the prompt lives decides whether it is gated

`eval_config_hash` is a SHA-256 over the canonical JSON of your config snapshot: `version`,
`prompts`, `defaults`, `evaluators`, `slices`. The `prompts` block is in the hash. What is inside
that block depends on the detection mode:

```yaml
# detection: manual — the prompt body IS the config
prompts:
  - id: replay
    detection: manual
    content: "You are a support agent. Answer in at most three sentences.\n\n{input}"
    variables: [input]

# detection: python_string — the config only points at the body
prompts:
  - id: customer_routing
    detection: python_string
    path: prompts.py
    variable: AGENT_SYSTEM_PROMPT
    variables: [query]
```

With `detection: manual`, editing one word of the prompt changes `prompts[].content`, which changes
the config snapshot, which changes `eval_config_hash`. Your PR's run is now incomparable with every
run on `main`. The action finds no compatible baseline, reports exactly that in the comment, and
exits green.

With `detection: python_string`, `content` must be null — the config carries only `path` and
`variable`, and the body is AST-extracted from the Python file at run time. Edit the prompt and the
hash does not move: the baseline lookup matches, the server diffs the runs directly, and the
regression shows up as a regression.

> A permanently green EvalShift check usually means no comparable baseline exists, not that the
> suite is passing. If the check went green the same week someone rewrote the system prompt inline,
> that is the mechanism, not a coincidence.

The same rule governs everything else in the hash: retuning `defaults`, adding an evaluator, or
renaming a slice all re-baseline you on purpose. Prompt wording is the one that gets edited weekly
and the one nobody expects to sever the comparison.

## What still changes when the text changes

Keeping the prompt out of the config hash does not make edits invisible to the pipeline:

- The response cache is keyed on `{model, prompt, inputs, temperature, max_tokens[, history]}`, so
  an edited prompt is a cache miss. You cannot accidentally score new wording with old outputs.
  (Offline projects should still set `defaults.cache: false` so fixture edits are not shadowed.)
- Every example is validated against every prompt — template variables covered — *before* any model
  call is dispatched. Dropping `{order_id}` from the template fails the run in seconds rather than
  after $9 of calls.
- `python_string` extraction never imports or executes your code. It AST-walks for a module-level
  string assignment and takes the last one; f-strings, concatenation, `.format()`, and function
  calls are rejected rather than evaluated.

That last rule has a sharp edge. A prompt assembled at runtime cannot be extracted, and the
documented workaround is to paste it into the config as `detection: manual` — which puts the body
back inside the hash and back into re-baselining on every edit. Prefer refactoring the template so
the literal is the literal and the variables are `{placeholders}` the suite fills in.

## Wire it once

Keep one copy of the prompt, in the application code that ships it, and point the config at it:

```yaml
prompts:
  - id: customer_routing
    detection: python_string
    path: app/prompts.py
    variable: AGENT_SYSTEM_PROMPT
    variables: [query]
    tools_path: .evalshift/tools.json
```

Your app imports `AGENT_SYSTEM_PROMPT`; EvalShift reads the same file. There is no second copy to
drift, and a reviewer looking at the PR diff sees the prompt change and the eval result side by
side. For agent prompts, `evalshift tools sync app/tool_definitions.py` extracts the tool schemas
the same way and writes `tools_path` into the config in place, preserving comments.

Then let the run answer the question the prose cannot. Both sides score paired per example, so
"the new prompt is more concise" becomes a delta per evaluator with a severity attached, and the
migration policy turns it into `pass`, `conditional_pass`, `fail`, or `inconclusive` without a
meeting. Roll out with `fail-on: never` for a week to collect baselines, then switch to
`fail-on: regression` once the comments match your judgement.

## The review question worth adopting

For every PR that touches a prompt: *did this run diff against a baseline, or against nothing?*
The comment answers it in one line. A prompt edit that produced no comparison has not been
tested — it has only been observed to compile.

## Keep reading

- [LLM regression testing in CI](/blog/llm-regression-testing-in-ci) — the gate itself, end to end.
- [How many eval cases do you need?](/blog/how-many-eval-cases-do-you-need) — sizing the suite that
  gate reads.
- [Evaluating agent tool calls](/blog/evaluating-agent-tool-calls) — what changes in the trace when
  a system prompt changes.
- [Configuration](/docs/configuration) and [Baselines](/docs/baselines) — the config fields and the
  baseline model in full.