Notes on shipping model changes
Field notes on migrating LLM models safely: golden suites, paired evaluation, regression gates in CI, and when to trust an LLM judge.
An EvalShift migration report that said PASS: a 71% cheaper model inside every policy budget. Where each threshold came from, and what PASS does not claim.
A candidate model cut cost 59% and latency 75% with zero failed calls. The migration report still said FAIL, because the agent's behavior changed.
A model swap is a behavior change with no diff to review. What moves besides the answer text, and why the worst regressions read as correct.
A prompt edit changes behavior the way a model swap does. Where the prompt text lives decides whether CI diffs it or passes on silence.
Suite size is the wrong question: n is counted per prompt, evaluator and slice. What the analysis drops, and how many pairs a verdict needs.
Agent behavior drifts in the trace, not the prose. The four tool-call evaluators, the modes worth changing, and the expectations not worth pinning.
Record real agent runs with the capture SDK, promote them into a golden JSONL suite, and understand every capture the pipeline drops on purpose.
A repeatable method for proving a model swap is safe: freeze a golden suite, run both models paired, and read the diff before your users do.
Wire a golden suite into GitHub Actions so every pull request gets a paired eval run, a base-branch diff, and a check that fails on real regressions.
LLM judges are useful and easy to fool. Where pairwise judging holds up, where it breaks, and how to stop a judge from silently deciding your migration.