Notes on shipping model changes
LLM migration and regression testing, from the field: golden suites, paired evaluation, regression gates in CI, and when to trust an LLM judge.
Model migrations
Switching, upgrading or retiring a model without changing what the agent does.
What LLM benchmarks measure, where the November 2026 leaderboards disagree on the best model, and how to test a shortlist on your own agent's traffic.
What an LLM router saves at November 2026 API prices, the costs list prices leave out, and how to check routing doesn't cut quality before it ships.
The 2026-2027 LLM deprecation schedule: every OpenAI, Anthropic and Google shutdown date, the replacements, notice periods and how to migrate.
The cheapest LLM APIs as of October 2026, from $0.05 per million tokens, and how to tell if a cheap model is good enough for your agent before you switch.
Gemini 2.5 Pro and Flash are deprecated, and the Gemini API reroutes them on December 1, 2026. Dates, replacements and a six-step migration checklist.
Sample production logs into a golden set, write deterministic assertions, diff baseline against candidate and gate it in CI. Built on six real failures.
An EvalShift migration report that said PASS: a 71% cheaper model inside every policy budget. Where each threshold came from, and what PASS does not claim.
A candidate model cut cost 59% and latency 75% with zero failed calls. The migration report still said FAIL, because the agent's behavior changed.
A model swap is a behavior change with no diff to review. What moves besides the answer text, and why the worst regressions read as correct.
A repeatable method for proving a model swap is safe: freeze a golden suite, run both models paired, and read the diff before your users do.
Evaluators and LLM judges
What to check in an agent's output and tool calls, and when an LLM judge can be trusted.
Ten open-source LLM evaluation frameworks compared on license, maintenance, telemetry defaults, judge costs, agent checks and CI, as of November 2026.
Braintrust alternatives compared: Langfuse, LangSmith, Phoenix, Opik, MLflow, DeepEval and Promptfoo on self-hosting, license, pricing and evals.
LangSmith alternatives compared: Langfuse vs LangSmith on pricing, self-hosting and retention, plus Braintrust, Phoenix, Opik, Laminar and MLflow.
The best LLM evaluation tools and frameworks in 2026, compared: what each grades, agent and CI support, licenses, pricing, and which to pick.
How to evaluate AI agents: six dimensions to measure, three levels to test at, sandbox vs mocks vs replay, benchmarks, and a release scorecard.
AI evals are tests that grade what a model or agent did. How agent evals differ from LLM evals, the three grader types, and how to write one in 7 steps.
LLM-as-a-judge uses one model to grade another's output against written criteria. How it works, how accurate it is, its biases, and when not to use it.
Agent behavior drifts in the trace, not the prose. The four tool-call evaluators, the modes worth changing, and the expectations not worth pinning.
LLM judges are useful and easy to fool. Where pairwise judging holds up, where it breaks, and how to stop a judge from silently deciding your migration.
Golden suites
Building an eval suite from real traffic, and sizing it to return a verdict.
Suite size is the wrong question: n is counted per prompt, evaluator and slice. What the analysis drops, and how many pairs a verdict needs.
Record real agent runs with the capture SDK, promote them into a golden JSONL suite, and understand every capture the pipeline drops on purpose.
CI and prompt changes
Gating pull requests on eval diffs, prompt edits included.
What LLM testing is, the types of LLM tests, six steps to test an LLM application, and what to rerun when the model underneath it changes.
A prompt edit changes behavior the way a model swap does. Where the prompt text lives decides whether CI diffs it or passes on silence.
Wire a golden suite into GitHub Actions so every pull request gets a paired eval run, a base-branch diff, and a check that fails on real regressions.