## LLM routing cuts costs. Here's how to check it doesn't cut quality

URL: https://www.evalshift.dev/blog/llm-router
Published: 2026-11-14
Tag: cost
Summary: What an LLM router saves at November 2026 API prices, the costs list prices leave out, and how to check routing doesn't cut quality before it ships.

Takeaways:
- An LLM router's saving is capped by the share of calls it moves times the price gap between the two models: at November 2026 list prices, on prompts under 100,000 tokens, moving 70% of calls from Claude Sonnet 5.5 to Claude Haiku 5.5 cuts the bill by 66.5%, while a router choosing between Claude Opus 5.5 and Sonnet 5.5 can never save more than 50%.
- Published router results are benchmark averages: RouteLLM's matrix-factorization router kept 95% of GPT-4's MT Bench score while sending 26% of calls to GPT-4, and LLMRouterBench (2026, 21 datasets, 33 models) found several routing methods, including the one commercial router it tested, did not reliably beat the best single model.
- LLM routing costs more than list prices suggest: a prompt cache belongs to the model that wrote it, so switching models mid-conversation pays to write the context again, and a cascade pays for both models on every escalated request.
- In AI agents the quality routing loses is behavior, not prose: in one measured swap the cheaper model was 58.8% cheaper with zero failed calls but chose a different tool on 25% of comparisons, and a router sending that traffic to it would ship those failures.
- To check an LLM router, replay 100 to 300 production inputs tagged by route on both the strong and the cheap model, write per-route limits before the run, score the router's decisions against the per-example results, and route only the routes where the cheap model held.

An LLM router picks a model for each request: the easy ones go to a cheap model, the hard ones
to a strong one. How much it can save depends on two numbers, the share of traffic it moves and
the price gap between the two models. At November 2026 list prices, on prompts under 100,000
tokens, Claude Haiku 5.5 costs a twentieth of Claude Sonnet 5.5, so a router that moves 70% of
calls from Sonnet to Haiku cuts the bill by 66.5%. A router choosing between Claude Opus 5.5 and
Sonnet 5.5, where Sonnet costs half as much as Opus, can never save more than 50%, even if it
routes everything.

What routing costs in quality is a separate number, and no router's published results tell you
what it is for your application. Those results are averages on public benchmarks: RouteLLM's
matrix-factorization router reached 95% of GPT-4's MT Bench score while sending 26% of calls to
GPT-4, and an independent 2026 benchmark across 21 datasets found that several routing methods,
including the one commercial router it tested, did not reliably beat the best single model. In
an agent, routing loses quality on a few routes: a wrong tool call, or an action the model claims
and never takes, inside text that reads fine. Averages hide that.

The check is to replay your own production traffic on the strong model and on the cheap one,
compare every example per route against limits you wrote before the run, and route only where the
cheap model held. The rest of this post covers the kinds of router, what routing saves at today's
API prices, the costs list prices leave out, and that check step by step. I build EvalShift,
which automates the comparison, but the method works with any tooling.

## What an LLM router is

An LLM router is a layer between your application and the model APIs that chooses which model
answers each request. The application sends one request; the router decides whether Haiku, Sonnet
or some other model handles it, usually by predicting whether the cheaper model is good enough.

LLM model routing covers several different things, and only some of them route on quality:

| Kind | Decides by | Examples | Routes on quality? |
| --- | --- | --- | --- |
| Rule-based or heuristic | Route, metadata, request parameters, a complexity score | Portkey conditional routing, LiteLLM auto routing (default scorer), a `switch` in your own code | Only as well as your rules |
| Gateway load balancing | Availability, latency, rate limits, price | LiteLLM's router strategies and fallbacks | No |
| Learned router | A classifier that predicts which model is good enough | RouteLLM, Not Diamond, Requesty, vLLM Semantic Router | Yes, as trained on someone else's data |
| Managed router | The platform's classifier over its own model pool | Azure Foundry model router, Amazon Bedrock Intelligent Prompt Routing, OpenRouter Auto Router | By the platform's quality estimates; OpenRouter ranks by other users' spending |
| Cascade | Try the cheap model first, escalate when a check fails | FrugalGPT | Yes, if the check is reliable |

The difference between an LLM router and an LLM gateway is mostly what the routing decision is
for. A gateway spreads requests across deployments so they stay up and inside rate limits:
LiteLLM's router picks among deployments by `simple-shuffle` (its default), least busy, lowest
latency or lowest cost, and falls back when one fails. A router in the narrow sense chooses the
model by how hard the request is. Many products do both, including LiteLLM, which added a beta
auto-routing mode that scores each request with a rule-based heuristic by default and sorts it
into simple, medium, complex and reasoning tiers.

A few details from the managed routers matter for testing them:

- **OpenRouter's Auto Router** (`openrouter/auto`) classifies each request into one of about 30
  task types, then ranks models by what OpenRouter's users spent on that task type over the past
  seven days. It adds no fee, and the
  [docs](https://openrouter.ai/docs/guides/routing/routers/auto-router) say it can pick a
  different model on every turn; with a session ID it reuses the earlier model only while that
  model is still one of the top candidates.
- **Azure's model router** has three modes. Balanced, the default, picks the cheapest model within
  1% to 2% of the top-quality model, Cost widens that band to 5% to 6%, and Quality ignores cost.
  It routes on the request's text only, and Microsoft updates model version 2025-11-18 in place
  ([docs](https://learn.microsoft.com/en-us/azure/foundry/openai/concepts/model-router)).
- **Amazon Bedrock Intelligent Prompt Routing** chooses between two models from one family by
  predicted response quality, with a "response quality difference" threshold you set. AWS says it
  is optimized for English prompts only and can't adjust its decisions to application-specific
  performance data ([docs](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html)).

Two of those three change their behavior without a deploy on your side. OpenRouter's ranking
follows a trailing seven days of other people's spending, and Azure's router changes when
Microsoft updates it. Keep that in mind for the last step of the check.

## LLM API pricing and how much routing saves

The ceiling on what routing saves is simple arithmetic: **saving = share of calls routed × (1 −
cheap price ÷ strong price)**. So the first input is the price gap between the models you would
route between.

Prices below are per million tokens, standard tier, read from each provider's pricing page on
November 6, 2026 ([OpenAI](https://developers.openai.com/api/docs/pricing),
[Anthropic](https://platform.claude.com/docs/en/about-claude/pricing),
[Google](https://aistudio.google.com/docs/pricing),
[DeepSeek](https://api-docs.deepseek.com/quick_start/pricing/)). The last column assumes 3,000
input and 400 output tokens per call, uncached, roughly one turn of a small tool-calling agent,
the same call size as in [the cheapest LLM API comparison](/blog/cheapest-llm-api).

| Provider | Model | Input | Output | 1,000 calls |
| --- | --- | --- | --- | --- |
| Anthropic | Claude Opus 5.5 | $4.00 | $20.00 | $20.00 |
| Anthropic | Claude Sonnet 5.5 | $2.00 | $10.00 | $10.00 |
| Anthropic | Claude Haiku 5.5 | $0.10 | $0.50 | $0.50 |
| OpenAI | `gpt-6.1-terra` | $1.00 | $5.00 | $5.00 |
| OpenAI | `gpt-6-luna` | $0.10 | $0.50 | $0.50 |
| Google | `gemini-3.8-flash` | $0.75 | $3.75 | $3.75 |
| Google | `gemini-3.1-flash-lite` | $0.25 | $1.50 | $1.35 |
| DeepSeek, off-peak | `deepseek-v4-pro` | $0.66 | $1.98 | $2.77 |
| DeepSeek, off-peak | `deepseek-flash` | $0.15 | $0.60 | $0.69 |

Haiku 5.5's price holds for prompts up to 100,000 tokens. Above that every token in the request
bills at $0.50 / $2.50, a quarter of Sonnet's price instead of a twentieth, which matters for an
agent deep into a long conversation.

Applied to same-provider pairs, the formula gives the most each router could save at list price:

| Route from → to | Cheap ÷ strong | 50% routed | 70% routed | 90% routed |
| --- | --- | --- | --- | --- |
| Claude Sonnet 5.5 → Haiku 5.5 | 5% | 47.5% | 66.5% | 85.5% |
| `gpt-6.1-terra` → `gpt-6-luna` | 10% | 45.0% | 63.0% | 81.0% |
| `deepseek-v4-pro` → `deepseek-flash` | 24.9% | 37.6% | 52.6% | 67.6% |
| `gemini-3.8-flash` → `gemini-3.1-flash-lite` | 36% | 32.0% | 44.8% | 57.6% |
| Claude Opus 5.5 → Sonnet 5.5 | 50% | 25.0% | 35.0% | 45.0% |

Three things to read from it:

- **The price gap matters as much as the router.** Moving 70% of traffic within the Anthropic
  pair saves 66.5%; moving the same share between Gemini 3.8 Flash and 3.1 Flash-Lite saves
  44.8%. The Gemini row moves on January 1, 2027, when 3.8 Flash's introductory price ends and it
  goes to $1.50 / $7.50, which takes the 70% saving to 57.4%.
- **Cross-provider pairs need measured cost, not list price.** I left them out because the same
  text is a different number of tokens on each provider. Anthropic says Claude 4.7 and later
  models count the same text as about 30% more tokens than earlier Claude models, so an LLM cost
  comparison across tokenizers is off before any routing starts.
- **The 90% column is a ceiling, not a forecast.** It assumes every routed call costs the cheap
  model's list price and nothing else changes. The next section is the list of things that do.

For comparison, here are the published savings claims. LMSYS reported that RouteLLM cut costs by
over 85% on MT Bench, 45% on MMLU and 35% on GSM8K while keeping 95% of GPT-4's performance
([blog](https://lmsys.org/blog/2024-07-01-routellm/)). Its best MT Bench router, trained on
augmented data, sent 14% of calls to GPT-4 (26% when trained on Arena data alone), and the pair
was GPT-4 and Mixtral 8x7B, much further apart in price than any pair above. AWS says Intelligent Prompt Routing can
cut costs by up to 30%. LiteLLM benchmarked its own auto-routing on 300 prompts graded by a
blinded pairwise judge: its semantic router, now deprecated, saved 58.2% and scored a 46.8% win
rate against the flagship model, where 50% means parity and a randomly shuffled control scored
39.2%. Its newer tier-based complexity router did not beat that control
([benchmark](https://docs.litellm.ai/docs/proxy/auto_routing_benchmark)). Every one of those
numbers is measured on someone else's prompts.

## What routing costs that the price table leaves out

Four costs move the real saving below the table:

- **Prompt caching stops working across a switch.** A prompt cache holds the attention state one
  model computed for your prefix, so no other model can use it. Google's caching API says
  cached content can only be used with the model it was created for, and Azure's router docs say
  caching benefits apply only when the same model handles consecutive requests. On Claude Sonnet
  5.5, a cached read costs $0.10 per million tokens and a five-minute cache write $2.50. Twenty
  thousand tokens into a conversation, an agent on Sonnet pays $0.002 a turn to read that context
  from the cache. If the router moves the conversation to Sonnet from another model, Sonnet has
  never seen it, and the first turn pays $0.05 to write it, 25 times as much. A
  [measured write-up on Towards Data Science](https://towardsdatascience.com/the-hidden-cost-of-llm-routing-why-switching-models-costs-more/)
  found that per-request routing at the gateway, under real pricing, delivered only 13 of the 40
  percentage points of saving the headline suggested, and destroyed almost twice as much cache as
  sticky routing.
- **A cascade pays twice for every escalation.** An escalated request costs the cheap call plus
  the strong one, and the user waits for both. With Haiku 5.5 in front of Sonnet 5.5, a cascade
  that escalates 30% of requests costs 35% of always-Sonnet, against 33.5% for a router that sends
  the same 30% straight to Sonnet. Cascades are also only as good as their check: RouterBench found
  cascade performance drops rapidly as the judge's error rate rises
  ([paper](https://arxiv.org/abs/2403.12031)).
- **Cheap models can take more turns per task.** An agent pays per finished task, not per call. A
  model that needs an extra tool round or retries a malformed call can cost more per task than its
  list price suggests; the [cheapest LLM API post](/blog/cheapest-llm-api) covers the mechanics.
- **The router itself.** A classifier call adds latency to every request, and some routers charge
  for it: Amazon Bedrock lists a separate charge of $1 per 1,000 requests for Intelligent Prompt
  Routing, twice what 1,000 calls to Haiku 5.5 cost at the call size above.

Routing is one LLM cost optimization among several, and the others are cheaper to try first. For
the OpenAI and Anthropic models in the table, cached input costs 0.05x to 0.1x of the input price.
OpenAI, Anthropic and Google all take 50% off batch requests for work that can wait. And if every
route holds on the cheap model, you don't need a router at all; you need a migration.

## Why router benchmarks don't tell you what your app loses

Every learned or managed router decides on a prediction of quality, or a proxy for one
(OpenRouter's is other users' spending), and the predictions are trained and scored on someone
else's data. RouteLLM's routers were trained mainly on Chatbot Arena preference data, augmented
with LLM-judge and MMLU labels, and tested on MT Bench, MMLU and GSM8K, which are chat and
question-answering sets. Its "95% of GPT-4 performance" means the routed mix scored 8.8 on MT
Bench against GPT-4's 9.3, which closes half of the gap between the two models, averaged over the
benchmark.

Independent benchmarks are more sobering. RouterBench, built from more than 405,000 inference
outcomes, found that its predictive routers did not significantly beat a baseline that mixes
models in fixed proportions without reading the query. LLMRouterBench (January 2026; 21 datasets,
33 models) found that many routing methods, including OpenRouter's, the one commercial router it
tested, fail to reliably beat the best single model. It reports OpenRouter at −24.7% against the
best single model, and its best methods cut cost by up to 31.7% while matching that model
([paper](https://arxiv.org/abs/2601.07206)). The paper
tested OpenRouter's router as it was then, and the current one ranks models by spend, so treat
that number as a reason to test, not a verdict on today's version.

Microsoft's own guide to
[evaluating its model router](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/evaluate-model-router)
makes the point directly: group prompts by workload category "so that an aggregate result doesn't
hide a category-specific regression". A Towards Data Science
[case study](https://towardsdatascience.com/we-built-a-routing-layer-to-cut-our-ai-costs-it-broke-the-product/)
shows what happens when you don't. The cheap tier of a routing layer matched the strong model on
94% of a 5,000-query holdout, quality monitoring that wasn't split by tier hid the long-tail
failures for months, and the team estimated the quality loss cost four to five times what routing
saved.

### In agents, the loss is behavior

Routing quality in an agent is mostly about what the agent *does*. I replayed 16 real examples
from an agent on its current model and a cheaper candidate: 58.8% cheaper, 75.2% faster, not one
failed or truncated call. The migration report still returned **FAIL**.

![Migration verdict card: FAIL, 5 of 7 budgets within policy, 61.4% equivalent, 25.0% improved, 13.6% regressed, cost −58.8%, latency −75.2%.](/blog-images/blog1/blog1-1.png "The verdict card from a swap that was 58.8% cheaper. The two budgets that failed are the two that describe behavior.")

The cheaper model picked a different tool on 25% of comparisons against a 10% limit, searched the
web where the current model answered in two sentences, and told a user an action was done without
calling the tool that does it. Every call succeeded at the API level. A router that sent that
agent's traffic to the cheaper model would have shipped those failures on the requests it routed
there, and a dashboard of cost, latency and error rate would have called it a win. The full
report is in [the new model was 59% cheaper and 75% faster, and I still wouldn't ship it](/blog/cheaper-faster-and-still-a-fail).

The opposite result is just as real. A customer-support agent with six tools moved to a cheaper
model over 120 production examples and held every budget, including zero regressions on its
security slice, at 71.2% lower cost
([the passing report](/blog/what-a-passing-migration-proves)). Price told me nothing about which
of the two would pass.

## Where routing breaks an agent

The regressions routing introduces cluster in four places:

- **Requests that look easy and aren't.** "Yes, cancel the second one" is five words. A router
  that scores the latest message sees an easy request; the work is in the conversation behind it.
  In the case study above, "where is my charge from" read as a simple billing lookup and was
  sometimes the start of a fraud report.
- **Switching models mid-task.** OpenRouter's Auto Router can choose a different model on every
  turn, while Requesty's auto routing pins its choice for 60 minutes per conversation. A model
  picked mid-task inherits another model's half-finished plan, with different tool-call habits
  and a cold cache.
- **The cheap model's own agent failures.** It calls a tool where none was needed or skips a
  required lookup, claims an action it never took, picks a neighboring tool, or passes an id with
  a typo. The [guide to evaluating agent tool calls](/blog/evaluating-agent-tool-calls)
  covers the check for each.
- **Drift with no deploy.** Thresholds are tuned on last month's traffic. When the mix shifts, the
  share routed to the cheap model shifts with it, and managed routers change their own pools and
  rankings on their own schedule.

## How to check an LLM router doesn't cut quality

Six steps. The first five run before the router takes traffic; the sixth keeps it honest after.

## Step 1: sample production traffic and tag it by route

Take 100 to 300 real inputs from production, stratified by route or intent so the rare and
expensive paths are in the sample, and tag each one with its route. For an agent, take whole
conversations, not single messages, so a mid-conversation switch is part of what you test. A
[golden suite built from production traffic](/blog/build-a-golden-suite-from-production-traffic)
is the reusable version of this.

Size the routes you plan to decide on, not just the total. Zero regressions out of 20 examples
still leaves a true rate of up to about 15% at 95% confidence (the rule of three, 3/*n*); zero
out of 100 brings it down to about 3%. [How many eval cases you need](/blog/how-many-eval-cases-do-you-need)
has the full sizing.

## Step 2: write the per-route limits before the run

For each route, a maximum regression rate, a maximum tool-selection divergence and a minimum
saving that makes routing it worth the risk. Routes that move money or touch security get zero
allowed regressions, zero tool divergence and zero argument drift. Deciding after you have seen the numbers is how a bad router gets approved, because a 66% saving
makes almost any regression look acceptable in hindsight.

## Step 3: run both models on every example

Run the strong model and the cheap model on every example, not only on the ones the router would
send cheap. With both outputs in hand you can score any routing policy offline: for each example,
the router's decision picks which of the two outputs it would have served. One paired run then
answers the question for every router and every threshold you are considering. RouteLLM's
evaluation works the same way, scoring routers from both models' results on each query.

Compare behavior, not just text: tool calls, arguments and structured fields per example, and read
every example that regressed. An LLM judge helps on open-ended answers, but
[know when to trust it](/blog/when-to-trust-an-llm-judge).

## Step 4: score the router's decisions, not just the cheap model

Two numbers per route. The **misroute rate** is the share of requests the router sent to the cheap
model on which the cheap model regressed against the strong one; that is the quality you are
giving up. The **missed-saving rate** is the share it kept on the strong model where the cheap one
would have held; that is money left on the table. A router with a low misroute rate overall and a
high one on the refund route is a router you can't ship as it is.

## Step 5: measure the blended cost on the same run

From the paired run, add up what the router's choices would have cost: for each example, the
measured cost of the output it would have served. Then add the router's fee per request, any
escalations, and a cache write for every turn where the chosen model differs from the previous
turn's. Compare the total with the always-strong baseline on the same inputs, not with the table
above. If it is much smaller than the arithmetic said, cache writes and extra turns are usually
why.

## Step 6: route per route, and re-run when anything moves

Send each route where it passed and keep the rest on the strong model. A static table from route
to model is the simplest router there is, and the easiest to test exactly. Then re-run the check
whenever the router's version, model pool or threshold changes, when either model changes
version, when the prompt or tools change, and when the traffic mix shifts. Watch the share of
traffic routed to the cheap model in production: when it moves without a change on your side, the
router or the traffic did.

## Doing this with EvalShift

This is the comparison EvalShift runs. It records real calls with its capture SDK and promotes them
into a frozen golden suite, and every distinct tag on an example becomes a slice, so tagging by
route gives you per-route results with no extra setup. `evalshift compare --from <strong-model>
--to <cheap-model>` replays the suite on both models, scores every pair with the evaluators you
configure (tool calls and arguments, structure, semantic similarity, an LLM judge), and measures
cost and latency from the run's own calls.

Per-route limits go in `migration_policy.slices` in `evalshift.yaml`, keyed by tag, and each one
overrides the top-level budget only where you set it:

```yaml
migration_policy:
  max_overall_regression_rate: 0.05
  max_tool_divergence: 0.05
  max_cost_increase: 0.0
  slices:
    refund:
      max_overall_regression_rate: 0.0
      max_tool_divergence: 0.0
      max_tool_argument_drift: 0.0
    security:
      max_overall_regression_rate: 0.0
      max_tool_divergence: 0.0
```

`max_cost_increase` caps a cost increase; it can't demand a saving, so read the minimum saving from
the report's cost delta. `evalshift init --profile cost-reduction` scaffolds budgets tighter than
the defaults. The report gives a verdict, the per-slice results and the per-example diffs, so the
routes that held become your routing table and the ones that broke stay on the strong model.

One limit: EvalShift compares two models, and it doesn't call your router. To score a learned
router, take its decision for each example and read the per-example results for the examples it
would have sent to the cheap model, as in Step 4. The
[step-by-step migration test guide](/blog/test-llm-model-migration-before-you-ship) shows the
whole flow, and the [migration policy docs](/docs/migration-policy) list every budget.

An LLM router's saving is arithmetic you can do before you build it. What it costs in quality is
something you can only measure, route by route, on your own traffic.

## FAQ

**What is an LLM router?** An LLM router is a layer between an application and the model APIs that
chooses which model handles each request, usually sending easy requests to a cheap model and hard
ones to a strong model. It can decide by fixed rules, by a learned classifier, by a platform's
quality estimates, or as a cascade that tries the cheap model first and escalates.

**How much does LLM routing save?** At most the share of calls routed times the price gap between
the two models. At November 2026 list prices, on prompts under 100,000 tokens, moving 70% of calls
from Claude Sonnet 5.5 to Claude Haiku 5.5 saves 66.5%, and moving 70% from `gpt-6.1-terra` to
`gpt-6-luna` saves 63%. Lost prompt caching, cascade escalations, extra turns and router fees
bring the measured saving below that.

**Does LLM routing reduce quality?** Usually somewhat, and the loss is uneven. Routers are scored
on benchmark averages, such as RouteLLM keeping 95% of GPT-4's MT Bench performance, while an
application loses quality in specific routes. In agents it shows up as wrong tool calls and claimed
actions that never happened, which a successful API response and fluent text can hide.

**What is the difference between an LLM router and an LLM gateway?** A gateway spreads requests
across deployments for availability, latency, rate limits and price, and falls back when one fails.
A router chooses the model by how hard each request is. Many products, including LiteLLM, do both.

**How do you evaluate an LLM router?** Replay 100 to 300 tagged production inputs on both the strong
and the cheap model, write per-route limits before the run, and score the router's decisions
against the per-example results: the misroute rate is the share of cheap-routed requests that
regressed. Measure the blended cost on the same run, and re-run when the router, the models or the
traffic change.

**Does LLM routing break prompt caching?** Switching models does. A prompt cache holds one model's
state for your prefix, so the first request after a switch writes the cache again at full or higher
price. A router that picks the model once per conversation, instead of per request, loses far less
of it.

**Is LLM routing worth it for AI agents?** Often, per route. Agents resend a long, cacheable prefix
every turn and fail in tool calls rather than prose, so per-request routing within a conversation
is the riskiest form. Testing each route on real traffic and routing only the routes where the cheap
model held keeps the saving on those routes and keeps out the failures the test can detect.

## Keep reading

- [The cheapest LLM API in 2026, and whether it is good enough for your agent](/blog/cheapest-llm-api)
  — the full price table and why list price is not what an agent costs.
- [The new model was 59% cheaper and 75% faster. I still wouldn't ship it.](/blog/cheaper-faster-and-still-a-fail)
  — the failing report, evaluator by evaluator.
- [This one passed: 71% cheaper, and the agent still does the same thing](/blog/what-a-passing-migration-proves)
  — per-slice budgets on a cheap swap that held.