LLM routing cuts frontier API costs by sending each request to the cheapest model that meets its quality bar. The strategies, from simplest to most involved, are static routing by task, rules, semantic routing, learned routers and cascades, with provider fallbacks for reliability. Move traffic to a cheaper model in shadow, canary and ramp stages, gated by an eval.
The price spread is what makes a router worth building. As of October 2026, OpenAI lists GPT-6 Astra at $10 per million input tokens and $50 per million output tokens, and GPT-6 Luna at $0.10 and $0.50. Anthropic lists Claude Opus 5 at $5 and $25, and Claude Haiku 4.5 at $1 and $5. Every request a router moves down a tier is billed at a fraction of the price.
This guide compares the strategies, shows how to combine them, and lays out the traffic-shifting playbook that holds quality steady while traffic moves. Routing is technique 3 in our ranking of 12 LLM cost optimization techniques, which also covers the cascade arithmetic. Our earlier pieces on the SLM-first 80/20 routing pattern and choosing between fine-tuned, API and RAG paths cover the architecture. For the management view of the same bill, see how to reduce AI token costs in your company.
What is LLM routing?
LLM routing is a layer between your application and its models that decides, per request, which model answers. The application sends a request along with what it knows about it. The router picks a target, such as a frontier tier, a cheaper tier, another provider or a custom model, and returns the answer in one consistent format.
Three separate decisions sit under the word "routing", and each runs on a different signal.
| Decision | Question it answers | Signal it uses |
|---|---|---|
| Routing | Which model gets the request first? | Task label, attributes or content |
| Escalation | What happens when the answer fails? | A check on the answer |
| Fallback | What happens when the provider fails? | Errors, timeouts and rate limits |
Keeping them apart makes each one testable on its own terms. A routing rule is judged on cost per task, an escalation check on how many bad answers it catches, and a fallback on how fast it recovers from an outage.
A router also has a cost of its own. Static rules add nothing per request, semantic and learned routers add an embedding or a classifier pass to every request, and a cascade adds a second API call to every escalation. That cost and latency come out of the saving the router produces, so the simplest strategy that captures most of the saving usually wins.
Which LLM routing strategies are there?
Six strategies cover almost every production router. They differ in what they decide on, what they add to each request, and how they fail.
| Strategy | Decides on | Added per request | Main risk |
|---|---|---|---|
| Static routing by task | Task label or endpoint | Nothing | One model for a whole task |
| Rules and heuristics | Request attributes | Nothing measurable | Rules drift as traffic changes |
| Semantic routing | Similarity to labeled examples | One embedding request | Near matches route wrong |
| Learned router | Classifier trained on outcomes | One classifier pass | Misroutes with no error raised |
| Cascade | A check on the cheap answer | An API call per escalation | Latency on escalated requests |
| Provider fallback | Errors and timeouts | Nothing until a failure | Fallback quality differs |
1. Static routing by task
Give each step of a workflow its own route and fix the model per route in configuration. A support pipeline might classify the ticket, extract the order details, draft a reply and summarize the case for a human agent. Each of those is a separate route with its own model, its own eval and its own cost per task.
# One route per workflow step. Each route has its own eval and its own budget.
ROUTES = {
"classify_ticket": {"model": "gpt-6-luna", "fallback": "claude-haiku-4-5"},
"extract_order": {"model": "claude-haiku-4-5", "fallback": "gpt-6.1-sol"},
"draft_reply": {"model": "claude-sonnet-5-5", "fallback": "gpt-6.1-sol"},
"case_summary": {"model": "claude-opus-5", "fallback": "gpt-6-astra"},
}
def route(task: str) -> dict:
return ROUTES[task]The model names are illustrative. The cheapest passing model for each step comes from running that step's eval against every candidate.
Static routing captures a large share of the saving for the least risk. The application already knows which step it is on, so nothing has to guess, and each route moves to a cheaper model on its own evidence. Its limit is granularity: every request on a route goes to the same model, the occasional hard one included, which is why static routing pairs well with a cascade inside each route.
2. Rules and heuristics
Rules route on attributes the application can read before any model runs: input length, language, document type, customer tier, and whether attachments or tools are involved. A rule such as "inputs over 50,000 tokens go to the model whose long-context quality you have tested" is cheap to run and easy to debug.
Every rule encodes an assumption about which requests are hard, and traffic changes underneath it. Log which rule fired on every request, check each rule against eval results for the slice of traffic it governs, and keep the rule set short enough that one engineer can read all of it.
3. Semantic routing
Semantic routing embeds the incoming request and compares it with labeled example sets, one per route. The request goes to the route whose examples it sits closest to, provided the similarity clears a threshold, and to a default route when it clears none.
It suits a single entry point, such as a chat box or an inbound email address, that receives many kinds of request. The cost is one embedding request and its latency on every request. The risk is the near match: a request that resembles a routine one and turns out to need something else. Set the default route to a frontier model, keep the threshold strict, and treat anything below it as out of scope for the cheaper routes.
4. Learned routers
A learned router is a classifier that estimates, per request, whether a cheaper model's answer will be acceptable. It is trained on logged outcomes from your own traffic: requests where the cheaper model's answer passed and requests where it failed.
Published results on benchmarks are encouraging. RouteLLM, from LMSYS researchers in 2024, trained routers on human preference data to choose between a stronger and a weaker model. It reported cost reductions of over 2 times in certain cases on widely recognized benchmarks without compromising response quality, and its routers kept their performance when different strong and weak models were substituted at test time.
On your own traffic, any router's estimate will be wrong on some requests, and a misroute produces a worse answer with no error to flag it. Train a router only once you have labeled outcomes from your own traffic, keep a check on the answer behind it, and audit a sample of its decisions against the eval every week.
5. Cascades with validation-based escalation
A cascade sends every request on a route to the cheaper model first, checks the answer, and escalates to a stronger model when the check fails. A cascade is only as good as its check, and the strongest checks are the ones code can run.
- Schema checks: the output parses, required fields are present, and enum values fall in the allowed set.
- Consistency checks: extracted totals add up, cited IDs exist in your database, and dates fall in range.
- Scope checks: the input belongs to the task the cheaper model was evaluated on.
- Confidence signals: token probabilities or a self-reported score. These are weaker than the checks above, so use them as one input among several.
- A judge model: a second model grades the answer. It catches more, and its cost lands on every request it grades.
FrugalGPT, from Stanford researchers in 2023, set out the LLM cascade as one of three strategies for cutting inference cost and learned which combination of models to use for each query. In its experiments it matched the best individual model of the time, GPT-4, with up to 98% cost reduction.
The arithmetic is in technique 3 of our ranking. With a cheaper model at 5% of the strong model's price and 30% of requests escalating, the cascade costs 0.35 of sending everything to the strong model. The escalation rate decides the saving, and escalated requests also wait for two models, so cascades suit work where an occasional slower answer is acceptable.
Hard cases reach the frontier model through this mechanism, decided on evidence available at the time of the request. A request counts as hard when its answer fails a check, when it falls outside the route's scope, or when its task label sends it to the frontier route directly.
6. Fallbacks across providers
Fallbacks handle the provider failing. They belong in a cost guide because a cheaper route with no fallback is a single point of failure, and the first outage sends the team back to the expensive default for good. Anthropic's API error reference lists a 429 rate limit error and a 529 overloaded error, which it says can occur when the API sees high traffic across all users. Every provider has equivalents.
Give every route a fallback model at a second provider, with retries and backoff before the switch and a circuit breaker that stops sending to a provider that keeps failing. Two details catch teams out.
- The fallback model has to pass the route's eval too. A fallback that was never evaluated on the task trades a visible outage for a quality drop nobody sees.
- Prompt caches stay with their provider. A prefix cached at one provider loses its cache discount when the request moves to another, so budget for a cost spike during a failover.
The same reference advises ramping traffic up gradually to avoid hitting acceleration limits after a sharp increase in usage. That is one more reason to move traffic in stages, as the playbook below does.
How do you combine routing strategies?
Most production routers layer three of the six. Static routing by task picks the route, a cascade inside each route escalates the answers that fail their checks, and fallbacks wrap every model with a backup at a second provider.
Add semantic routing in front only when one entry point receives many kinds of request. Add a learned router to a route only when its volume is high enough, and its labeled outcomes plentiful enough, that a better first guess saves more than the router costs.
| Your situation | Start with |
|---|---|
| Distinct steps in a workflow | Static routing by task |
| One entry point, many request types | Semantic routing with a frontier default |
| Answers that code can check | A cascade inside the route |
| High volume with labeled outcomes | A learned router behind a check |
| One provider behind every route | Fallbacks before anything else |
How do you move traffic to a cheaper model safely?
Move traffic in stages and let a gate decide when each stage advances. The stages are the same whether the cheaper target is a lower frontier tier, another provider's model or a custom model: offline eval, shadow, canary, ramp, then steady state with continuous scoring.
| Stage | Who sees the candidate's answers | Gate to advance |
|---|---|---|
| Offline eval | Nobody; it answers replayed requests | Meets the bar on the eval set |
| Shadow | Nobody; it is scored in the background | Holds the bar on live traffic |
| Canary | A small share of users | No rise in escalations or complaints |
| Ramp | A growing share of users | Every gate holds at each step |
| Steady state | Every request on the route | Continuous scoring stays above the bar |
Offline eval. Replay a held-out sample of logged requests through the candidate and score it against the route's eval. This is the cheapest stage, and it rules out most candidates.
Shadow. Send a copy of live requests to the candidate in the background while the incumbent keeps answering. Users never see the candidate's output. Score both answers on the same requests, which shows how the candidate handles live inputs your eval set never anticipated. Shadow traffic is billed twice, so mirror a sample when the route is large.
Canary. Serve a few percent of live requests from the candidate, with the cascade's checks between it and the user. An answer that fails a check falls back to the incumbent frontier model inside the same request, so a weak candidate shows up as a rise in escalations and latency while users still get an answer that passed.
Ramp. Raise the share in steps, holding each step long enough to see a full cycle of your traffic, such as a week that includes a month-end. Advance only when every gate holds.
Steady state. Keep scoring a sample of the candidate's live answers against the eval, and keep the rollback trigger armed. A route can degrade weeks after a ramp, when the inputs drift or a model alias starts pointing at a new version.
Make rollback automatic and per route. A rise in the escalation rate above its baseline, an eval score below the bar, p95 latency over the route's budget or a spike in the fallback rate should each send the route's traffic back to the incumbent while someone investigates.
What should you measure on each route?
Measure every route separately, because a router's averages hide the one route that is losing money or quality. Log one record per API call with the route, the stage and the outcome.
log.info("api_call", extra={
"task_id": task_id,
"route": "extract_order",
"stage": "canary", # offline, shadow, canary, ramp or steady
"model": model,
"input_tokens": usage.input_tokens,
"output_tokens": usage.output_tokens,
"cost_usd": cost,
"check_passed": passed,
"escalated": not passed,
"latency_ms": elapsed_ms,
})Then roll the records up per route and per stage.
| Metric | What it tells you | Act when |
|---|---|---|
| Escalation rate | Share the cheaper model cannot finish | It rises above baseline |
| Cost per successful task | Unit cost with retries and escalations | It exceeds the route's target |
| Eval score | Whether the bar still holds | It drops below the bar |
| p95 latency | Wait time, escalations included | It exceeds the route's budget |
| Fallback rate | Provider health | It spikes |
Cost per successful task decides whether a route earns its place. A cheaper model with a high escalation rate pays for two models on every escalated request, and a retry pays for the same request twice. Our guide to cost per task and AI unit economics covers how to set the target each route is held to, and agentic AI cost optimization applies the same per-step view to agents, where every step of a run is a routing decision.
Where does a custom model fit in a routing strategy?
A custom model is one more routing target, and it slots into the strategies above unchanged. Static routing gives a high-volume task its own route, the custom model becomes the cheaper target on that route, the frontier model stays behind it as the escalation path, and the traffic-shifting playbook moves the route across.
It pays where every other target still leaves a large line on the bill: one task at high, steady volume, where even the cheapest frontier tier that passes the eval costs more than anything else you run. A model trained for one job carries only the capability that job needs, so it costs less to serve and answers faster. Our breakdown of when a custom model beats a frontier API on cost works through the break-even.
In our QMSum study, a 1.2B model trained for query-focused meeting summarization had an estimated serving cost of $0.00053 per query, against $0.104 for Claude Opus 5 and $0.0130 for GPT-5.6 Luna at synchronous list prices. It answered in 0.39 seconds on a single GPU against 9.0 seconds for Claude Opus 5 over the API. The estimate excludes training cost, and quality is task-specific, which is why the route moves only after the custom model passes your evals.
The adaptive model endpoint packages that route for you. An adaptive model endpoint is one API endpoint for one of your tasks: frontier models answer at the start, a custom model trained on your requests and your team's feedback takes over once it passes your evals, and hard cases stay on frontier models.
With the whole route behind one endpoint, your router keeps a single entry for the task however many versions ship behind it.
It runs the playbook above on that one route. Each new version of the custom model runs in the background on real requests and is scored against your evals, which is the shadow stage. Nothing switches without your sign-off, and once a version passes, a growing share of requests moves to it, which is the canary and the ramp. Hard cases keep going to frontier models. We monitor the model and retrain it as your task changes, and every new version goes through the same gate. The endpoint is currently invitation only, and we are developing a more streamlined version for wider access.
For your router, adopting it means pointing one route at a new model name. The model runs Ertas-managed, in your cloud or on-premise, and you own it.
Frequently Asked Questions
What is an LLM router?
An LLM router is a layer between an application and its models that picks which model answers each request. It can decide on a task label, request attributes, similarity to labeled examples or a trained classifier. Most routers also handle escalation when an answer fails a check, and fallback when a provider fails.
Does an LLM router add latency?
Static routing and rules add almost nothing, because they read information the application already has. Semantic routing adds an embedding request and a learned router adds a classifier pass. A cascade adds the most on escalated requests, which wait for both the cheaper model and the stronger one.
Is a learned router better than routing rules?
A learned router can make finer decisions than rules once it has labeled outcomes from your traffic to learn from, and RouteLLM reported cost reductions of over 2 times in certain cases on benchmarks in 2024. Rules are cheaper to run and easier to debug. Start with static routing by task, and add a learned router to a route only when its volume justifies it.
How do you route between OpenAI and Anthropic models?
Put one interface in front of both, normalize the response format, and give each route a primary model and a fallback at the other provider. Run every model a route can reach against that route's eval, the fallback included, and expect prompt cache discounts to reset when traffic moves from one provider to the other.
What is the difference between LLM routing and an LLM cascade?
Routing picks a model before any model runs, from what is known about the request. A cascade runs the cheaper model first and escalates to a stronger one when the answer fails a check. Most production systems use both: routing to choose the route, and a cascade inside it.
How do you move traffic to a cheaper model without hurting quality?
In stages: an offline eval on replayed requests, shadow scoring on live traffic, a canary on a small share of users, then a ramp in steps. Gate each stage on the eval score, the escalation rate and latency, keep the frontier model as the fallback inside every request, and roll back automatically when a gate fails.
The Short Version
Route by task first, because the application already knows which step it is on. Put a cascade inside each route where code can check the answer, give every model a fallback at a second provider, and add semantic or learned routing only where the volume pays for it. Move traffic to any cheaper target in stages, gated by an eval, and measure cost per successful task on every route.
If one route carries most of your token bill, book a call and we will scope it with you.
Sources: OpenAI API pricing, Anthropic pricing, Anthropic API errors, RouteLLM (Ong et al., 2024) and FrugalGPT (Chen, Zaharia and Zou, 2023). Prices verified 6 October 2026.