The cheapest token is the one you send to the right-sized model. Routing is how you stop paying frontier prices for routine work.
TL;DR
- Model routing directs each request to the smallest model that can handle it.
- Reserve frontier models for hard reasoning; send routine work to cheap, fast ones.
- Route on task type, complexity signals, or a cheap classifier — and verify with evals.
- It is one of the highest-leverage cost levers in any LLM app.
The problem with one model for everything
It is tempting to wire your whole product to a single frontier model — one integration, consistent behavior. But most production traffic is not hard. Classifications, extractions, formatting, and short summaries dominate volume, and a frontier model is wildly overpowered for them. You end up paying premium prices for work a model costing a fraction as much would do just as well.
How routing works
A router sits in front of your models and decides where each request goes. Three common strategies, in increasing sophistication:
- Static routing by task. You know the task type up front, so you hardcode the mapping: summaries → small model, architecture → frontier model.
- Signal-based routing. Use cheap heuristics — input length, presence of code, complexity keywords — to pick a tier.
- Classifier routing. A tiny, cheap model first classifies the request's difficulty, then routes accordingly.
Request ─▶ Router ─┬─ trivial ─▶ small model (cheap, fast)
├─ standard ─▶ mid model
└─ hard ─▶ frontier model (expensive, smart)
Spend your token budget like a real budget: the most on the few decisions that matter, the least on the many that don't.
A worked example
Imagine a support assistant handling 100k requests a day. Most are FAQ-style and a small model answers them perfectly. Maybe 5% are genuinely complex and need the frontier model. Routing the easy 95% to a model costing a fraction as much can cut spend dramatically — often by more than half — with no measurable quality loss on the easy traffic and better quality on the hard traffic, because you can now afford the best model where it counts.
Guardrails so routing doesn't bite you
Routing trades a little risk for a lot of savings. Keep it honest:
- Eval each tier on representative traffic so you know the small model genuinely clears the bar for what you send it.
- Build an escalation path. If a small model's answer fails a confidence or validation check, retry on a bigger one.
- Monitor drift. Re-check routing decisions as your traffic mix and the models change.
The upfront-investment payoff
Routing is classic high-CapEx, low-OpEx engineering: it takes work to set up the tiers, the classifier, and the evals, but it pays back every single request thereafter. As models proliferate and volumes grow, the teams that route deliberately run circles — economically — around the teams that send everything to the biggest model and hope.