Your router does not know your evals

A model router looks intelligent because it can read a prompt and name a model. That is not enough.

The router has not seen your failure cases. It does not know which mistakes your users tolerate, which ones break the product, or where a smaller model quietly matches a frontier model. Unless it knows your evaluation data, its confidence is mostly borrowed from somebody else's benchmarks.

Routing is an evaluation problem before it is a model-selection problem.

The expensive default

The safest-looking architecture is simple: send every request to the best frontier model you can buy.

It feels responsible. Nobody wants to explain that a weaker model caused a bad answer. The frontier model becomes insurance against the fear of choosing wrong.

But this is not an engineering argument. It is a fear argument.

A model like GPT-5.6 Sol may be the right answer for a difficult slice of the workload. That does not make it the right answer for every extraction, classification, rewrite, support question, or routine code edit. Model capability is not one scalar that transfers cleanly to every task. Paying for the strongest general model on every request can hide the fact that much of the workload never needed it.

The frontier labs benefit when "best model" and "default model" become the same decision. Your cost curve does not.

Why a generic router cannot know

Most routing products start with model descriptions, public benchmarks, prompt features, or a separate LLM that guesses how hard the request looks. Those are useful priors. They are not evidence about your application.

A prompt can look easy and still sit on a product-specific fault line. Another can look complicated while a small model handles it reliably because the output pattern is narrow and repetitive.

The missing variable is your eval set.

A router should learn from examples that reflect your actual traffic and your definition of a good answer. Without that, it cannot know:

• which errors carry real product cost
• where two models are functionally tied
• when style matters more than factual depth
• which requests need tools, long context, or stronger reasoning
• how quality changes across languages, customers, and edge cases

A generic router can estimate difficulty. Only your evaluations can estimate fitness.

Ask the models instead of guessing

The first version of a useful router does not need to be clever. It needs data.

Take a representative sample of requests and send each one to several candidate models. Keep the full result: answer, latency, token usage, tool behavior, and cost. Then score the answers against the criteria that matter for the product.

At small scale, a subject matter expert can compare them directly. At larger scale, a strong judge model can produce an initial ranking and explain its decision. The judge should not become an unquestioned source of truth. Have domain experts agree or disagree with a sample of its judgments, especially on close calls and high-risk tasks.

That review does two jobs. It catches judge bias, and it turns vague preferences into a usable rubric.

The resulting table is much more valuable than a leaderboard:

request features → model outcome → quality score → latency → cost

Now routing becomes a supervised decision problem.

Optimize for the cheapest model that clears the bar

The objective is not to predict which model is "best" in the abstract. It is to choose the cheapest model likely to satisfy the quality requirement for this request.

One simple formulation is:

route(x) = argmin_m cost(m, x), subject to quality(m, x) ≥ τ

Here, x is the request, m is a candidate model, and τ is the minimum acceptable quality for that task. The threshold does not have to be universal. A brainstorming feature and a financial workflow should not share the same tolerance for error.

The router itself can be small. Depending on the traffic, it might be a classifier over prompt embeddings and metadata, a calibrated scoring model, or a compact LLM trained to predict the evaluation outcome. The hard part is not the routing architecture. The hard part is producing labels that represent reality.

This is where many teams reverse the order. They buy a router, turn it on, and hope its idea of difficulty matches their application. I would rather build the eval set first and let the routing strategy fall out of the results.

The escalation path matters

A router should be allowed to admit uncertainty.

If the predicted quality margin is thin, route upward. If the request is outside the training distribution, route upward. If a cheap model produces a low-confidence or malformed result, retry with a stronger model. High-risk categories can bypass the router entirely.

That creates a cascade:

1. Start with the cheapest model that has earned trust on this slice.
2. Validate what can be validated automatically.
3. Escalate when confidence, structure, or policy checks fail.
4. Send genuinely hard or unfamiliar requests to the frontier model.

The frontier model still matters. It moves from universal default to targeted escalation.

Make sure the router generalizes

A router can overfit just like any other model. Randomly splitting near-duplicate prompts into train and test sets will produce a comforting number and a brittle system.

Hold out entire task families, customers, time windows, or prompt templates. Test on traffic collected after the training period. Track performance separately for rare but expensive failures. Re-run the comparison when candidate models change, because a route learned against last month's model versions can become stale quickly.

Also measure the whole system, not only routing accuracy. A router that saves model cost but adds too much latency, judge overhead, or operational complexity may not save anything.

The useful metric is closer to quality-adjusted cost under real traffic:

total model cost + routing overhead + expected failure cost

That final term is why the cheapest model does not automatically win.

Evaluation is the control plane

The deeper lesson is not that every company needs a custom routing model. Some workloads are too small or too high-risk for routing to matter. The lesson is that model choice should be earned by evidence from the workload.

If you have enough volume, run the comparison. Ask multiple models the same questions. Use a judge to scale the first pass, then calibrate that judge with people who understand the domain. Train a router on the disagreements, not on marketing categories like "fast," "smart," or "best."

You may find that the frontier model wins more often than expected. You may also find large regions where it does not win by enough to justify its price. Both outcomes are useful because both replace fear with measurement.

A router that has never seen your evals does not know where to send your request. It is guessing with confidence.