Model router

A model router is the component that picks which model serves each incoming request, making that choice once per request and before the call goes out to any provider. It reads something cheap about the work in front of it: the prompt text, a task label the application already carries, or a small classifier’s score. It then dispatches to one member of a candidate set spanning at least one expensive tier and one cheap one. The choice has to vary with the request, because a dispatcher that names the same model every time is a configuration line with a routing table drawn around it.

Because the decision is taken per request, a router’s quality is a property of a traffic mix rather than a property of any model in its set. Score the same router on two request distributions and you get two cost curves and two pass rates, since the share of work it hands to each tier moves with the input.

A routing result is therefore only reportable with its mix attached: what fraction of requests went to each tier, over what denominator, drawn from what sample of real traffic.

The evidence under this entry is our own pre-registered routing study, what reaching up the model tiers actually bought on checkable work. On short, machine-checkable tasks it measured no capability separation between Anthropic’s Opus 4.8 and Fable 5 at either effort level tested, an inconclusive null on non-refused samples of 4 and 7 trials. That is the situation a router exists to exploit: where the expensive tier buys nothing you can measure on a slice of traffic, the routing decision is the only place a saving can come from.

The bands a router sorts models into are capability tiers, the ordered list it walks when a call comes back unusable is a fallback chain, and the signal that most often forces a re-route is a model’s refusal rate. Which kind of task belongs on which band is separate work from building the dispatcher, and the case for rarely picking the model at the top covers that half.

Evaluating a router against the models it can choose

Run one request set through the router, then through each fixed model it is allowed to choose, and compare the arms on pass rate and cost per request together. Every arm sees the same items, which makes the comparison paired, so a paired test is the correct instrument. The McNemar test for paired eval runs answers whether a routed pass rate differs from the always-expensive baseline by more than run-to-run noise. Report three quantities beside the headline: the share of requests each tier received with its denominator, a confidence interval on every arm’s pass rate, which the Wilson and Clopper-Pearson calculator produces from k of n, and the run’s answer coverage. That third one earns its place because a refused or truncated request leaves the denominator without announcing itself, and a router changes how often that happens.

Hu et al.’s RouterBench, a 2024 arXiv preprint, places a router on a cost-quality curve instead of a single point, and publishes over 405,000 recorded inference outcomes so that curve can be redrawn without re-running the models. Ong et al.’s RouteLLM, from the same year, reports routers trained on human preference data cutting cost by over two times in certain cases with no quality loss they could measure on their benchmarks, which is the scale of saving that makes the paired comparison worth running.

Model router vs fallback chain

A fallback chain is the ordered list of models a system tries after a call has already failed, refused, or timed out, walking down the list until one of them returns something usable. It acts on an attempt that went wrong. A model router acts before any attempt exists, choosing which model receives the request in the first place.

A system missing either one pays for it, and the two bills are different. A router with nothing behind it hands a refused or rate-limited request straight back to the caller, so the routing choice can be perfectly correct while the request still fails. A chain with nothing in front of it sends every request to the same first model, so cheap traffic pays the premium tier’s latency and price on its way to an answer a smaller model would have produced. What the system serves once the chain is exhausted is graceful degradation, a separate question from routing.

The sequential-only version of this idea has a literature of its own. Chen, Zaharia and Zou’s FrugalGPT, a 2023 arXiv preprint, calls it an LLM cascade and reports matching OpenAI’s GPT-4 on their benchmark tasks at up to 98% lower cost by querying cheap models first and escalating when a scored confidence falls short. Most production systems carry both, and the chain is one of the levers a budget for failure in a production agent spends deliberately. Count chain activations as a reading on the router in front of it, since the chain only fires on requests the first choice could not serve.

Model router vs the router inside a mixture of experts

Mixture-of-experts architectures contain a component with the same name doing a different job. Shazeer et al.’s 2017 paper introduced a trainable gating network that selects a sparse combination of expert sub-networks for each example, and the work that followed, including the 2021 Switch Transformer paper, describes that selection step as routing. That router runs inside a single model’s forward pass. It decides per token, picks among sub-networks that share weights and a tokenizer, and is trained jointly with the model it lives in.

A model router in the sense this entry defines sits outside every model, decides per request, and chooses among separately trained and separately billed APIs. That one is configured by an engineer rather than learned from data. The gating sense is the older and settled one, and the sense this entry defines is what an engineer configuring a deployment means.

The ambiguity surfaces when you read a diagram. A router drawn inside one model’s architecture is the gate, a router drawn in a serving topology is the dispatcher, and a claim about per-token sparsity looks much like a claim about per-request cost on the page. Check which side of the API boundary the box sits on.

A routing decision and the model that answered are separate facts

A routing decision is a plan the request carries, and the response is the only record of what actually served it. Server-side fallback can substitute a different model inside a single API call, so the model your router selected and the model that produced the tokens can differ with nothing in the application marking the swap. In the study above, the premium tier’s refusal rescue fired on 20 of 28 low-effort calls, every rescued answer came back from the cheaper Opus 4.8, and the pass rate with rescue enabled reached 96.4% (27 of 28, 95% CI 82.3–99.4%). In that arm, most of the traffic addressed to the expensive tier was served by the inexpensive one at premium price.

Record the model name the provider returns on each response rather than the one you asked for, a discipline that logging refusals and silent model fallbacks sets out call by call. Without that field, a routing evaluation is scoring a plan.

A router and the chain behind it decide together which model your traffic actually reaches, one choosing before the attempt and one after it failed, so a change to either moves the same numbers and both belong in one evaluation. The policy a router executes, worked out task by task and priced against what reaching up a tier costs, lives in the routing guide built on that study. An application whose traffic is uniformly hard won’t get a saving out of any of this, because every request would route to the expensive tier anyway and the router’s own latency and failure surface get added for nothing.