LLM routing · Inference cost

RouteWise

How much can a simple learned router save by sending each prompt to the cheapest model likely to get it right?

Saidur Rahman PulokIndependent projectHELM Capabilities & Lite2026

TL;DR

RouteWise measures LLM routing offline on Stanford HELM’s per-question results, with real token counts and list prices. The saving depends heavily on how good the cheap models already are. On 2025 models, a router matches the best single model for about 25–30% less. On 2024 models, where cheap models had big blind spots, it matches GPT-4o for 94% less.

  • 79.6% vs 76.5%best routed accuracy vs. best single model (Qwen3 235B), 2025 suite
  • −25–30%cost to match Qwen3 235B on 2025 models
  • −94%cost to match GPT-4o on 2024 models
Cost against accuracy for 13 single models and four routers on HELM Capabilities
Figure 1. Cost per 1,000 prompts (log scale) against accuracy on 2,987 HELM Capabilities prompts, 5-fold cross-validated. Dots are single models; the dashed line is the best random mix of two single models; the star is an oracle that always picks the cheapest correct model.

Abstract

Routing between LLMs is a common way to cut serving costs, but published savings often depend on an unstated baseline. RouteWise is a small, honest version that can be rerun: real prompts, real per-model outcomes, cross-validation, confidence intervals, and the single-model baselines that make savings look smaller. Stanford’s HELM has already asked every model the same questions and scored each answer, so all RouteWise adds is prices and routing.

Four simple routers estimate the probability that each model answers each prompt correctly, and a shared rule trades that off against expected cost. On 13 models released in 2025, open-weight Qwen3 235B is the strongest single model, and routing saves a modest 25–30% at its accuracy while reaching 79.6% when cost does not matter. On 10 models from 2024, cheap models were strong on some tasks and weak on others, and the same routers match GPT-4o for 94% less. A claim like “routing saves 90%” only means something next to the best single model it is compared with.

Method

HELM resultsper question, per model List pricesper 1M tokens Routing tablecorrect?, tokens, cost Routerp(correct | prompt, model) Selection rulemax p − λ · expected cost Sweep λcost–accuracy curve

Data

Two HELM leaderboards publish every question, each model’s answer, the token counts and whether the answer was scored correct. RouteWise keeps the prompts every model in the pool answered.

SuiteTasksPromptsModels
HELM CapabilitiesMMLU-Pro, GPQA, IFEval, Omni-MATH2,98713, released 2025
HELM LiteGSM8K, MATH, MMLU, MedQA, LegalBench, OpenBookQA5,55110, released 2024

Prices are public list prices; open-weight models use Together AI’s prices, since HELM ran them through Together.

Routers

RouterWhat it uses
Task-awareEach model’s training accuracy on the prompt’s task
kNN (TF-IDF)The 25 most similar training prompts, smoothed toward the task average
Logistic (TF-IDF)One logistic regression per model on TF-IDF features and the task label
Logistic (text only)The same, with no task label anywhere, including the cost estimate
  • Selection: every router shares one rule, choosing the model that maximises p_correct − λ · expected_cost; sweeping λ traces a cost–accuracy curve. Expected cost uses only what is known before sending: the prompt’s real token count and the model’s average output length on training prompts.
  • Evaluation: 5-fold cross-validation stratified by task, so no router sees the prompt it routes. Savings carry 95% bootstrap intervals over prompts, framed by the best mix of two single models and an oracle.
  • Missing output tokens: HELM records zero output tokens for every Gemini and DeepSeek-V3 answer, which would make them look almost free. RouteWise imputes the median output length of the other models on the same prompt.
  • Hidden reasoning tokens: reasoning models such as GPT-5, o3, Grok 4 and Gemini 3 Pro bill for thinking tokens HELM doesn’t record, so their costs can’t be recovered and they are left out of the pool.

Results: 2025 models

HELM Capabilities, 13 models including Claude 4.5, GPT-4.1, Llama 4, Qwen3, Kimi K2 and gpt-oss. The best single model is the open-weight Qwen3 235B Instruct at 76.5% for $1.85 per 1,000 prompts.

RouterBest accuracy ↑$ / 1k to match Qwen3 ↓Saving vs Qwen3 (95% CI)
Task-aware79.6%1.3029.8% (−0.3 to 40.6)
kNN (TF-IDF)78.6%1.4819.7% (−1.0 to 26.7)
Logistic (TF-IDF)78.4%1.3924.9% (5.4 to 32.6)
Logistic (text only)77.9%1.5914.0% (4.2 to 19.8)
Oracle (upper bound)92.1%0.77

Table 1. Routers on HELM Capabilities, 5-fold cross-validated. Best value per column in bold.

Share of prompts the task-aware router sends to each model when matching Qwen3 235B
Figure 2. Where the task-aware router sends prompts when matching Qwen3 235B: 48% to gpt-oss-120b, 33% to Qwen3 and 18% to Llama 4 Maverick.

Results: 2024 models

HELM Lite, 10 models including GPT-4o, Claude 3.5, Gemini 1.5, Llama 3.1, Qwen2.5 and DeepSeek-V3. GPT-4o is the most accurate single model at 79.2% for $3.35 per 1,000 prompts.

RouterBest accuracy ↑$ / 1k to match GPT-4o ↓Saving vs GPT-4o (95% CI)
Task-aware82.5%0.24492.7% (92.2 to 93.9)
kNN (TF-IDF)82.9%0.18794.4% (93.3 to 94.9)
Logistic (TF-IDF)83.6%0.18894.4% (93.9 to 95.1)
Logistic (text only)83.4%0.18794.4% (93.8 to 95.0)
Oracle (upper bound)94.5%0.184

Table 2. Routers on HELM Lite, 5-fold cross-validated. Best value per column in bold.

Cost against accuracy for 10 single models and four routers on HELM Lite
Figure 3. Cost against accuracy on 5,551 HELM Lite prompts.

What the two suites say together

A router saves the most when cheap models are strong on some tasks and weak on others. That was true of 2024 models, where Gemini 1.5 Flash scored 93% on MATH but 33% on GSM8K (partly because of HELM’s answer format). Among 2025 models, cheap open-weight models are strong across the board, so the cheapest good choice is often obvious and a router’s job is smaller.

A claim like “routing saves 90%” only means something next to the best single model it is compared with.

Are the probabilities trustworthy?

The selection rule trusts each router’s probability that a model gets a prompt right, so the out-of-fold predictions were checked for calibration. All four routers are well calibrated: when a router says 70%, the model is right about 70% of the time, and expected calibration error is below 0.03 everywhere.

Better probabilities don’t automatically mean better routing, though. On the 2025 suite the TF-IDF logistic router has the best Brier score (0.188 against 0.202 for task-aware), but it doesn’t route better.

Calibration of the four routers on the 2025 suite
Figure 4. Reliability of the routers’ out-of-fold probabilities on HELM Capabilities.

Related work

RouteWise doesn’t propose a new routing method; it is a small, reproducible evaluation of simple ones on public data. The difference is the data: HELM’s per-question results for current models, with real token counts, plus the corrections those counts need.

Caveats and next steps

Caveats

  • List prices, not your bill: batch discounts, caching and provider choice all move them.
  • Imputed output lengths make Gemini and DeepSeek-V3 costs estimates, not measurements.
  • HELM’s prompts and scoring, not production traffic; some scores reflect answer formatting more than ability.
  • Most routers use the task label, which real traffic rarely has.
  • One run per model with binary correctness; the matched point is picked on the evaluation curve, so it is slightly optimistic.

What comes next

  • Reasoning models, using a source that records thinking tokens.
  • A stronger label-free router, using sentence embeddings or a small fine-tuned encoder in place of TF-IDF.
  • A cascade that tries a cheap model first and escalates on low confidence.

Run it

Requires Python 3.11+ and uv. Runs on a laptop in a few minutes; no GPU or API keys needed.

git clone https://github.com/saidurpulok/routewise && cd routewise
uv sync --locked --group dev
uv run routewise fetch   # about 260 MB of HELM files for the 2025 suite
uv run routewise run     # cross-validates the four routers
uv run routewise plot    # redraws the plots from results.json

Citation

@software{pulok2026routewise,
  author  = {Pulok, Saidur Rahman},
  title   = {RouteWise: Cost-Aware LLM Routing Evaluated on HELM},
  year    = {2026},
  version = {0.1.0},
  url     = {https://github.com/saidurpulok/routewise}
}