TL;DR
RouteWise measures LLM routing offline on Stanford HELM’s per-question results, with real token counts and list prices. The saving depends heavily on how good the cheap models already are. On 2025 models, a router matches the best single model for about 25–30% less. On 2024 models, where cheap models had big blind spots, it matches GPT-4o for 94% less.
- 79.6% vs 76.5%best routed accuracy vs. best single model (Qwen3 235B), 2025 suite
- −25–30%cost to match Qwen3 235B on 2025 models
- −94%cost to match GPT-4o on 2024 models
Abstract
Routing between LLMs is a common way to cut serving costs, but published savings often depend on an unstated baseline. RouteWise is a small, honest version that can be rerun: real prompts, real per-model outcomes, cross-validation, confidence intervals, and the single-model baselines that make savings look smaller. Stanford’s HELM has already asked every model the same questions and scored each answer, so all RouteWise adds is prices and routing.
Four simple routers estimate the probability that each model answers each prompt correctly, and a shared rule trades that off against expected cost. On 13 models released in 2025, open-weight Qwen3 235B is the strongest single model, and routing saves a modest 25–30% at its accuracy while reaching 79.6% when cost does not matter. On 10 models from 2024, cheap models were strong on some tasks and weak on others, and the same routers match GPT-4o for 94% less. A claim like “routing saves 90%” only means something next to the best single model it is compared with.
Method
Data
Two HELM leaderboards publish every question, each model’s answer, the token counts and whether the answer was scored correct. RouteWise keeps the prompts every model in the pool answered.
| Suite | Tasks | Prompts | Models |
|---|---|---|---|
| HELM Capabilities | MMLU-Pro, GPQA, IFEval, Omni-MATH | 2,987 | 13, released 2025 |
| HELM Lite | GSM8K, MATH, MMLU, MedQA, LegalBench, OpenBookQA | 5,551 | 10, released 2024 |
Prices are public list prices; open-weight models use Together AI’s prices, since HELM ran them through Together.
Routers
| Router | What it uses |
|---|---|
| Task-aware | Each model’s training accuracy on the prompt’s task |
| kNN (TF-IDF) | The 25 most similar training prompts, smoothed toward the task average |
| Logistic (TF-IDF) | One logistic regression per model on TF-IDF features and the task label |
| Logistic (text only) | The same, with no task label anywhere, including the cost estimate |
- Selection: every router shares one rule, choosing the model that maximises
p_correct − λ · expected_cost; sweeping λ traces a cost–accuracy curve. Expected cost uses only what is known before sending: the prompt’s real token count and the model’s average output length on training prompts. - Evaluation: 5-fold cross-validation stratified by task, so no router sees the prompt it routes. Savings carry 95% bootstrap intervals over prompts, framed by the best mix of two single models and an oracle.
- Missing output tokens: HELM records zero output tokens for every Gemini and DeepSeek-V3 answer, which would make them look almost free. RouteWise imputes the median output length of the other models on the same prompt.
- Hidden reasoning tokens: reasoning models such as GPT-5, o3, Grok 4 and Gemini 3 Pro bill for thinking tokens HELM doesn’t record, so their costs can’t be recovered and they are left out of the pool.
Results: 2025 models
HELM Capabilities, 13 models including Claude 4.5, GPT-4.1, Llama 4, Qwen3, Kimi K2 and gpt-oss. The best single model is the open-weight Qwen3 235B Instruct at 76.5% for $1.85 per 1,000 prompts.
| Router | Best accuracy ↑ | $ / 1k to match Qwen3 ↓ | Saving vs Qwen3 (95% CI) |
|---|---|---|---|
| Task-aware | 79.6% | 1.30 | 29.8% (−0.3 to 40.6) |
| kNN (TF-IDF) | 78.6% | 1.48 | 19.7% (−1.0 to 26.7) |
| Logistic (TF-IDF) | 78.4% | 1.39 | 24.9% (5.4 to 32.6) |
| Logistic (text only) | 77.9% | 1.59 | 14.0% (4.2 to 19.8) |
| Oracle (upper bound) | 92.1% | 0.77 |
Table 1. Routers on HELM Capabilities, 5-fold cross-validated. Best value per column in bold.
- Open-weight models lead this mix. Qwen3 235B and gpt-oss-120b are the two most accurate models, at a fraction of closed-model prices. Matching Claude Sonnet 4.5’s 71.7% costs $0.86 per 1k prompts instead of $9.31, mostly because cheap open models already beat it on these tasks.
- Against the strongest model, the saving is modest. Matching Qwen3 costs 25–30% less with routing, and only the logistic router’s interval clears zero.
- Routing still beats every single model. The task-aware router reaches 79.6%, 3.1 points above Qwen3, by sending each task to the model that is best at it, and up to 4.0 points above the best mix of two single models at the same cost.
- Knowing the task helps; reading the prompt doesn’t. Text-based routers do no better than the task-aware one, and without the task label the saving drops to 14%.
- The headroom is large. The oracle reaches 92.1% for $0.77, so a better quality predictor has a lot to gain.
Results: 2024 models
HELM Lite, 10 models including GPT-4o, Claude 3.5, Gemini 1.5, Llama 3.1, Qwen2.5 and DeepSeek-V3. GPT-4o is the most accurate single model at 79.2% for $3.35 per 1,000 prompts.
| Router | Best accuracy ↑ | $ / 1k to match GPT-4o ↓ | Saving vs GPT-4o (95% CI) |
|---|---|---|---|
| Task-aware | 82.5% | 0.244 | 92.7% (92.2 to 93.9) |
| kNN (TF-IDF) | 82.9% | 0.187 | 94.4% (93.3 to 94.9) |
| Logistic (TF-IDF) | 83.6% | 0.188 | 94.4% (93.9 to 95.1) |
| Logistic (text only) | 83.4% | 0.187 | 94.4% (93.8 to 95.0) |
| Oracle (upper bound) | 94.5% | 0.184 |
Table 2. Routers on HELM Lite, 5-fold cross-validated. Best value per column in bold.
- The headline saving is mostly DeepSeek-V3. On its own it matches GPT-4o to within 0.1 points for 89% less. Against DeepSeek-V3, the logistic router is 48% cheaper at the same accuracy, or 3.1 points more accurate at the same cost.
- Routing beats every single model. With no cost penalty the logistic router reaches 83.6%, 4.5 points above GPT-4o, for $2.25 per 1k prompts.
- Here the prompt text helps. Reading the prompt makes matching GPT-4o 23% cheaper than the task label alone, and the text-only router does just as well.
What the two suites say together
A router saves the most when cheap models are strong on some tasks and weak on others. That was true of 2024 models, where Gemini 1.5 Flash scored 93% on MATH but 33% on GSM8K (partly because of HELM’s answer format). Among 2025 models, cheap open-weight models are strong across the board, so the cheapest good choice is often obvious and a router’s job is smaller.
A claim like “routing saves 90%” only means something next to the best single model it is compared with.
Are the probabilities trustworthy?
The selection rule trusts each router’s probability that a model gets a prompt right, so the out-of-fold predictions were checked for calibration. All four routers are well calibrated: when a router says 70%, the model is right about 70% of the time, and expected calibration error is below 0.03 everywhere.
Better probabilities don’t automatically mean better routing, though. On the 2025 suite the TF-IDF logistic router has the best Brier score (0.188 against 0.202 for task-aware), but it doesn’t route better.
Related work
RouteWise doesn’t propose a new routing method; it is a small, reproducible evaluation of simple ones on public data. The difference is the data: HELM’s per-question results for current models, with real token counts, plus the corrections those counts need.
- FrugalGPT (Chen, Zaharia and Zou, 2023) cuts cost with an LLM cascade: try cheap models first and stop when a scorer is confident.
- RouterBench (Hu et al., 2024) is a benchmark of precomputed outputs for evaluating routers, with the same kind of cost–quality curves and oracle.
- RouteLLM (Ong et al., ICLR 2025) trains routers between a strong and a weak model from human preference data.
Caveats and next steps
Caveats
- List prices, not your bill: batch discounts, caching and provider choice all move them.
- Imputed output lengths make Gemini and DeepSeek-V3 costs estimates, not measurements.
- HELM’s prompts and scoring, not production traffic; some scores reflect answer formatting more than ability.
- Most routers use the task label, which real traffic rarely has.
- One run per model with binary correctness; the matched point is picked on the evaluation curve, so it is slightly optimistic.
What comes next
- Reasoning models, using a source that records thinking tokens.
- A stronger label-free router, using sentence embeddings or a small fine-tuned encoder in place of TF-IDF.
- A cascade that tries a cheap model first and escalates on low confidence.
Run it
Requires Python 3.11+ and uv. Runs on a laptop in a few minutes; no GPU or API keys needed.
git clone https://github.com/saidurpulok/routewise && cd routewise
uv sync --locked --group dev
uv run routewise fetch # about 260 MB of HELM files for the 2025 suite
uv run routewise run # cross-validates the four routers
uv run routewise plot # redraws the plots from results.json
Citation
@software{pulok2026routewise,
author = {Pulok, Saidur Rahman},
title = {RouteWise: Cost-Aware LLM Routing Evaluated on HELM},
year = {2026},
version = {0.1.0},
url = {https://github.com/saidurpulok/routewise}
}