Findings

We measure the ceiling before we build. Everything below is our own measurement on real models with real costs, reproducible from the repository. We publish the negative results too — including the ones that argue against our own product. If routing had no value we would rather know it than sell it.

Why this page exists. Most routing projects justify themselves with a per-sample oracle number. That number is not achievable by any router, and it is the single most oversold statistic in this space. What follows is what a router can actually capture.

How to read a routing claim

Three different numbers get called "the routing gain," and they differ by more than an order of magnitude:

The gap between the first and the third is where the marketing lives.

1. Per-sample oracle numbers oversell by ~30×

On RouterBench — 11 real models, 36,497 real questions, real dollar costs — we reproduced the oracle exactly (our 0.9613 matches RouterBench's "≥1 model correct"), then asked what a prompt-conditioned router can actually reach.

Almost the entire +9.2-point "oracle gap" is per-question luck, not routable signal. The marketed 84 % cost cut on this pool becomes ~10 % once you require the router to decide from the prompt alone.

2. You cannot beat — or cheaply match — the best single model

Pool composition decides everything. Every pool containing a dominator has ~zero routable accuracy gap:

And this ceiling is fundamental, not a weak-encoder artifact. We trained the strongest supervised difficulty predictor the data allows — gradient-boosted P(success) per model, using ground-truth labels — as a near-upper-bound on any prompt-conditioned router:

held-out iso-accuracy cost saving (curated 5-model pool):
  tiny CMA-ES head on MiniLM (what a miner submits) : 12.7%
  HistGBT supervised, MiniLM                        : 13.6%
  HistGBT supervised, + length + task-id (generous) : 13.2%
  per-sample oracle (uses hidden luck)              : ~84%   <- unreachable

The strongest supervised model barely beats the tiny head and gets nowhere near the oracle. Whether a model answers a given hard question correctly is largely unpredictable from the prompt. No encoder upgrade recovers it. This retires the "a better encoder is the lever" hypothesis for broad traffic.

3. But routing is ~2.2× cheaper at intermediate quality targets

This is the correction that the "beat the top model" framing misses, and it is a real win.

"Can a router cheaply match GPT-4?" is the wrong question. The right one: for a quality target between your pool models' accuracies, is per-question routing cheaper than the naive alternative — blindly splitting traffic across single models (the convex hull of their cost/accuracy points)?

Tested on a realistic mid-difficulty workload (70 % of RouterBench, not a cherry-picked slice), with the actual trained tiny router, on held-out data:

pool: mixtral-8x7b ($1.4, 0.555) | gpt-3.5 ($2.5, 0.681) | gpt-4 ($35.1, 0.871)
  target acc ~0.87 (top): routing ~1.0x   -- can't beat the top cheaply
  target acc ~0.83:       routing 2.2x cheaper than blind model-mixing
  target acc ~0.82:       routing 2.3x cheaper than blind model-mixing

Both statements are true at once: you cannot get the top model's accuracy cheaply, and you can hit an intermediate target ~2× cheaper than any single model or blind mix. The routable signal is enough to sort easy-ish from hard-ish — just not to predict the top model's marginal wins.

4. Value is a function of the traffic, not the algorithm

cheap+strong-2 (mistral-7b vs GPT-4), supervised iso-accuracy cost saving:
  ALL broad traffic        (n=36,497, cheapAcc 0.30) :  6.6%
  routing-friendly subset  (n=   660, cheapAcc 0.89) : 97.8%   <- achievable ~= oracle
  hard subset              (n=35,837, cheapAcc 0.29) :  4.4%

On homogeneous, easy-dominated traffic a router captures ~98 % of the oracle — essentially all of it — because difficulty is predictable there. The catch: such traffic is only ~2 % of RouterBench. The eval/traffic distribution is a first-class design choice, not a detail.

5. Composition doesn't escape the dominator problem

If picking one model can't beat the best single, maybe composing several can. We measured the composition ceiling four ways:

The agentic read is the decisive one — real agent loops emit patches that apply, so it measures problem-solving, not diff formatting. With a 7-model pool (frontier generalists + coding specialists), not one task was uniquely solved by a non-opus model, and the tasks opus missed were solved by nobody. The best model's solve set was the entire oracle. Diversity did not create solve-diversity. (n=5 — the real limit; the diversity dimension is what's well-tested here.)

6. Even hand-picked specialist diversity fails

The flip-hypothesis — "with a pool where no single model dominates, routing wins" — deserved a direct test on curated, domain-diverse traffic. Four domains × 40 questions, five cost-varied domain-skewed models, real cost, honest 50/50 train/test (the router's picks are chosen on train, scored on test):

BEST-SINGLE   llama-3.3-70b         acc 0.800   $0.005
DOMAIN-ROUTER (achievable)          acc 0.800   $0.007   +0.000 acc, MORE expensive
ORACLE        per-query upper bound acc 0.875   $0.005   +0.075 = per-query noise, unroutable

The achievable router equals best-single exactly, and costs more. Two structural reasons:

1. Cheap generalists have converged. The code specialist did not win code (0.70 — tied). There is no per-domain winner for a router to exploit.
2. The cheapest model is already the best-single. llama-3.3-70b matches gpt-4o at 1/20th the cost. There is no expensive model worth routing away from.

The +0.075 oracle headroom is within-domain, per-query variance — invisible from the prompt, therefore unroutable.

7. The one axis that is not flat: scaffolds

Every read above varied the model with the scaffold fixed. But an orchestration competition rewards the scaffold — the agent code. So we fixed a cheap base model, varied the scaffold, and used hard, non-saturated tasks (MATH-500 L4–5 + MBPP).

Status: preliminary — the first non-flat result we have found.

• A cheap scaffolded agent matched a naive frontier agent at 1/17th the cost (0.60 vs 0.58, $0.011 vs $0.189).
• Oracle over scaffolds = 0.72 vs best-single-scaffold 0.60 (+0.12) — different scaffolds win different tasks, and the per-domain best differs (math → self-refine, code → code-execution).

Honest caveats, because this is the result that could flip a five-measurement verdict:

• n = 50. The +0.020 in the first bullet is within noise — a tie, not a win. The real story there is cost, not accuracy.
• The winning lever is tool access (a Python interpreter), not prompt cleverness.
• The +0.12 oracle gap is an upper bound we have not yet shown is capturable. In §6 an identical-looking oracle gap turned out to be pure noise.
• Both models are close in ability; a genuinely weak base vs a real frontier might not close.

What this means for Thirty Spokes

We make no routing-quality claim. On current models, on every traffic mix we can construct, an achievable router does not beat the best single model on accuracy — and the best single model is often the cheapest one.

What survives measurement:

• Cost at intermediate quality targets is a real, defensible win (§3).
• The competition surface is orchestration, not model choice (§7). That is precisely what this subnet rewards: miners submit an agent — routing model plus orchestration source — and are scored on benchmark quality per dollar. The one axis with measurable headroom is the one the mechanism competes on.
• Verifiability is the moat, not routing cleverness. The routing algorithm is commoditized; a free open-source router beats the funded leaders. What is not commoditized is proving an agent ran as published, at the cost it claims.

How this squares with the literature

RouteLLM, Mixture-of-Agents, and Avengers are real results, and they are frequently cited as proof the routing thesis works. Our reads do not contradict their experiments — they contradict the inference drawn from them today:

• Those results date from an era when the frontier model was far better and far more expensive than the cheap ones. That gap has largely closed: in §6, the cheapest model in the pool was the best single model.
• MoA's central claim — that diverse models beat the same model called N times — is the one we tested directly in §5's agentic read, with a 7-model pool spanning frontier generalists and coding specialists. Zero tasks were uniquely solved by a non-dominant model. Pool diversity did not produce solve diversity.

What would change our mind

Two concrete, falsifiable conditions:

1. Genuinely non-overlapping specialist traffic — a stream where a cheap or small model uniquely wins a prompt-predictable slice no generalist covers (a low-resource language; a proprietary niche a small fine-tune beats). Curated domain diversity already failed (§6), so the bar is high. Run the measurement on a sample before building anything: the achievable router, not just the oracle, must beat best-single.
2. A capturable scaffold gap — a single achievable meta-agent that approaches the 0.72 oracle of §7. If one cheap self-selecting agent captures most of that headroom, there is a real, monetizable king. If it collapses to best-single, §7 was noise too.

Reproduce

Every number above comes from a script in the repository.

# routing ceiling on RouterBench (no API key needed)
python scripts/real_pool_experiment.py        # gate + cost-quality frontier
python scripts/real_pool_comparison.py        # ceiling principle across pools
python scripts/encoder_ceiling_experiment.py  # is the ceiling encoder-bound?
python scripts/routing_value_experiment.py    # the 2.2x at intermediate targets

# composition + specialist + scaffold (needs OPENROUTER_API_KEY)
python scripts/live_composition.py            # MATH-500 composition ceiling
python scripts/live_swe_agentic.py            # agentic SWE, 7-model pool
python scripts/specialist_ceiling_experiment.py   # domain-router vs best-single
python scripts/scaffold_diversity_experiment.py   # scaffolds vs naive frontier

Full write-ups: REALDATA_FINDINGS.md, COMPOSITION_FINDINGS.md, STRATEGY.md.

Honest caveats

• RouterBench is late-2023/early-2024 models. The qualitative findings transfer (a dominator kills the accuracy moat; cost savings are distribution-bound); the exact percentages do not.
• The agentic SWE read is n=5. Its diversity dimension is well-tested (7 models); its task count is not.
• The routable ceiling is a cross-fit estimate at a fixed router capacity, deliberately conservative — not a hard bound.
• §7 is preliminary and its headline gap is a tie, not a win. We are not building on it yet.