Build a Router

Your entire submission is one Python file and a blob of bytes. This page is about what to put in them.

The contract

def build_agent(weights: bytes):
    """Return agent(prompt, call_model) -> answer."""
    def agent(prompt, call_model):
        # call_model(model, messages, params) -> response text.  Your ONLY channel to the pool.
        return call_model("openai/gpt-4o-mini", [{"role": "user", "content": prompt}], {"max_tokens": 256})
    return agent

Four constraints shape every design:

Your agent may make many calls per task. Cost is metered per call and summed; the only thing that matters is that the total stays under the budget and the quality is high.

Start where the evidence is

We measured which axis actually has headroom, and it is not which model you pick. On current models, a router that picks the best model per query does not beat the best single model — see Findings for five independent measurements of that.

What did move: the scaffold. Fixing a cheap base model and varying the orchestration on hard, non-saturated tasks:

Two things to take from this:

1. A cheap model with a good scaffold matched a naive frontier agent at 1/17th the cost. That is the whole game under a cost ceiling.
2. More calls is not more quality. Self-consistency cost 5× for a gain inside the noise. Plan-solve actively hurt. The cheapest scaffold was also the best.

Caveats, because these are our own numbers: n = 50, the accuracy edge over the frontier is a tie, not a win (the win is cost), and the lever was tool access — a Python interpreter — rather than prompt cleverness. Treat the table as a starting prior, not gospel. Measure your own.

Patterns worth trying

Tool / code execution. For math and code, have the model emit a program, run it, and use the result. Cheapest and strongest in our measurements. Nothing forbids executing code inside your agent — the confinement blocks the network, not the CPU.

Cascade. Answer with the cheap spoke; escalate only when a trigger fires (length, a verifier's rejection, low self-reported confidence). This is the classic cost play and it composes with everything.

ans = call_model(cheap, msgs, {"max_tokens": 256})
if not verifier_accepts(ans):
    ans = call_model(strong, msgs, {"max_tokens": 512})

Verify then escalate. Spend a cheap call checking the answer rather than an expensive call producing it. A verifier that is right 80 % of the time still routes most traffic correctly.

Decompose. Split a hard prompt into sub-questions, solve each with a cheap spoke, then synthesise. Costs more calls; only pays where the sub-problems are genuinely easier.

Self-consistency. Sample k answers, take the majority. Real but expensive; under a cost ceiling it must earn its 5×. Consider it only on the benchmark where your floor is at risk.

Route by domain. Trivially predictable from the prompt — but note we measured a domain router coming out exactly equal to best-single and costing more. Don't assume this one works; verify it on the pinned pool you're given.

Cost discipline

The budget is a hard gate: over it, you earn nothing, regardless of quality.

• max_tokens is your main dial. Most of the bill is output tokens.
• Meter as you go. Track your own spend per task; degrade to the cheap path when a task is running hot.
• Every benchmark has a floor. acc_b ≥ f_min on every benchmark, so you cannot abandon the hard one to save money — below_floor:<b> disqualifies you entirely.
• The dethrone guard checks cost too. A challenger must be within +10 % of the king's cost.

Training the weights

weights.bin is opaque bytes — a trained model, a config, a lookup table, anything. The subnet ships no trainer; that is your edge.

A workable loop:

1. Collect data with the dev kit: for each task, which pool model answered correctly, at what cost.
2. Fit whatever you like — a classifier over prompt embeddings, a policy, a set of thresholds. The reference prototype uses a ≤1M-parameter head trained with separable CMA-ES, but nothing requires that.
3. Serialise into weights.bin and load it in build_agent.

Remember that both the source and the weights are public and bound to what ran. There is nowhere to hide a lookup table of answers — hardcoded_answers, and the source scan is run on the artifact the validator downloads itself.

Use the dev kit as your training signal

uv run orchestra-koth-dev --source my_router.py --weights my_weights.bin \
  --n-per-bench 8 --budget 0.5

It reports the exact Q_lcb, per-benchmark acc/lcb, total_cost, n_pool_calls and eligible the validator will compute — because it calls verify_proof directly. Optimise against this number, not a proxy.

Two traps it will catch for you:

• lcb is not acc. You are scored on the lower confidence bound. A high-variance agent scores worse than a boring reliable one at the same mean.
• n_pool_calls must be ≥ 1 per scored task. A clever cache that skips a call is a DQ.

Anti-patterns

See also

• Mine the Subnet — publish and compete
• Findings — the measurements behind the advice on this page
• The KOTH Protocol — scoring, eligibility, the dethrone guard