A Constraint-First Model Router for AI Agents
How a local, inspectable Router improved verified Agent success while lowering normalized cost per successful task—and what the evidence still does not prove.
Auto routing should not mean asking another model to guess which model to use. It should be a fast, inspectable decision that rules out failure first, then spends only where the task justifies it.
We rebuilt our Auto policy around that idea. The new Router preserves the previous Router's capability tiers and recovery semantics, but replaces the fixed primary model inside each tier with a small, task-fit portfolio. It reads request-side signals, removes models that cannot satisfy the contract, ranks only qualified candidates, and pins the winner for the Agent session.
In our initial three-arm Agent checkpoint, verified task success was 57% for the new Router, 49% for the previous Router, and 67% for a fixed Opus 5 baseline. The new Router's success point estimate improved by 8 percentage points. Its total token-normalized cost increased by 8.8% versus the previous Router, but normalized cost per successful task decreased by 6.4%. Compared with using Opus 5 for every task, the new Router used 91.1% less normalized token cost while giving up 10 points of task success.
Those results are promising, but they are not a SOTA claim. The old/new confidence interval still crosses zero, fixed-flagship non-inferiority is not met, and the new Router's p95 Agent-session latency is too high. This report presents the improvement and those limits together.
From a static tier to a qualified portfolio
The previous policy was intentionally simple:
- Classify the request into a capability tier.
- Select that tier's configured primary model.
- Follow the tier's ordered fallback chain if the primary fails.
That design is fast, predictable, and operationally legible. Its weakness is that it treats many requests inside the same tier as interchangeable. A one-step tool lookup, a multi-record policy workflow, a deep web investigation, and a terminal repair can all look “agentic” while requiring very different model behavior.
Static tiers also combine two questions that should remain separate:
- Can this model satisfy the request contract?
- Among models that can, which one is the best fit for this task?
The new Router answers them in that order.
One bounded, local decision before the first model token. No routing-model call.
- 01
Read request signals
Task shape, tools, input size, workflow complexity, risk, and urgency.
- 02
Enforce the contract
Remove candidates that fail tools, vision, context, output, or catalog requirements.
- 03
Build a task-fit portfolio
Only models with explicit affinity for the detected work are allowed to compete.
- 04
Rank once, then pin
Select locally, pin the model for the Agent session, and retain ordered recovery.
| Decision property | Previous Router | New Router |
|---|---|---|
| Safety boundary | Static capability tier | Static tier retained |
| Primary inside a tier | Fixed model | Locally ranked task-fit portfolio |
| Tools, vision, context, output | Mostly tier/config driven | Hard eligibility before scoring |
| Generic fallback promotion | Can inherit tier order | Cannot displace explicit task fit on price alone |
| Performance observations | Not used within tier | Weak, time-decayed refinement |
| Recovery | Ordered fallback chain | Ordered fallback chain retained |
Four stages, one local decision
1. Read deterministic request signals
The Router inspects request metadata and bounded prompt features already available in process:
- task shape, including chat, reasoning, code, terminal work, tool use, web research, structured output, long context, and vision;
- visible tool names and tool count;
- estimated input and requested output size;
- task-domain signals such as policy workflows, deep research, and terminal operations;
- workflow complexity, including parallel actions, cross-record operations, multi-clue investigation, and multi-artifact repair;
- high-stakes, safety, and latency-sensitive intent.
This is not a free-form LLM classifier on the hot path. The features are local, deterministic, testable, and available before inference.
2. Apply hard eligibility
Before any model receives a score, the Router removes candidates that cannot satisfy the request:
- no tool-calling support for a tool-required request;
- no vision support for image input;
- insufficient context window or output capacity;
- incompatible structured-output path;
- unavailable in the active model catalog.
A cheaper or faster model cannot compensate for failing a hard contract. Capability errors never enter the preference optimization stage.
3. Build a task-fit portfolio
The Router then admits only models with explicit affinity for the detected task. This is where a routine lookup separates from a complex policy workflow, a simple terminal artifact separates from a cross-runtime repair, and ordinary search separates from a multi-hop evidence investigation.
The previous fallback chain still contributes a curated prior and recovery path. It no longer implies that every fallback is equally qualified to become the primary model for every task.
4. Rank locally and retain recovery
The default Auto profile combines six bounded factors:
| Component | Auto weight | Role |
|---|---|---|
| Task quality | 0.47 | Prefer models validated for the detected work |
| Capability | 0.20 | Preserve the request contract after hard filtering |
| Estimated cost | 0.18 | Reward efficient qualified candidates |
| Speed | 0.07 | Improve ordinary interactive latency |
| Reliability | 0.03 | Prefer stable candidates |
| Curated order | 0.05 | Retain a small, explainable prior |
Eco increases the cost weight. Premium increases quality and reliability. High-stakes and latency-sensitive requests adjust the balance without changing eligibility.
The ordering is the important property: hard requirements determine who may compete; scoring determines who wins.
Why the Router chooses once per Agent task
An Agent trajectory can contain many model and tool turns. Routing every turn independently creates new failure modes: incompatible tool-call styles, context discontinuities, difficult cost attribution, and an evaluation where the treatment changes halfway through the task.
For this checkpoint, the Router sees the first actionable user turn and the real tool surface, selects once, and pins the model for the complete trajectory. The ordered fallback path is reserved for genuine execution failure. There is no auxiliary routing-model request and no invisible mid-task model drift.
Mid-task escalation remains a useful future direction. It should arrive as an explicit state machine—such as escalation after repeated failed validation—not as opportunistic per-turn switching.
Measuring Agent outcomes instead of prompt labels
Our early Router fixture measured prompt-level classification. It was useful for rapid regression testing, but it could not answer the question that matters: does the selected model complete the Agent task?
The current checkpoint evaluates full sessions in a common host framework. It draws from three public Agent benchmark families and preserves their workload shape and validators.
Public benchmark families, complete trajectories, source-native validators.
SWE-derived repository repair is present inside the terminal slice, but is intentionally not reported as a standalone SWE-bench score.
| Public source family | Share of strict cohort | What it measures |
|---|---|---|
| τ-bench family | 55% | Stateful tool use under domain policies |
| BrowseComp | 25% | Multi-hop web research ending in an exact answer |
| Terminal-Bench | 20% | End-to-end terminal and repository work |
SWE work is represented: about 2% of the strict cohort consists of SWE-derived repository-repair tasks nested inside the Terminal-Bench slice, evaluated with official Terminal-Bench tests. That is useful Agent evidence, but it is not a standalone SWE-bench Verified result and the blended score should not be presented as one.
Every task ran through the same host framework and validator with three policies:
- Previous Router: previous rules and tier order;
- New Router: constraint-first portfolio selection;
- Fixed Opus 5: the same flagship model for every task, bypassing Router selection.
Within each task triple, all three arms used the same frozen model catalog, payment chain, pricing snapshot, task definition, tool surface, trajectory contract, and scorer. Only semantically compatible complete triples are pooled. Task success is binary and validator-backed.
Cost is reconstructed from actual input and output tokens multiplied by the cohort's pinned public model prices. Wallet settlement is recorded separately for accounting and is not used as the Router-efficiency metric.
Infrastructure failure is not counted as an incorrect model answer. 10.7% of materialized triples were invalid or incomplete; they remain visible in the ledger rather than being silently converted to quality failures or deleted.
The three-arm checkpoint
Binary, validator-backed full-session outcomes. Bars share a zero baseline.
Point-estimate lift over the previous Router. Statistical uncertainty remains material.
The source-level point estimates moved in the same direction:
- stateful tool-use tasks: 70.9% → 81.8%;
- deep web-research tasks: 16.0% → 20.0%;
- terminal and repository tasks: 30.0% → 35.0%.
The research and terminal slices are hard and comparatively limited, so these subgroup values are diagnostics, not independent release claims. The SWE-derived work is included in the terminal result but is intentionally not reported as a standalone rate because its cohort share is too small for a stable claim.
Same tasks, compared pairwise between the previous and new Router policies.
The pairwise breakdown matters. New-only wins exceed previous-only wins, but joint failures still account for a substantial share of the checkpoint. Model selection improved the measured frontier; it did not remove the limits of the host scaffold, tools, or trajectory budget.
More total spend, less cost per successful task
The new Router spent 8.8% more normalized token cost than the previous Router across the complete triples. It also solved 16.3% more tasks. As a result, normalized cost per verified success fell from $0.1406 to $0.1315—a 6.4% improvement.
Against fixed Opus 5, the trade-off is larger. The new Router gave up 10 points of success, but used only 8.9% of the normalized token cost. Cost per success was 89.5% lower.
Reconstructed from realized input/output tokens and pinned public prices; wallet settlement is accounted separately.
New Router cost per success vs previous Router
Total token-normalized cost vs fixed Opus 5
This is the economic case for routing. The objective is not to minimize the price of each call. It is to minimize the cost of completing useful work subject to an acceptable quality boundary.
Why fixed Opus 5 scored 67%, not 90%+
The fixed flagship result is a weighted Agent score, not a standalone SWE benchmark result:
- stateful tool use: 90.9%;
- deep web research: 40.0%;
- terminal and repository work: 35.0%.
84.8% of the flagship arm's failures came from the research and terminal slices. The result measures the combination of model, host framework, tools, trajectory budget, and benchmark difficulty. It is not directly comparable with a vendor score produced by a specialized coding or research scaffold.
This also limits what the Router result proves. A Router can only choose among the outcomes its host framework allows models to achieve. Better model selection and a stronger Agent scaffold are complementary, not substitutes.
The Router is locally fast; the selected trajectories are not
The decision path makes no inference call. Its runtime is negligible compared with a multi-minute Agent session.
The measured p95 nevertheless regressed from 464.0 seconds to 705.9 seconds. The likely mechanism is not classifier overhead. The new policy selects stronger trajectories for difficult tasks and sometimes allows them to run longer.
The local Router is fast; the selected model mix and longer trajectories drive the measured tail.
New vs previous p95. This fails the current latency guardrail.
The next iteration needs trajectory-aware controls: estimated session cost, tool-loop efficiency, and explicit escalation budgets. Optimizing the scoring function by another fraction of a millisecond would not address this tail.
What the statistics allow us to say
The strict three-arm checkpoint gives the clean headline: 57% versus 49%, or +8 points.
The scorecard also contains a pair-only evidence set that is 3% larger because a small fraction of pairs lacked a valid fixed-flagship arm. On that larger set, the quality difference is +6.8 points with a paired bootstrap 95% interval of −1.9 to +15.5 points. The interval crosses zero.
The new Router trails fixed Opus 5 by 10 points in the strict three-arm cohort; the paired 95% interval is −19 to −1 point.
Paired bootstrap 95% intervals on the pair-only set, which is 3% larger than the strict three-arm set — so the old/new point estimate reads +6.8 here against the +8 pp headline. Axis spans −25 to +20 percentage points.
Release gate remains closed. The old/new interval crosses zero, fixed-flagship non-inferiority is not met, and p95 latency is above the guardrail.
The evidence supports this statement:
In this Agent checkpoint, the new Router had a higher success point estimate than the previous Router and a lower normalized cost per successful task, while using a small fraction of fixed-flagship token cost.
It does not yet support these statements:
- the new Router is statistically proven to outperform the previous Router across the production distribution;
- the new Router matches Opus 5 quality;
- the Router is SOTA;
- the current version has passed every release gate.
The machine-readable scorecard correctly records releaseEligible: false.
Evaluation discipline matters as much as the formula
A credible Router evaluation must separate four failure classes:
- Router error: the policy selected an inappropriate but technically usable model.
- Model-quality failure: the selected model completed the trajectory but failed the validator.
- Host or Adapter failure: tool history, shell state, context, or finalization was represented incorrectly.
- Payment or provider failure: the request never produced a valid model outcome.
Treating the last two classes as wrong answers makes a Router look worse for infrastructure reasons. Retrying model-quality failures until they pass makes it look better by spending more attempts. The frozen evaluator fails closed on infrastructure, preserves every invalid row, and never changes the task list in response to intermediate quality results.
One generic host issue appeared across model arms: a tool-less final-synthesis request carried structured tool history that required a tool configuration. We fixed the host behavior only after reproducing the same failure across policies, closed the affected cohort, and froze a new Adapter hash. We did not rewrite prior outcomes or patch individual tasks.
What comes next
This checkpoint is evidence of direction, not the end of evaluation.
- 01
Broaden SWE evidence
Add a matched SWE-bench Verified slice with its official container grader.
- 02
Measure scaffold loss
Compare fixed Opus 5 under Franklin and a reference SWE Agent scaffold.
- 03
Freeze a larger cohort
Pre-register public source quotas and keep task selection outcome-blind.
- 04
Control the trajectory
Add session-cost, tool-loop, and escalation budgets before a new latency gate.
- 05
Shadow before traffic
Measure coverage and policy divergence without a second inference request.
The architecture is deliberately designed to improve without adding an inference call to routing. Better public task evidence can update task affinity and portfolio priors; hard capability rules remain inspectable and fail closed.
Closing
Routing is not the art of finding the cheapest model. It is the engineering discipline of choosing the least expensive qualified model that can finish the work—and knowing when the evidence is not strong enough to claim victory.
The initial Agent checkpoint moved the measured frontier in the direction we wanted: +8 points of verified task success, −6.4% normalized cost per successful task, and −91.1% normalized token cost versus fixed Opus 5. It also exposed the next work clearly: latency, Agent-scaffold parity, deeper SWE coverage, and statistical power.
That is a better foundation than a Router benchmark designed only to produce a winning headline.