Model router: kimi-k3 and deepseek-v4-flash both resolved 34 of 50
A pinned control run showed 33x the input price bought no extra resolved tasks on our workload.
Our model router picks a serving model from a price-ordered candidate list. After we replaced hand-written scores with measured Artificial Analysis indices, we ran a 50-task SWE-bench Verified subset twice: routed, and pinned to the cheapest candidate.
The short answer
- Same result: the router chose kimi-k3 on 47 of 47 decisions, at 33x the input price of the cheapest candidate. Both runs resolved 34 of 50 tasks.
- Lower spend: the pinned deepseek-v4-flash run cost $17.30 in model spend against $39.62, which is 44%.
- Policy change: the 4.4-point agentic index gap did not transfer to our workload, so index gaps alone no longer justify a price step.
Routed and pinned runs
| Basis | Routed (kimi-k3 on 47/50) | Pinned (deepseek-v4-flash) |
|---|---|---|
| Resolved, same-harness basis | 34 / 50 | 34 / 50 |
| Model spend | $39.62 | $17.30 |
| Total billed (models + compute) | $45.27 | $25.38 |
| Total tokens | 27.2M | 33.9M |
When the answer changes
- One workload, one round: each configuration ran once on 50 of the dataset's 500 instances, with a 25-minute per-task limit. The tie means no measurable difference here, with a noise band of roughly one to two tasks. These are not leaderboard numbers.
- Harness basis: the pinned run used a newer harness and reached 39 of 50. We excluded the 5 extra tasks because the routed run never got the same help.
- Historical roster: Kimi K3 left the Plori Router candidate pool on August 14. Prices and results are snapshots from August 5.
The full post has how the two models failed differently and how to set up a pinned control run.