One agent used 1.06M tokens and never wrote the patch
Why a token budget is not enough, and the two earlier stops we added outside the model.
One agent in a 50-instance SWE-bench Verified run used 1.06 million tokens and never wrote the patch. The storage was fine: a control file written through the same mount survived. The agent had kept working without saving anything.
The short answer
- Keep the hard token or time budget, but put the stop decision outside the model. The budget caps the worst case and cannot tell busy work from progress.
- Stop earlier only on exact repeats. The runtime counts the same tool action returning the same failure again. After a small number of repeats it tells the agent to change approach, then stops executing the calls.
- Checkpoint at 60% of the token budget, while tools still work. The agent is asked to write its best partial artifact and say whether the approach is converging.
What the benchmark runs showed
| Run | Result |
|---|---|
| 30-attempt pilot, unresolved | 15 of 30 |
| Unresolved with no result file | 10 |
| Spend on those 10 attempts | $5.2416 of $13.4832 (38.9%) |
| 50-task report, unresolved tasks | 24 of 50 |
| Cost of unresolved tasks | $14.6807 of $28.5404 (51.4%) |
Neither figure is a loop rate. In the pilot, patch_empty means /swebench.patch was missing, not that the agent made no edits. The 50-task unresolved group also includes wrong patches, timeouts, and platform or harness failures.
When it applies
- Exact matching misses some loops: a changed argument, duration or line number makes two similar failures look different. Successful commands that never advance the task also reach the budget. We accept these misses because a false stop can end a run that is slowly learning something useful.
- Long-running tools: the detector cannot tell that a tool has stopped making progress. That needs tool-specific evidence, such as a process heartbeat.
- Evidence limits: the 1.06-million-token case is one task under one configuration. We have no clean production estimate of credits saved by the stop policy.
The full post has the definition of progress, why iteration limits are not enough, and what the user sees when an agent is stopped.