Your Coding Model May Be Cheap. The Harness Can Still Double the Bill.
Maestro Brief · Published by Maestro Mojo
2026-09-18
Maestro’s take.
The model is only half the bill.
The coding agent wrapped around it decides how much context gets sent, which tools are exposed, and how many tokens each step burns.
A new benchmark suggests that changing this wrapper can move cost much more than correctness.
TL;DR
Arena tested seven models inside three coding-agent harnesses: Claude Code, Codex CLI, and the minimal open-source Pi agent.
On the two benchmarks tested, the same model usually passed at roughly similar rates across harnesses. The cost did not stay similar. Some model–harness pairs cost as much as five times more than another pairing of the same model.
That does not prove Pi is the best agent for your repository. It does prove that “Which model?” is an incomplete buying question.
Ask: Which model, in which harness, at which effort, for this job?
What they actually tested
The study covered 21 model–harness combinations: seven models, each run through Claude Code, Codex CLI, and Pi.
Each combination attempted the same 30 sampled SWE-bench Lite tasks and 30 Terminal-Bench 2.0 tasks three times. The researchers used each harness’s high-effort setting, capped attempts at 100 agent turns, used official benchmark graders, and calculated cost from a fixed direct-API price list dated September 1, 2026.
That is a useful controlled test. It is not a normal month of development.
Two examples show the gap
Observed benchmark results. Each row summarizes 30 sampled tasks, three attempts per task, high-effort settings, and estimated direct-API list prices dated September 1, 2026.
| Model and benchmark | Harness | Observed success rate | Mean API cost per attempt |
|---|---|---|---|
| Claude Fable 5 · SWE-bench Lite | Claude Code | 97.8% | $1.33 |
| Claude Fable 5 · SWE-bench Lite | Pi | 96.7% | $0.67 |
| GPT-5.6 Sol · Terminal-Bench 2.0 | Codex CLI | 78.9% | $0.76 |
| GPT-5.6 Sol · Terminal-Bench 2.0 | Pi | 83.3% | $0.42 |
For Fable 5, the Claude Code pairing bought 1.1 percentage points of observed success at roughly twice the API cost.
For Sol on Terminal-Bench, Pi was cheaper and had a higher observed success rate in this sample.
Across shared models on SWE-bench Lite, the study reports that Claude Code cost about 2.0× Pi and 1.6× Codex by geometric mean. Average success-rate differences stayed within roughly two percentage points.
One measurable contributor appears before the first edit. The study measured Claude Code’s mean initial context at more than ten times Pi’s across the seven models. More instructions and larger tool schemas can mean more input tokens on every turn. Total cost also depends on caching, generated tokens, and later calls.
What this does not prove
It does not prove rich harness features are useless.
The test did not price the value of checkpoints, permissions, remote sessions, team controls, integrations, or a developer’s familiarity with a tool.
It measured direct API list-price cost. It did not measure what a subscription user actually pays out of pocket.
It tested 60 sampled tasks from two public benchmarks. The authors note that models may have seen those benchmarks during training. Real repositories are messier. Human review time was not part of the score.
The result is a reason to test the pair. It is not permission to crown a universal winner.
Why Maestro users should care
Maestro can route work across models and agent environments. That makes the routing unit important.
Do not route only by model name. Route by model + harness + effort + job.
A cheaper attempt that needs three retries is not cheap. A more expensive harness that prevents a production mistake may be an excellent bargain.
The number that matters is not tokens alone. It is the cost of an accepted result, including the developer minutes needed to review and repair it.
Do this
Do: Pick five ordinary tasks from a real repository. Run each task through the model–harness pairs you are considering. Record pass or fail, API cost, retries, and human review minutes.
Do not: Switch your whole team because one public benchmark says a minimal harness is cheaper.
Why: Benchmarks control the task. Your repository decides whether the harness’s extra machinery saves work or merely sends more tokens.
One thing to try
Take one well-specified bug with a reliable test.
Run it three times with the same model in two harnesses. Stop both at the same acceptance test. Then compare total API cost and review time—not the prettiest transcript.
Same model. Different wrapper. Different bill.
Measure the pair.
Sources considered
- Arena.ai: HarnessTax — How Much Does the Harness Matter for Coding Agents? — study design, benchmark results, cost methodology, limitations, and the reported model–harness comparisons.
Maestro’s opinion and summary were generated with AI, then independently reviewed before publication.