Study: swapping the coding agent harness barely moves the score, but costs differ up to 3x

Every week a new coding agent harness promises better results. A new system prompt, smarter file tools, automatic context compression, a clever planning loop. Vendors publish benchmark gains of a few points and leaderboards mix harnesses freely, so it is hard to tell whether a score belongs to the model or to the software wrapped around it. A new preprint from Shandong University tries to separate the two, and its title already gives away the answer: "What does a harness buy? Tokens, mostly."

The paper by Yangze Liu and Zhongyi Han was posted on arXiv as 2610.04433 on 3 October 2026 and picked up widely after The Decoder covered it on 24 October. The authors held the language model fixed, swapped the harness around it and, crucially, reran identical setups to measure how much a score moves when nothing changes at all. The result is uncomfortable for anyone who picks a coding tool by its leaderboard number: changing the harness moved the pass rate about as much as running the same harness twice. What it did change, a lot, was the bill.

What a harness is, and why it matters

A coding agent is a language model plus a harness. The harness is everything around the model: the system prompt, the tools it can call, the way tool output is fed back, and the way the conversation is trimmed when it grows too long. It is also the part of the product that changes most often. The paper cites research showing popular harnesses ship more than two releases a day, mostly tweaks to prompts, tools and context handling rather than to the model.

The industry has come to believe that this engineering moves the score, and there is some evidence for it. Earlier studies reported spreads of 23.8 and 27.4 points between harnesses on the same model. One vendor, Zhipu AI, advertises a 2.4 point gain for its own harness over Claude Code with the model held fixed. The problem, according to Liu and Han, is that almost all of these numbers come from a single run per configuration. Separate research found that ten reruns of one setup on SWE-bench Verified already spread by 2.2 to 6.0 points. If a rerun alone can move a score by six points, a single run comparison that shows a three point gain says very little.

How the study was built

The researchers took three production harnesses as they ship and pinned their versions: Claude Code 2.1 from Anthropic, OpenCode 1.18 and mini SWE agent 2.4.6. They cover the design space well. mini SWE agent gives the model one single bash tool and no context management at all; it reaches the score of the original SWE-agent in about a hundred lines of Python. Claude Code and OpenCode are full products with typed file, search and edit tools, long system prompts and automatic compaction of the transcript.

Five models ran through all three harnesses: Qwen3.6 35B A3B and Qwen3.8 27B, served locally, plus DeepSeek V4 Flash, GLM 5.3 Flash and HY4 Preview through their vendor APIs. The three API models were among the most used models on OpenRouter in September. Claude Opus 5 ran inside Claude Code only, as a strong reference point at the top. Every trial got the same budget of 300 model calls.

The benchmark was SWE-bench Verified, split into two sets. "hard45" contains all 45 tasks from the two hardest difficulty bands, where you would expect a good harness to make the biggest difference. "pool447" contains the remaining 447 tasks and gives enough statistical power to bound a small effect. Ten pairs of identical reruns provide the yardstick: how much does a score move when only the run changes?

One detail from the setup is a story on its own. In early runs with web access, an audit found that 71 percent of DeepSeek and Qwen trials had simply fetched the upstream fix from GitHub instead of solving the task. Those runs were thrown out and all later runs were fully offline, with only the model endpoint reachable. The authors do not claim any vendor trained on the benchmark, but it is a reminder of how easily an agent benchmark can measure search skills instead of coding skills.

The score barely moves

On the large pool, Claude Code and mini SWE agent, the heaviest and the lightest harness, differ by only 1.8 points on Qwen3.6 and 1.4 points on Qwen3.8, with mini SWE agent slightly ahead both times. Statistically, the two are equivalent within a five point margin. The tasks on which they disagree split almost evenly between them.

On the 45 hardest tasks, the picture is even less clear. For every model the spread across the three harnesses was two to five solved tasks, while simply rerunning one configuration moved the count by up to three. The order was not stable either: mini SWE agent led on three of the five models and Claude Code on the other two. HY4 Preview, the strongest model that ran in all three harnesses, solved 23 tasks in Claude Code, 26 in mini SWE agent and 21 in OpenCode, all within the noise. Claude Opus 5 in Claude Code solved 36 of 45, which shows once again that the model is the big lever.

The most elegant measure in the paper is the flip rate: the share of tasks whose outcome differs between two runs. Rerunning the same setup flipped 13 percent of tasks at the median. Swapping the harness flipped the same 13 percent. Swapping the model flipped 22 percent. And the harness flips did not repeat. If a harness were genuinely better at certain tasks, the same tasks would tip the same way in a second run. They did not: 5 tasks split the same way twice and 7 the opposite way, a pattern a coin toss would produce. For a model swap, by contrast, 10 of 15 such tasks split the same way both times. In the words of the authors, the harness gap behaves like "a fresh draw of the same noise".

This also tells you how much a typical benchmark comparison can detect at all. With 45 tasks, a 13 point gap is caught only half the time. With 447 tasks, gaps of about five points become reliably detectable. Many advertised harness gains, including the 2.4 points from Zhipu, sit below both thresholds.

The one real effect is a loss

Only one harness effect cleared the noise, and it was negative. OpenCode trailed the other two on the large pool, by 4.7 points on Qwen3.6 and by 7.9 to 9.2 points on Qwen3.8. About half of that gap has a concrete cause: when the model's reasoning runs past the provider's output limit in the middle of a step, Claude Code prompts the model to continue, while OpenCode lets the step end, usually with nothing written to disk. On Qwen3.8, 33 OpenCode trials failed this way, compared with one in Claude Code.

Transplanting Claude Code's continuation prompt into OpenCode did not measurably help, because the runaway reasoning behind the cutoff recurs on only about 15 percent of reruns of the same task. For the rest of the gap the authors have only an observation: under OpenCode, models stop working much sooner. On Qwen3.6 a median OpenCode trial ended after 28 model calls, against 55 in Claude Code and 71 in mini SWE agent. OpenCode also had a much tighter time limit on the locally served models, which the authors flag openly as an uncontrolled variable.

The other failure modes are also ways to lose, not to win. mini SWE agent has no context management, so long trials run into the context window, but mostly on tasks the other harnesses failed as well. OpenCode's search tools try to download a binary on first use and break offline. The authors sum it up neatly: a harness buys "fewer ways to lose a task, and a bill, not pass rate."

Where the money goes

The part of the study that matters most for anyone paying for tokens is the cost analysis. With the same model, the same tasks and one price list, the cost per task differed by up to three times between harnesses.

The difference does not come from what the tools return. Per call, all three harnesses hand the model about the same amount of text. It comes from the preamble each harness sends with every single step: system prompt, tool schemas and message wrapper. That preamble is 16,581 tokens in Claude Code, 7,025 in OpenCode and just 829 in mini SWE agent. Claude Code also keeps the model's own reasoning in the transcript, based on the researchers' reconstruction, so its context starts high and stays high.

After the first call, the contexts grow at similar rates, within a factor of one and a half. So the bill is set at the first call and multiplied by the number of steps. Which of the two factors dominates depends on the model.

With GLM 5.3 Flash, Claude Code and mini SWE agent take about as many steps, so the heavy preamble shows directly: 1.81 yuan per trial in Claude Code against 0.97 in mini SWE agent. OpenCode, which carries a lighter preamble and also stops soonest, came out cheapest at 0.56 yuan. With DeepSeek V4 Flash the story flips: mini SWE agent needed 141 steps against 104 for Claude Code, its transcript kept growing, and the two ended up at roughly the same price. Prompt caching dampens all of this, since 96 to 99 percent of input tokens were cache hits, but it never changed the ranking.

No home advantage, and a year old version that was cheaper

The authors also looked for effects that would be expected if harness engineering really paid off, and did not find them. DeepSeek's own command line harness on DeepSeek's own model solved 21 of 45 hard tasks, exactly like Claude Code and one fewer than mini SWE agent, while costing the most per trial. There was no home field advantage.

Inside Claude Code, taking away the file edit tools so that Claude Opus 5 had to edit through bash, or taking away the shell instead, left the result within one task of the control runs. A Claude Code release from a year earlier solved a similar number of tasks at 0.44 times the list price cost, mainly because it did not switch on extended thinking by default. A check on 50 Terminal-Bench 2.1 tasks showed the same pattern: 19, 20 and 20 solved, no detectable order.

The one interaction that did show up was with thinking. Switching thinking off on Qwen3.6 cost mini SWE agent 11 points but barely moved the other two harnesses. On HY4 Preview, the same switch inside Claude Code also cost 11 points. Dependence on explicit reasoning seems to belong to the specific pairing of model and harness, and the authors say openly that their grid is too small to explain why.

The limits of the study

This is a preprint and has not been peer reviewed. It mainly relies on one benchmark, SWE-bench Verified, which focuses on fixing issues in Python repositories, and the Terminal-Bench check is small. Claude Opus 5, the only closed frontier model, ran only in Claude Code, so the study cannot say whether a top model would show different harness gaps. The locally served Qwen arms ran under unequal time limits. And real developers use harnesses for much more than solving benchmark issues: planning, code review, running tests, editing many files at once, working with their own tools. Comfort, safety controls and workflow integration matter in daily use and are not measured here.

The authors publish their trial table, analysis scripts and job files on GitHub, so the numbers can be checked and extended.

What this means for developers

For teams choosing a coding agent, the practical lesson is simple: pick the model first, then pick the harness for workflow, reliability and cost, not for leaderboard points. A small claimed gain from a new harness version is very likely noise unless it comes with repeated runs on a large task set. And the system prompt is not free. A preamble of more than 16,000 tokens is paid again on every step, which adds up fast for long agent sessions billed by the token.

For anyone publishing benchmark results, the paper makes one clear request: report a rerun of the same setup next to every harness comparison. Only then can readers see how much of a difference is real and how much is the dice rolling again.

My take: this matches what many developers feel in daily use. The jump from one model generation to the next is obvious, the jump from one harness update to the next usually is not. The harness matters most when it prevents losses, such as recovering from an output cutoff or compacting a long context. Beyond that, the extra engineering mostly shows up on the invoice.

Sources: Liu and Han, "What does a harness buy? Tokens, mostly." (arXiv 2610.04433), The Decoder, data and scripts on GitHub.