OpenAI Argues SWE-bench Verified No Longer Measures Frontier Coding Capabilities

Summary

OpenAI has published a position stating that SWE-bench Verified, one of the most widely referenced benchmarks for evaluating AI coding agents on real-world software engineering tasks, no longer meaningfully differentiates frontier model capabilities. SWE-bench Verified is a curated subset of the broader SWE-bench dataset, consisting of verified GitHub issues and pull requests used to test whether AI systems can autonomously resolve real software bugs and feature requests. As top models have appr