METR Research Finds Many SWE-bench-Passing PRs Would Not Be Merged into Production

Summary

METR (Model Evaluation & Threat Research) has published a research note examining the quality of pull requests generated by AI coding agents that pass the widely-used SWE-bench benchmark. Their analysis found that many of the PRs that technically pass the benchmark's test suites would not meet the standards required for merging into a real production codebase. This raises significant questions about the validity of SWE-bench as a proxy for real-world software engineering capability. This findin