The Evaluation Crisis and the Death of SWE-bench as a Reliable AI Coding Benchmark

Summary

Zencoder, an AI coding tool vendor, published an analysis arguing that SWE-bench — the widely used benchmark for evaluating AI coding agents' ability to resolve real-world GitHub issues — has reached a point of diminishing reliability. The piece likely examines how benchmark saturation, data contamination, and overfitting to specific evaluation formats have undermined SWE-bench's ability to meaningfully differentiate AI coding tools, a phenomenon sometimes called Goodhart's Law applied to AI eva