Hugging Face: AI Evals Are Becoming the New Compute Bottleneck

Summary

A Hugging Face research post argues that AI evaluation has crossed a critical cost threshold where rigorous benchmarking now rivals or exceeds training costs—inverting deep learning's traditional economics. The evidence is stark: a single GAIA agent benchmark run costs $2,829; HAL's holistic evaluation across 9 models costs $40,000; and training-in-the-loop benchmarks (PaperBench, MLE-Bench) cost $5,000–$9,500 per evaluation run. Crucially, agent benchmarks compress only 2–3.5× versus 100–200× f