Humanity's Last Exam: A 2,500-Question Benchmark Tests AI Reasoning and Context Understanding

Summary

Science 2.0 examines "Humanity's Last Exam," a 2,500-question evaluation tool designed to test AI systems across math, humanities, science, ancient languages, and other subfields. The benchmark's creators argue it goes beyond pattern recognition by requiring expert knowledge and contextual understanding — an area where large language models historically struggle. The article uses the example of image-generation models producing humans with six fingers as evidence that LLMs lack genuine understan