Humanity’s Last Exam: Tracking the Hardest Benchmark

Humanity’s Last Exam (HLE) has gone from sub-10% scores at its January 2025 launch to a frontier cluster in the 55-65% range by August 2026 — but three independent trackers disagree by nearly 20 percentage points on where exactly that frontier sits, because they’re no longer running the same test.

HLE was built by the Center for AI Safety (CAIS) and Scale AI, released January 23, 2025, specifically to replace benchmarks that had saturated. The original paper is blunt about the motivation: “LLMs now achieve over 90% accuracy on popular benchmarks like MMLU,” which made those tests useless for measuring further progress. Nineteen months later, HLE is showing early signs of the same fate, and the disagreement between leaderboards is itself the most interesting data point.

What HLE actually tests

The final public set contains 2,500 questions, whittled down from roughly 70,000 submissions and 13,000 human-reviewed candidates, contributed by nearly 1,000 experts across 500+ institutions. CAIS and Scale paid $5,000 for each of the top 50 questions and $500 for the next 500, then ran a bug-bounty program to catch errors post-launch (closed March 21, 2025).

Per the paper, subject coverage is heavily math-weighted: mathematics 41%, physics 9%, biology/medicine 11%, humanities/social science 9%, computer science/AI 10%, engineering 4%, chemistry 7%, and other subjects 9%. About 14% of questions require reading an image; 24% are multiple choice, and the rest are short-answer with an exact, unambiguous solution — questions are supposed to be “Google-proof,” meaning retrieval alone shouldn’t get you the answer. Scoring is pass@1, graded by an LLM equality checker against the reference answer, per Artificial Analysis’s methodology notes.

The questions read nothing like MMLU trivia. Public examples include decoding a two-step substitution cipher, explaining an FTIR spectrum shift in tardigrade hydrogel proteins, and computing stress patterns in Old Russian phrases with no linguistic background assumed. This is the same design philosophy behind GPQA — expert-authored, hard to shortcut — just scaled up and broadened past science into humanities and formal logic.

From single digits to a 55-65% cluster

At launch, the gap between AI and the benchmark’s intent was stark. Scale AI’s own release said current models “were only able to answer fewer than 10 percent of the expert questions correctly.” Reported scores from that round: GPT-4o at 2.7%, Claude 3.5 Sonnet at 4.1%, OpenAI’s o1 at 8.0%, Gemini 2.0 Flash Thinking at roughly 6.6%, and DeepSeek R1 at roughly 8.5%. CAIS framed the intended human comparison point as domain experts averaging around 90% accuracy — but only within their own specialty, not across the full interdisciplinary spread of the exam.

By August 2026, frontier models are answering a majority of questions correctly on at least some trackers. That’s a roughly six-fold jump in accuracy in under two years — fast even by LLM benchmark standards, and a sign the “final closed-ended academic benchmark,” as the paper called it, is approaching the saturation it was built to avoid.

Three trackers, three different leaderboards

This is where HLE gets messy. As of the third week of August 2026, the three most-cited independent trackers don’t agree on who’s winning or by how much:

Tracker	Top model	Top score	Question set / protocol	As of
Artificial Analysis	Claude Fable 5 (max effort)	55.5%	2,158 text-only questions (multimodal excluded for comparability)	Aug 22, 2026
BenchLM.ai	Claude Opus 5	64.7%	Full public set, 53 models tracked, mixed tool-assisted/closed-book	Aug 22, 2026
Scale AI / SEAL Labs	Gemini 3.1 Pro Preview (thinking, high)	46.4%	Official CAIS/Scale leaderboard protocol	Aug 16, 2026

The gap between Artificial Analysis’s 55.5% and BenchLM’s 64.7% ceiling — on the same benchmark, in the same week — is nearly nine points. Scale AI’s own leaderboard, run by the benchmark’s creators, puts the leader almost 20 points below BenchLM’s. None of these are wrong exactly; they’re answering different questions. BenchLM’s methodology notes put it plainly: “A model with search, browsing, or code execution is not taking the same test as a closed-book model.” Artificial Analysis strips out the ~14% of multimodal questions entirely for cross-model comparability, which mechanically shifts scores versus a tracker scoring the full set. Add in different grader models and different retry/effort settings, and you get three defensible but incompatible leaderboards. BenchLM’s own read of its data — the top three models sit within 2.6 points of each other — is itself a hint that HLE is nearing saturation for tool-assisted frontier models, even while closed-book scores lag well behind.

The bigger problem: the answer key has errors

Cross-tracker disagreement compounds a more basic issue: some of HLE’s reference answers are wrong. An independent investigation by FutureHouse, published September 16, 2025, found that roughly 30% of the text-only chemistry and biology answers were likely incorrect when checked against published literature — including a reference answer naming oganesson as “Earth’s rarest noble gas” despite oganesson not occurring terrestrially at all. CAIS and Scale partially replicated FutureHouse’s findings and, according to reporting cited on Wikipedia’s HLE entry, acknowledged the issue and committed to continuous revision rather than disputing the scale of it.

The response was HLE-Rolling, a continuously updated variant meant to fold in community-flagged corrections and fresh questions, plus a cleaned “HLE Bio/Chem Gold” subset published on Hugging Face for the two most error-prone domains. This matters for anyone citing HLE scores as a precise capability measurement: a model’s score partly reflects how well it matches an answer key that is itself still being corrected. It’s a similar dynamic to what shows up in hallucination benchmarks, where the grading pipeline’s own reliability is as much the story as the model’s.

What this means for evaluating models today

If you’re using HLE to compare models, three things follow from the above. First, always check whether a score is text-only or includes the multimodal 14% — they’re not interchangeable, and Artificial Analysis’s decision to exclude multimodal questions is a defensible but score-lowering choice relative to trackers that include them. Second, check whether tools were allowed; a browsing-enabled agent and a closed-book model are, functionally, taking different exams even though both get labeled “HLE.” Third, treat any single-point comparison (“Model X beats Model Y by 2 points”) with skepticism when the gap between trackers on the same model can exceed that margin. This is the general HLE lesson: cross-checking scores across at least two sources, the approach laid out in our guide to evaluating LLMs, isn’t optional caution here — it’s close to mandatory given how differently HLE gets implemented in practice.

FAQ

Is Humanity’s Last Exam still a useful benchmark in 2026?
Partially. It remains harder than MMLU or GPQA and still separates frontier models from mid-tier ones, but the top cluster is now within a few points of each other on at least one tracker (BenchLM), and known answer-key errors in chemistry and biology (around 30% per FutureHouse’s estimate) mean some of the remaining gap between models may reflect grading noise rather than capability differences.

Why do HLE scores vary so much between leaderboards?
Different trackers use different question subsets (full 2,500 vs. the 2,158 text-only questions), different tool-use protocols (closed-book vs. search/code-execution enabled), and different LLM graders. Artificial Analysis, BenchLM.ai, and Scale AI’s own SEAL Labs leaderboard each report different leaders and different top scores for the same calendar week in August 2026.

What was the human baseline on HLE?
CAIS reported that domain experts average roughly 90% accuracy on HLE questions within their own specialty. That figure doesn’t describe a generalist human answering the full interdisciplinary 2,500-question set — no such baseline has been published — so it should be read as a per-domain ceiling, not a single human score to benchmark models against.

Last updated August 24, 2026. This page is refreshed as benchmarks and scores move.