MMLU is functionally retired for frontier models — GPT-5.1, Gemini 3 Pro, and Claude Opus 4.5 no longer report scores on it, and the field has moved to MMLU-Pro, whose 10-way multiple choice and harder questions restored the gap between top models that the original benchmark had compressed to fractions of a percentage point.
What MMLU Actually Measures
MMLU (Massive Multitask Language Understanding) was introduced by Dan Hendrycks and coauthors in September 2020 as a 57-subject, four-option multiple-choice test spanning elementary math, US history, law, and professional domains (Hendrycks et al., arXiv 2009.03300). At release, even the largest GPT-3 model barely cleared random-chance-plus-20-points. It became the default “how smart is this model” number for four years — which is exactly what killed it.
The Saturation Timeline
GPT-4 scored 86.4% on MMLU in March 2023. On llm-stats.com’s current MMLU leaderboard (100 models tracked, last updated July 15, 2026), GPT-5 sits at the top with 92.5%, followed by o1 at 91.8% and o1-preview/GPT-4.5 tied at 90.8%. That’s a real climb over three years — but the practically useful signal collapsed well before GPT-5 arrived. By late 2024, models as different in cost and architecture as Claude 3.5 Sonnet (90.4%) and Llama 3.1 405B (87.3%) were clustering within a few points of each other, and the top of the leaderboard was separated by fractions of a percent. A test where dozens of models score between 85% and 92% has stopped discriminating between them in any way that matters for choosing a model.
Notably, none of the current frontier flagships — Gemini 3 Pro, GPT-5.1, Claude Opus 4.5 — appear on that leaderboard at all. Labs have largely stopped reporting original-MMLU scores for new releases, which is its own kind of retirement notice.
Enter MMLU-Pro: What Actually Changed
MMLU-Pro (Wang et al., June 2024, arXiv 2406.01574) is not a bigger MMLU — it’s a harder one. It expands answer choices from 4 to 10 per question, strips out the trivial and noisy items, and leans into reasoning-heavy questions across 14 domains and roughly 12,000 curated items. The paper reports the redesign drops accuracy by 16–33 percentage points relative to MMLU on the same models, and cuts prompt-sensitivity (how much scores wobble across 24 prompt phrasings) from 4–5% down to about 2%. It also flipped a longstanding result: on MMLU, chain-of-thought reasoning barely helped or even hurt; on MMLU-Pro, CoT reliably improves scores, which the authors take as evidence the new question set actually requires multi-step reasoning rather than recall.
Two Trackers, Same Benchmark, Different Leaders
This is where the “MMLU-Pro fixed everything” narrative gets a caveat. Pull the same benchmark from two independent trackers on the same week and the leaderboards don’t agree — because they’re not measuring the same thing. llm-stats.com aggregates 129 models’ self-reported scores (last updated June 23, 2026) and has Qwen3.7 Max on top at 89.6%. Artificial Analysis, which runs its own independent evaluations rather than trusting vendor-published numbers, has Gemini 3 Pro Preview (high) leading at 89.8% as of July 13, 2026, with Claude Opus 4.5 (Reasoning) close behind at 89.5% — and neither Qwen3.7 Max nor most of llm-stats’ top tier appear in Artificial Analysis’s independently-run set at all, because Artificial Analysis only tests a smaller roster it verifies directly.
The disagreement itself is the finding: self-reported leaderboards and independently-run leaderboards are answering different questions (“what did the vendor publish” vs. “what did we measure ourselves”), and neither is wrong so much as incomplete. Anyone citing a single MMLU-Pro number without naming which tracker produced it is citing a number that could shift several points depending on who ran the eval.
| Tracker | Leader (MMLU-Pro) | Score | Methodology | As of |
|---|---|---|---|---|
| llm-stats.com | Qwen3.7 Max | 89.6% | Self-reported, 129 models | Jun 23, 2026 |
| Artificial Analysis | Gemini 3 Pro Preview (high) | 89.8% | Independently benchmarked | Jul 13, 2026 |
| Artificial Analysis | Claude Opus 4.5 (Reasoning) | 89.5% | Independently benchmarked | Jul 13, 2026 |
| llm-stats.com | MiniMax M2.1 (best open-weight) | 88.0% | Self-reported | Jun 23, 2026 |
The Data-Quality Problem MMLU-Pro Didn’t Fully Fix
Harder questions don’t help if the answer key is wrong. The 2024 paper “Are We Done with MMLU?” (Gema et al., published at NAACL 2025) manually re-annotated 5,700 MMLU questions across all 57 subjects and estimated that 6.49% of the original benchmark contains errors — parsing mistakes, questions with multiple defensible correct answers, or missing context. The error rate wasn’t evenly spread: the Virology subset alone had 57% of analyzed questions flagged as flawed. The resulting MMLU-Redux dataset, when used to re-score models, produced meaningfully different rankings than the original label set had implied — meaning some of the “progress” tracked on MMLU over the years was partly an artifact of a noisy answer key, not just improving models. MMLU-Pro’s own paper claims it “eliminates trivial and noisy questions,” but it inherits MMLU’s question pool and construction process rather than being independently re-annotated at the same scrutiny level as MMLU-Redux, so treat “fixed” as “improved,” not “audited.”
What This Means for Evaluating Models Today
If you’re choosing between models based on benchmark scores, the practical takeaway is threefold. First, ignore any current comparison that leads with plain MMLU — the top of that leaderboard is noise, and several frontier labs have stopped reporting on it for exactly that reason. Second, when you do cite MMLU-Pro, name the tracker: an 89.8% from an independently-run eval and an 89.6% from a self-reported leaderboard are not interchangeable claims, even though they look like the same number. Third, MMLU-Pro is itself already showing the early signs of the compression that killed its predecessor — the current spread from rank 1 to rank 10 on llm-stats.com’s tracker is about 3 points (89.6% to 86.7%), which is tighter than it was a year ago. Our companion piece on GPQA saturation covers the same pattern on a harder benchmark, and our SWE-bench breakdown is a useful contrast — a task-based benchmark where the “harder variant” strategy has so far held up better than the multiple-choice format has. For a broader framework on how to weigh any single benchmark number, see our guide to evaluating LLMs.
FAQ
Is MMLU still worth citing in 2026?
Only as a floor check, not a differentiator. With frontier models clustered between roughly 87–92.5% and several labs no longer reporting scores on it at all, MMLU can confirm a model isn’t badly broken but can’t tell you which of two strong models is actually better.
What is MMLU-Redux and how is it different from MMLU-Pro?
MMLU-Redux is a manually re-annotated 5,700-question subset built to measure and correct MMLU’s label errors (6.49% overall, per Gema et al.); MMLU-Pro is a separately constructed, harder benchmark built to restore discriminative power via 10-way answers and reasoning-focused questions. They solve different problems — one is a data-quality audit, the other is a difficulty redesign — and MMLU-Pro was not built by applying MMLU-Redux’s corrections.
Why do llm-stats.com and Artificial Analysis show different MMLU-Pro leaders?
llm-stats.com compiles vendor self-reported scores across a wide roster (129 models as of June 2026); Artificial Analysis independently runs the evaluation itself on a smaller, directly-tested set of models. Different inputs, different leaderboards — check the methodology tag before comparing a score across sites.
Last updated July 23, 2026. This page is refreshed as benchmarks and scores move.