Evaluating an LLM means measuring what it can actually do — with versioned benchmarks, appropriate metrics, and enough skepticism to catch the ways all three can mislead you. That one sentence is the whole discipline; everything below is the practice of doing it honestly.
This is the hub page for our benchmark coverage. Every individual benchmark we explain links back here, and this page links out to each of them as they publish.
Why LLM evaluation is genuinely hard in 2026
Three structural problems make evaluation harder than reading a leaderboard:
1. Saturation. Frontier models now cluster within about a point of each other on older benchmarks. On GPQA, the top three models sit within 1.3 points (95.5% to 94.2% as of July 1, 2026, per BenchLM’s tracker). When everyone scores 94–95%, the benchmark has stopped discriminating — it tells you a model is frontier-class, not which one is better.
2. Contamination. Benchmarks published before a model’s training cutoff may be in its training data. A model can “know” the answers without the capability the benchmark claims to measure. This is why live, rolling benchmarks (problems added after training cutoffs) increasingly matter.
3. Leaderboard disagreement. The same benchmark can produce different rankings on different trackers. As of this week, one tracker reports Claude Mythos 5 leading SWE-bench Verified at 95.5% (BenchLM, July 8), while another reports Claude Fable 5 at 95.0% as the leader (LLM Stats, July 2026). Neither is necessarily wrong — they differ in harness versions, scaffolding, and which submissions they accept. If you quote a score without naming the tracker, harness, and date, you have not quoted a score.
The three layers of evaluation
Layer 1 — Capability benchmarks. Standardized task sets with automatic scoring: GPQA for graduate-level reasoning, SWE-bench Verified for real-repository coding, MMLU-Pro for broad knowledge, BFCL for tool calling. Fast, reproducible, comparable — and gameable.
Layer 2 — Quality metrics. For tasks without a single right answer: exact-match and pass@k for code, LLM-as-judge scores (MT-Bench style) for open-ended output, plus latency, cost per task, and refusal rates. Metrics are only as good as their judge — LLM-as-judge in particular inherits the judge model’s biases.
Layer 3 — Human preference. Arena-style Elo rankings from blind pairwise votes (LMArena and successors). These capture “which output do people prefer,” which correlates with — but is not — capability. Preference can be won with confident formatting.
A serious evaluation uses all three layers and treats disagreement between them as signal, not noise.
Which benchmark for which question
| If you’re asking… | Look at | Watch out for |
|---|---|---|
| Can it reason through hard science? | GPQA Diamond | Saturated at the frontier; top 3 within ~1.3 pts |
| Can it fix real code? | SWE-bench Verified, Aider Polyglot | Scaffolding differences move scores several points |
| Can it use tools reliably? | BFCL, tau-bench | Young benchmarks; methodology still shifting |
| Is it broadly knowledgeable? | MMLU-Pro | Original MMLU is contaminated and saturated |
| Do people like its answers? | Arena Elo | Preference ≠ capability |
| Can it handle the hardest known questions? | Humanity’s Last Exam | Newsy; scores move fast between releases |
Data as of July 10, 2026. Sources linked above; individual explainer pages coming for each benchmark.
How to read any score critically: a five-question checklist
Which version? Benchmark and model both. “SWE-bench” (original), “Lite,” “Verified,” and “Pro” are four different tests with leaders ranging from 62.7% to 95.5% this month. A score without a version is meaningless.
What harness and scaffolding? Especially for agentic benchmarks, the scaffold (retry logic, tool access, context management) can matter as much as the model.
Is there a confidence interval? Single-run scores on a few hundred problems carry meaningful variance. A 0.5-point lead without error bars is a tie.
Could it be contaminated? Check the benchmark’s release date against the model’s training cutoff.
Who reported it? Vendor-reported numbers use the vendor’s best scaffold. Independent re-runs typically come in lower. Treat vendor claims as upper bounds until reproduced.
A practical evaluation workflow
(1) Define the task you actually care about — “best model” is not a question. (2) Pick two benchmarks closest to that task from the table above. (3) Cross-check scores on at least two trackers, noting version, harness, and date. (4) Shortlist two or three models, then run your own eval on 30–50 examples from your real workload — your data beats every public benchmark. (5) Measure cost and latency alongside quality; a 1-point capability edge rarely survives a 5x cost difference. (6) Re-test on every model version bump: capabilities regress as well as improve.
FAQ
What is the single best LLM benchmark? There isn’t one. The closest to a general-purpose signal today is a composite: GPQA Diamond for reasoning plus SWE-bench Verified for coding plus arena Elo for preference — cross-checked on more than one tracker.
Are LLM benchmarks trustworthy? Directionally yes, precisely no. They reliably separate frontier from non-frontier models. They do not reliably separate the top three models from each other — saturation, contamination, and harness variance are all larger than the gaps between leaders.
How often do rankings change? Monthly, sometimes weekly. Frontier releases in 2026 have reshuffled coding leaderboards several times in a single quarter — which is why every score on this site carries a date.
Last updated July 10, 2026. This page is refreshed as benchmarks and scores move; changes are logged here.