HumanEval, the 164-problem coding benchmark OpenAI released in 2021, no longer distinguishes frontier models from each other because most of them have already seen its answers.
That is not a rhetorical claim. Peer-reviewed contamination studies have found 8–18% verbatim overlap between HumanEval’s prompts and the large pretraining corpora (RedPajama-Data-1T, StarCoder-Data) that modern code models train on, per “Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models” (arXiv, 2024). A benchmark whose answers are partially inside the training set stops measuring reasoning and starts measuring memorization. HumanEval is a useful case study in exactly how that happens, how long it takes anyone to notice, and what a coding benchmark looks like once it has been replaced.
What HumanEval originally measured
HumanEval was introduced by Chen et al. in the July 2021 OpenAI paper “Evaluating Large Language Models Trained on Code”. It consists of 164 hand-written Python programming problems, each with a function signature, docstring, and hidden unit tests, scored with the pass@k metric the paper introduced (the probability that at least one of k sampled completions passes all tests). At launch, Codex — the model the paper was built to evaluate — scored 28.8% pass@1. The base GPT-3 model scored 0%; the contemporary open alternative GPT-J reached 11.4%. In that context, HumanEval was a genuine differentiator: most models failed most problems, and small architectural or training differences moved the score meaningfully.
That gap is why HumanEval became the default coding benchmark for half a decade. It is also why, once frontier labs started optimizing directly against it, the gap collapsed.
Where the score sits in mid-2026 — and why trackers disagree
Three independent leaderboards published HumanEval snapshots in July 2026, and they do not agree with each other — which is itself the point. PricePerToken’s July 21, 2026 snapshot places Claude Sonnet 4.5 on top at 97.6% pass@1, with DeepSeek R1 close behind at 97.4%. llm-stats.com, tracking 66 models on a 0–1 scale, has OpenBMB’s MiniCPM-SALA leading at 95.1%, with the tracked average already at 0.80. BenchLM.ai’s July 2026 table, built from a smaller set of exact-source runs, puts DeepSeek V4 Pro Base first at just 76.8% — a full 20 points below the other two trackers, which BenchLM itself flags by noting its HumanEval table is too small to support a broad frontier verdict.
None of these disagreements are noise. They reflect real methodological differences — prompt formatting, temperature and sampling settings, how many completions are drawn per problem, and which problems get re-run after a model update — layered on top of a test set that’s small (164 problems) and largely memorized. When a benchmark is saturated, small implementation choices start to dominate the reported score more than the model’s actual capability does.
| Tracker | Top model (HumanEval) | Score | Notable spread |
|---|---|---|---|
| PricePerToken (Jul 21, 2026) | Claude Sonnet 4.5 | 97.6% | Frontier cluster within ~1 point |
| llm-stats.com (66 models tracked) | MiniCPM-SALA | 95.1% | Tracked average: 80% |
| BenchLM.ai (Jul 2026, smaller sample) | DeepSeek V4 Pro Base | 76.8% | ~20 points below the other two trackers |
Separately, aggregator commentary circulating in mid-2026 (not an original tracker) cites Gemini 3.1 Pro at 94.3%, Claude Opus 4.6 at 91.3%, and Qwen3.5-Plus at 88.4% — consistent with the broader pattern: frontier models cluster in the high 80s to high 90s, with a compressed spread that leaves little room to distinguish genuine capability gains from evaluation noise.
The contamination evidence, specifically
Score compression alone doesn’t prove contamination — models could simply have gotten better at the underlying skill. Two independent lines of evidence point to leakage rather than pure capability gain. First, the corpus-overlap study cited above directly searched pretraining sets for exact string matches of HumanEval’s natural-language prompts and found meaningful overlap in both RedPajama-Data-1T and StarCoder-Data, two corpora used to train open code models. Decontamination filters based on string matching missed a portion of this overlap; the same research notes that simple paraphrasing or translation of the test data is enough to bypass string-based decontamination entirely.
Second, the EvalPlus project directly tested whether HumanEval’s grading was too permissive rather than models being contaminated. Its “Is Your Code Generated by ChatGPT Really Correct?” paper expanded HumanEval’s test suite roughly 80x — from fewer than 10 test cases per problem to more than 700 on average — producing HumanEval+. Under that harder test suite, pass@k scores for the same models dropped by 19.3 to 28.9 percentage points, and model rankings changed. That result shows two failure modes stacked on top of each other: HumanEval’s original tests were too weak to catch subtly wrong code, and on top of that, some fraction of the remaining correct answers were plausibly memorized rather than derived.
What replaced it
The fix the field converged on is structural rather than incremental: instead of patching one static test set, use a benchmark that keeps generating new, unseen problems. LiveCodeBench scrapes new competitive-programming problems from LeetCode, Codeforces, and AtCoder on a rolling monthly basis, timestamps every problem’s publication date, and lets evaluators filter the test set to only problems published after a given model’s training cutoff — which makes memorization structurally impossible for the filtered subset, not just statistically unlikely.
For agentic and real-world coding work, SWE-bench, particularly the Verified variant, has become the more commonly cited benchmark in 2026, since it grades models on resolving actual GitHub issues against a real codebase rather than self-contained function-completion problems. Between LiveCodeBench’s contamination resistance and SWE-bench Verified’s realism, HumanEval’s role has shifted from “the coding benchmark” to a historical floor check — useful for confirming a model still clears a low bar, useless for ranking frontier systems against each other.
This is not a story unique to coding. HumanEval’s trajectory — fast adoption, optimization pressure, saturation, contamination evidence, replacement by a harder or rotating test — is the same arc MMLU followed before MMLU-Pro, and the same pressure now facing GPQA as frontier scores climb toward its ceiling. A static, publicly available test set has a shelf life measured in the labs’ next few training runs, not years.
The knock-on effect matters beyond benchmark trivia. Frontier labs’ own reported coding gains partly rest on the same contaminated evaluation surface — a pattern worth keeping in mind alongside documented model limitations more broadly, where benchmark scores and real-world reliability frequently diverge.
FAQ
Is HumanEval still worth running at all in 2026?
As a sanity check, yes — a model that fails HumanEval badly has a real problem. As a way to rank or compare frontier models, no: the score compression (88–99% across most trackers) and documented 8–18% corpus overlap mean the remaining differences are more likely measurement artifacts than capability differences.
Why do HumanEval scores vary by 20 points between trackers?
Different trackers use different prompt templates, sampling temperatures, numbers of completions per problem, and model snapshots, and some track far fewer models with more careful exact-source verification than others. On a saturated, 164-problem test set, those methodology choices can swing the reported score more than the model’s actual coding ability does.
What should I check instead of HumanEval?
For general code-generation capability, use LiveCodeBench filtered to problems published after the model’s training cutoff. For realistic engineering tasks, use SWE-bench Verified, which grades models against real GitHub issues rather than isolated function stubs. See our guide to evaluating LLMs for how to weigh multiple benchmarks together rather than trusting a single leaderboard.
Last updated July 28, 2026. This page is refreshed as benchmarks and scores move.