LiveCodeBench doesn’t try to keep its test questions secret — it timestamps them instead, so anyone can rescore a model using only the problems published after that model’s training cutoff. That single design choice, introduced in the original 2024 paper by Naman Jain and colleagues (arXiv:2403.07974), turned a static coding test into what researchers now call a “rolling benchmark.” Two years and six dataset releases later, the approach has held up well enough that three separate trackers are still running it in August 2026 — and, tellingly, none of them agree on who’s winning right now.
What LiveCodeBench Measures
LiveCodeBench continuously scrapes new competitive-programming problems from LeetCode, AtCoder, and Codeforces, then evaluates models across four scenarios: code generation, self-repair, test-output prediction, and code execution (with an optional chain-of-thought variant), according to the official repository. Scoring uses pass@1 and pass@5 against hidden test cases rather than a human-graded rubric.
The benchmark ships in dated releases rather than one frozen set. The GitHub README lists six so far: release_v1 (400 problems, May 2023–March 2024) through release_v6 (1,055 problems, May 2023–April 2025). Each new version adds problems published after the previous cutoff, so the pool of “unseen” material keeps growing even as older problems age into potential training data.
The Contamination Problem It Was Built to Solve
Static benchmarks decay because their answers eventually end up in training corpora — scraped from solution repos, forum posts, or the benchmark’s own leaderboard pages. Our HumanEval contamination case study covers how that played out for the field’s original coding test. LiveCodeBench’s authors built the dating mechanism specifically to sidestep this: in the original paper, they filtered results to problems released after August 2023 specifically to check whether DeepSeek’s coding models had been contaminated by earlier LiveCodeBench problems already circulating online. The same start-date filter (the runner’s --start_date flag) lets anyone repeat that check for any model with a known training cutoff.
The Fix Isn’t Airtight
Timestamping closes one contamination path but not every gap. The project’s own ERRATA.md lists roughly a dozen problems that accept multiple correct outputs but are graded against only one, two interactive problems the autograder can’t evaluate at all, and four with erroneous test cases scraped from the source contests. None of that is contamination in the training-leak sense — it’s autograding noise the maintainers openly disclose, which is itself unusual candor for a benchmark this widely cited. It also means two labs running the “same” LiveCodeBench release can land slightly different scores just from how they handle those edge cases.
The Academic Critique: Rolling Windows Aren’t a Complete Answer
A 2025 paper by Rahmatullah Musawi and Lu Sheng, “Towards Contamination Resistant Benchmarks” (arXiv:2505.08389), names LiveCodeBench’s approach directly as an example of “a dynamic benchmark that receives continuous updates, with new test instances added after a model’s training cutoff,” then flags its central weakness: because developers rarely disclose exact training cutoffs, or whether they’ve continued training a model after a stated cutoff, the paper argues “it is almost impossible to determine whether developers have continued training their models after the cutoff using the updated instances.” In other words, the rolling window assumes good-faith disclosure from labs about their own training data — something LiveCodeBench’s maintainers have no independent way to verify. The same paper proposes synthetic, cipher-based tasks as an alternative that sidesteps the disclosure problem entirely, but for a domain-specific skill like competitive programming, that kind of substitution isn’t really available. It’s part of why LiveCodeBench, imperfect as it is, remains the reference point for coding contamination checks rather than being replaced outright.
Three Trackers, Three Different Leaders
Checking the leaderboard on August 25–26, 2026 makes the bigger problem visible: “LiveCodeBench” isn’t one number. It’s whatever release, date window, sampling settings, and problem-pruning choice a given tracker used.
| Tracker | #1 model | Score | Models shown | Data date |
|---|---|---|---|---|
| BenchLM.ai | Qwen3.7 Max | 91.6% | 7 | Aug 25, 2026 |
| Vals AI | Claude Fable 5 | 89.78% | 26 | Aug 19, 2026 |
| Price Per Token (via Artificial Analysis) | Gemini 3 Pro Preview | 91.7% | 194 | Aug 26, 2026 |
BenchLM’s own methodology note describes its page as “a sourced result ledger, not one controlled rerun” — it compiles published numbers from labs and other trackers rather than running one harness, and it carries only seven models. Vals AI, by contrast, reimplements the benchmark itself across 26 models and reports Claude Fable 5 at 89.78%, with Claude Opus 5, Gemini 3.1 Pro Preview, Gemini 3.6 Flash, and GPT-5.2 Codex all within about two points of each other on the hard split. Price Per Token’s table, sourced from Artificial Analysis across 194 models, doesn’t even include Claude Fable 5, Claude Opus 5, or Gemini 3.1 Pro Preview — its best Claude entry is Claude Opus 4.5 Thinking at 87.1%. These aren’t rounding differences; they’re three different model rosters, scored under three different setups, all labeled “LiveCodeBench.”
Why the Numbers Diverge
Score comparability on LiveCodeBench depends on matching release version, date window, scenario, pass@k, sampling count, temperature, and whether a tracker uses the pruned code_generation_lite set or the original full test cases — a point BenchLM’s own methodology page makes explicitly. A model evaluated on release_v5 problems isn’t taking the same test as one evaluated on release_v6, and a lab self-reporting a number under undisclosed settings isn’t comparable to an independent rerun at all. Vals AI’s data does converge on one real trend, though: the easy split is close to solved, so hard problems — not easy or medium ones — now decide the ranking, and the leading models sit within roughly two percentage points of each other there.
That pattern echoes what happened to MMLU, where saturation forced a harder successor, and it’s the same reasoning behind community-run tests like Aider Polyglot, which vendors can’t self-report into. It’s also why repository-level evaluations such as SWE-bench exist alongside LiveCodeBench: a high competitive-programming score says nothing about navigating an existing codebase, a caveat Vals AI’s own FAQ makes explicit as well.
What This Means If You’re Evaluating a Model
Don’t quote a bare LiveCodeBench percentage without its release version, date window, and tracker attached. Ask whether the number comes from a controlled rerun (Vals AI, Artificial Analysis) or a self-reported ledger (BenchLM’s current page says as much). And treat any single-digit percentage-point gap near the top as noise until you’ve confirmed both models were scored on the identical problem set — our guide to evaluating LLMs covers how to build that kind of apples-to-apples comparison across benchmarks generally.
FAQ
Does LiveCodeBench eliminate contamination entirely? No. Dating problems by release date closes the “the answer was already in the training set” path, but it doesn’t stop newer problems from being scraped and discussed publicly after release, and it doesn’t fix autograding noise — the project’s own errata lists a dozen-plus problems with test-case issues that produce false negatives regardless of contamination.
Why do different LiveCodeBench leaderboards show different #1 models? Because trackers use different releases (v1 through v6), different date windows, different pass@k settings, and different problem sets (pruned “lite” versus original), and some — like BenchLM’s current page — aggregate self-reported numbers rather than rerunning the benchmark themselves. As of late August 2026, BenchLM, Vals AI, and Price Per Token/Artificial Analysis each show a different model in first place.
Is LiveCodeBench still a hard benchmark for frontier models? On the easy split, no — it’s close to saturated. On the hard split, yes: Vals AI’s data shows the top models clustered within about two percentage points of each other there, which is now the tier that actually separates frontier coding models.
Last updated August 26, 2026. This page is refreshed as benchmarks and scores move.