Aider’s polyglot benchmark scores frontier coding models on 225 hand-picked Exercism problems across six languages, and on its own official leaderboard the top score is still GPT-5 (high reasoning) at 88.0% pass_rate_2 — a number that hasn’t moved since August 23, 2025, over a year before this was written. That stall is not a flaw in the benchmark; it’s a direct consequence of how the benchmark resists gaming, and it’s worth understanding both halves of that trade-off.
What the benchmark actually measures
Aider — the open-source AI pair-programming CLI — maintains a benchmark harness built on 225 of the hardest exercises from Exercism, split across C++, Go, Java, JavaScript, Python, and Rust, according to the project’s own benchmark README. Each model gets two attempts per problem: if the first attempt fails its unit tests, the model sees the failure output and gets one retry. The headline metric, pass_rate_2, is the percentage of the 225 problems solved correctly after that second attempt.
Crucially, this isn’t a pure reasoning test. It’s an end-to-end test of whether a model can turn a natural-language coding request into a correctly formatted code edit that Aider can mechanically apply to real files — the harness also reports percent_cases_well_formed, which tracks how often the model’s diff or edit format parses cleanly at all. A model can reason its way to a correct fix and still lose points if it can’t format the edit the way the tool expects.
The exercises were deliberately chosen because, per the original December 2024 write-up, they resisted easy solutions by the frontier models of the time. That was itself a response to benchmark saturation: Aider’s earlier Python-only Exercism benchmark had become oversaturated near the top, the same failure mode covered in our HumanEval contamination case study.
The current leaderboard
Per Aider’s own hosted leaderboard (pulled today), here is where the top of the field sits:
| Model | pass_rate_2 | Well-formed edits | Cost (full run) | Run date |
|---|---|---|---|---|
| GPT-5 (high) | 88.0% | 91.6% | $29.08 | 2025-08-23 |
| GPT-5 (medium) | 86.7% | 88.4% | $17.69 | 2025-08-25 |
| o3-pro (high) | 84.9% | 97.8% | $146.32 | 2025-06-28 |
| Gemini 2.5 Pro Preview 06-05 (32k think) | 83.1% | 99.6% | $49.88 | 2025-06-06 |
| o3 (high) | 81.3% | 94.7% | $21.23 | 2025-06-25 |
| Grok 4 (high) | 79.6% | 97.3% | $59.62 | 2025-07-11 |
| DeepSeek-V3.2-Exp (Reasoner) | 74.2% | 97.3% | $1.30 | 2025-10-03 |
| Claude Opus 4 (32k thinking) | 72.0% | 97.3% | $65.75 | 2025-05-25 |
| DeepSeek-V3.2-Exp (Chat) | 70.2% | 98.2% | $0.88 | 2025-10-03 |
| o1 (high) — benchmark debut leader | 61.7% | 91.5% | $186.50 | 2024-12-21 |
Two things jump out. First, well-formed-edit rate and pass rate don’t track together: o3-pro and the Gemini 2.5 Pro entry both format edits more reliably than GPT-5 (high) does, yet solve fewer problems — confirming the benchmark really is measuring two separate skills, not one. Second, DeepSeek-V3.2-Exp’s reasoner variant matches or beats several models that cost 20–100x more per full run, the same cost-efficiency story SWE-bench Explained tells about DeepSeek on agentic coding tasks.
Since the benchmark launched in December 2024 with o1 (high) at 61.7%, the frontier has moved up roughly 26 percentage points in a little under nine months — and then stopped moving on this particular leaderboard, which is the more interesting story.
Edit format is a hidden variable
The leaderboard table above hides a methodological detail that changes rankings if you ignore it: the “edit format” column. Aider supports several formats for how a model expresses a code change — diff (unified-diff-style hunks), diff-fenced (diffs wrapped in fenced code blocks, used by some Gemini runs), whole (the model rewrites the entire file), and architect (one model plans the change in natural language, a second “editor” model produces the actual diff). These aren’t cosmetic choices. The o3 (high) + gpt-4.1 architect-mode entry, for example, reaches 100.0% well-formed edits precisely because GPT-4.1 as the dedicated editor model is good at producing clean diffs, even though o3 alone is doing the reasoning. A model’s raw coding ability and its benchmark score are only loosely coupled once edit format enters the picture, which is why Aider’s own documentation recommends comparing models within the same edit format rather than across the full leaderboard indiscriminately.
This also means a vendor claiming “our model scores X% on Aider polyglot” without specifying edit format is citing an incomplete number — the same model can post meaningfully different scores under whole versus diff depending on how disciplined it is about instruction-following on structured output, independent of its underlying coding competence.
Why “vendors can’t game it” is only half true
The claim in this post’s own headline needs a caveat. The 225 exercises and their canonical solutions live in a public GitHub repository, Aider-AI/polyglot-benchmark. Nothing stops a lab from including that repository in a pretraining corpus, and nothing in the harness detects if they did. That is a real and open contamination vector — the same category of risk we’ve documented for HumanEval and that our benchmark evaluation guide flags as the single biggest reason to distrust any single leaderboard number in isolation.
What actually makes polyglot benchmark harder to game isn’t secrecy — it’s governance. Results are contributed via pull request to the public data files in aider/website/_data, and per the harness README, reproducing a result requires the model name, edit format, and exact commit hash of the aider codebase at benchmark time. A vendor can’t simply publish a self-reported score; someone has to actually run the docker harness against the real 225 problems and submit the raw yaml. That process is slow, unpaid, and community-run, which is exactly why the top score is stuck at an August 2025 number: nobody has yet submitted a PR benchmarking whatever labs shipped between the last recorded run and today, even though multiple newer frontier models exist. A benchmark that resists gaming by being hard to update is trading one problem for another.
Where it sits among coding and agent benchmarks
Polyglot benchmark measures single-turn code editing, not multi-step tool use. For the broader agentic-coding picture — repo-scale tasks, terminal environments, and tool-calling loops — see our companion pieces on SWE-bench (real GitHub issue resolution), Terminal-bench (agentic terminal tasks), and our map of agent benchmarks covering tau-bench and OSWorld. None of these four measure the same thing, and a model that leads on one can lag on another — which is itself evidence against any single benchmark being the full picture, a point we return to constantly on this site.
FAQ
What does pass_rate_2 actually mean on the Aider leaderboard?
It’s the percentage of the 225 exercises solved correctly after a second attempt, where the model is shown its own unit test failures from attempt one before retrying. pass_rate_1 (first-try accuracy, without that feedback) is reported separately and is always lower — for GPT-5 (high) it’s 52.0% versus the 88.0% pass_rate_2 figure, per the same August 2025 leaderboard entry.
Can vendors still game the Aider polyglot leaderboard?
Not by self-reporting a number — every entry requires a PR with a reproducible commit hash and raw run data. But the exercises and reference solutions are public on GitHub, so training-data contamination remains an open, undetected risk the maintainers themselves haven’t ruled out.
Why hasn’t the top score changed since August 2025?
Because updates depend on volunteers running the docker harness and submitting results, not on Aider’s maintainers proactively benchmarking every new release. That’s the direct cost of the PR-gated governance model that makes the benchmark hard to fake in the first place.
Last updated August 20, 2026. This page is refreshed as benchmarks and scores move.