ARC-AGI-2 is the benchmark where two different scoreboards tell two different stories: public leaderboards already show several models clearing 85%, while the official ARC Prize contest — run under strict cost limits on a private task set — has not confirmed anyone claiming the $2,000,000 grand prize for that same threshold.
That gap is the whole story. ARC-AGI, created by François Chollet in 2019 and now maintained by the ARC Prize Foundation, was built on a simple premise: give AI systems visual grid puzzles that are easy for humans and deliberately resistant to memorization, then see whether “reasoning” claims hold up when the test can’t be crammed for. Five years and a 50,000x scale-up in base LLM compute later, ARC-AGI-1 is close to saturated. ARC-AGI-2, released in 2025, was designed to reopen the gap — and the current scores show it has, at least for now.
What ARC-AGI-2 actually tests
ARC-AGI-2 keeps the input-output grid format of the original but adds tasks aimed at specific failure modes the ARC Prize team observed in frontier models: symbolic interpretation (assigning meaning to symbols beyond their visual pattern), compositional reasoning (applying multiple interacting rules at once), and contextual rule application (applying a rule differently depending on context rather than fixating on surface pattern-matching) (ARC Prize, arc-agi/2).
Each of the three eval sets — public, semi-private, and private — now holds 120 tasks, up from 100 in the original. Every task must be solvable by at least two of the 400+ members of the general public who took part in a calibration study in San Diego in early 2025, each within two attempts (pass@2). That calibration is what lets the ARC Prize Foundation claim scores are comparable across the three sets, within about one percentage point (ARC Prize, arc-agi/2).
The scores disagree — and the disagreement is the news
Cross-checking two independent trackers on the same date turns up a genuinely large spread. As of September 2, 2026, BenchLM.ai’s self-reported leaderboard has GPT-5.6 Sol at 92.5%, Claude Opus 5 at 90.4%, and Claude Fable 5.1 at 90%, with GPT-5.5 in fourth at 85% (BenchLM.ai, ARC-AGI-2 leaderboard). llm-stats.com, refreshed the same week, shows a different frontier entirely: GPT-5.5 leads at 85.0%, followed by Gemini 3.1 Pro at 77.1% and GPT-5.4 at 73.3% — none of the three top BenchLM entries appear at all. llm-stats.com also flags its own data plainly: of 17 tracked models, zero results are verified and 14 are self-reported, with an overall benchmark status of “Unverified” (llm-stats.com, ARC-AGI v2).
That’s not a rounding difference — it’s a full model generation. Two trackers pulling from vendor claims and community submissions on the same benchmark, in the same week, can disagree about who’s even in the top three.
The official ARC Prize leaderboard is stricter by design and reports something closer to reality under constraints. Its scatter-plot leaderboard plots score against cost per task and explicitly caps eligible runs at $10,000 to run, discarding anything above that limit; Kaggle-contest submissions are held to a roughly $50 compute budget across all 120 evaluation tasks (ARC Prize, leaderboard). Under those limits, the ARC Prize 2025 technical report puts the top Kaggle-constrained score on ARC-AGI-2 at just 24% — nowhere near the 85% grand-prize bar, and nowhere near the 90%+ figures circulating on unconstrained public leaderboards (ARC Prize 2025 Technical Report, arXiv). ARC Prize 2026 keeps the same target: the first system to score 85%+ on the private evaluation set within Kaggle’s efficiency limits wins the Grand Prize from a $2,000,000 pool; if nobody clears it, the prize rolls to 2027. At the time of writing, no confirmed report of that threshold being cleared under contest conditions turned up in search.
So when a vendor or a tracker says a model “beat ARC-AGI-2,” ask which leaderboard: the unconstrained public one, where frontier models can burn arbitrary compute per task, or the cost-capped official one that the prize money is actually tied to. They are not measuring the same thing, and for a piece on how to evaluate LLMs that distinction — compute-unlimited self-reported scores versus constrained, audited ones — is close to the whole game.
Even the human baseline is contested
ARC-AGI-2’s calibration guarantees that every task is solvable by at least two humans within two tries — a task-level solve rate of 100%. But “100% task solvability” and “human accuracy” are different numbers, and sources report the second one inconsistently. The ARC-AGI-2 paper states that, aggregated across all attempts, humans succeeded on 66% of attempted test pairs, and on 75% of tasks when aggregated by task (Chollet et al., ARC-AGI-2 paper, arXiv:2505.11831). BenchLM.ai instead cites “average individual human performance” at 66% but frames it as a single flat number rather than a per-attempt aggregate, while a secondary write-up citing the same underlying 400-person study puts individual average accuracy closer to 60%. The headline “humans solve X%” figure moves by six points depending on how you slice attempts versus tasks — worth knowing before quoting it as a fixed number.
| Source | Snapshot date | Top model / score | Basis |
|---|---|---|---|
| BenchLM.ai | Sep 2, 2026 | GPT-5.6 Sol — 92.5% | Self-reported, unconstrained public leaderboard |
| llm-stats.com | Sep 2026 (last updated Sep 3) | GPT-5.5 — 85.0% | Self-reported (0 of 17 verified), flagged “Unverified” |
| ARC Prize official (Kaggle-constrained) | 2025 contest close | Top submission — 24% | Audited, ≈$50/submission compute cap |
| ARC Prize Grand Prize bar | 2026 contest, ongoing | 85% target, unclaimed as of writing | Private eval set, Kaggle efficiency limits |
What this means for “reasoning” claims
ARC-AGI-1 tells a cautionary tale about what happens after a benchmark gets this kind of attention: it’s now widely described as saturated, with several systems clearing 90%+ once test-time adaptation methods — pioneered by ARC Prize 2024 entrants and by OpenAI’s o3 — arrived in late 2024 (ARC Prize, arc-agi/2). ARC-AGI-2 was built specifically to reopen that gap by removing tasks that were vulnerable to brute-force search and adding the symbolic, compositional, and contextual task types described above. On the evidence above, it’s already showing the same pattern of divergence between self-reported public scores and constrained official ones that ARC-AGI-1 went through — which is presumably why ARC Prize has moved on again. ARC-AGI-3, released alongside the 2026 contest, drops static grids for interactive environments requiring exploration, planning, memory, and goal acquisition; one early report put Gemini 3.1 Pro’s score on it at just 0.37%, suggesting this next version has plenty of room left before the same saturation argument repeats. That interactive framing puts ARC-AGI-3 closer in spirit to agentic evaluations like those covered in our map of AI agent benchmarks and the low, slow-moving scores on OSWorld than to the static puzzle format of ARC-AGI-1 and -2.
None of this settles whether any of these systems “reason” in a meaningful sense — a question our piece on Humanity’s Last Exam runs into from a different angle — but it does mean a bare “X% on ARC-AGI-2” headline number is close to meaningless without a citation to which leaderboard, which date, and which cost constraint produced it. It’s also a reminder that self-reported model scores and hallucinated benchmark claims aren’t unrelated problems; see our breakdown of how hallucination is measured for the parallel issue of unverifiable claims compounding across sources.
FAQ
Is ARC-AGI-2 saturated yet?
Not on the numbers checked here. Public trackers disagree sharply about who leads (85–92.5% depending on source), and the official Kaggle-constrained leaderboard — the one tied to the $2,000,000 grand prize — had only reached 24% as of the 2025 contest close, far short of the 85% grand-prize bar (ARC Prize; ARC Prize 2025 Technical Report).
Why do public leaderboards show much higher scores than the official ARC Prize contest?
Public, self-reported leaderboards let models use arbitrary amounts of compute per task. The official ARC Prize leaderboard caps eligible runs at $10,000 to run, and its Kaggle contest track holds submissions to roughly $50 across all 120 evaluation tasks — a constraint explicitly designed to test efficiency, not just raw capability (ARC Prize).
What’s the difference between ARC-AGI-1, -2, and -3?
ARC-AGI-1 (2019) is now considered largely saturated at the frontier. ARC-AGI-2 (2025) added harder symbolic, compositional, and contextual-rule tasks and expanded each eval set to 120 tasks. ARC-AGI-3, launched alongside ARC Prize 2026, moves from static grid puzzles to interactive environments testing exploration, planning, memory, and goal acquisition — an early report put a leading model’s score at 0.37% (ARC Prize, arc-agi/2).
Last updated September 3, 2026. This page is refreshed as benchmarks and scores move.