ARC-AGI-2 is the benchmark where two different scoreboards tell two different stories: public leaderboards already show several models clearing 85%, while the official ARC Prize contest — run under strict cost limits on a private task set — has not confirmed anyone claiming the $2,000,000 grand prize for that same threshold.
That gap is the whole story. ARC-AGI, created by François Chollet in 2019 and now maintained by the ARC Prize Foundation, was built on a simple premise: give AI systems visual grid puzzles that are easy for humans and deliberately resistant to memorization, then see whether “reasoning” claims hold up when the test can’t be crammed for. Five years and a 50,000x scale-up in base LLM compute later, ARC-AGI-1 is close to saturated. ARC-AGI-2, released in 2025, was designed to reopen the gap — and the current scores show it has, at least for now.
What ARC-AGI-2 actually tests
ARC-AGI-2 keeps the input-output grid format of the original but adds tasks aimed at specific failure modes the ARC Prize team observed in frontier models: symbolic interpretation (assigning meaning to symbols beyond their visual pattern), compositional reasoning (applying multiple interacting rules at once), and contextual rule application (applying a rule differently depending on context rather than fixating on surface pattern-matching) (ARC Prize, arc-agi/2).
Each of the three eval sets — public, semi-private, and private — now holds 120 tasks, up from 100 in the original. Every task must be solvable by at least two of the 400+ members of the general public who took part in a calibration study in San Diego in early 2025, each within two attempts (pass@2). That calibration is what lets the ARC Prize Foundation claim scores are comparable across the three sets, within about one percentage point (ARC Prize, arc-agi/2).
The scores disagree — and the disagreement is the news
Cross-checking two independent trackers on the same date turns up a genuinely large spread — and it keeps moving. As of September 10, 2026, BenchLM.ai’s self-reported leaderboard has GPT-6 Astra (released September 3) leading at 95%, ahead of GPT-5.6 Sol (92.5%) and Claude Opus 5 (90.4%), across 23 tracked models (BenchLM.ai, ARC-AGI-2 leaderboard). llm-stats.com’s own September 2026 refresh corroborates that new leader specifically — GPT-6 Astra tops its 19-model table too, also at 95.0% — a rare case of two independent trackers agreeing on both the leader and the score (llm-stats.com, ARC-AGI v2). That agreement doesn’t extend to the rest of the table, though: llm-stats.com still flags most of its data as self-reported and largely unverified, the same caveat that applied to the snapshot below just over a week earlier.
That earlier snapshot is worth keeping as a reminder of how fast this moves: on September 2, 2026, BenchLM had GPT-5.6 Sol on top at 92.5%, with Claude Opus 5 second at 90.4% and Claude Fable 5.1 close behind at 90% — while llm-stats.com’s table that same week showed a completely different frontier (GPT-5.5 leading at 85.0%, followed by Gemini 3.1 Pro at 77.1% and GPT-5.4 at 73.3%), with none of BenchLM’s top three appearing at all. Both that disagreement and the reshuffle that followed it happened inside the same two-week window that saw Anthropic ship Claude Fable 5.1 (September 1) and OpenAI ship GPT-6 Astra (September 3) — new frontier models routinely obsolete a “current leader” claim within days.
That’s not a rounding difference — across both snapshots, the gap between trackers has spanned a full model generation. Two trackers pulling from vendor claims and community submissions on the same benchmark, in the same week, can disagree about who’s even in the top three — though by mid-September they had, for once, converged on the same leader and score.
The official ARC Prize leaderboard is stricter by design and reports something closer to reality under constraints. Its scatter-plot leaderboard plots score against cost per task and explicitly caps eligible runs at $10,000 to run, discarding anything above that limit; Kaggle-contest submissions are held to a roughly $50 compute budget across all 120 evaluation tasks (ARC Prize, leaderboard). Under those limits, the ARC Prize 2025 technical report puts the top Kaggle-constrained score on ARC-AGI-2 at just 24% — nowhere near the 85% grand-prize bar, and nowhere near the 90%+ figures circulating on unconstrained public leaderboards (ARC Prize 2025 Technical Report, arXiv). ARC Prize 2026 keeps the same target: the first system to score 85%+ on the private evaluation set within Kaggle’s efficiency limits wins the Grand Prize from a $2,000,000 pool; if nobody clears it, the prize rolls to 2027. At the time of writing, no confirmed report of that threshold being cleared under contest conditions turned up in search.
So when a vendor or a tracker says a model “beat ARC-AGI-2,” ask which leaderboard: the unconstrained public one, where frontier models can burn arbitrary compute per task, or the cost-capped official one that the prize money is actually tied to. They are not measuring the same thing, and for a piece on how to evaluate LLMs that distinction — compute-unlimited self-reported scores versus constrained, audited ones — is close to the whole game.
Even the human baseline is contested
ARC-AGI-2’s calibration guarantees that every task is solvable by at least two humans within two tries — a task-level solve rate of 100%. But “100% task solvability” and “human accuracy” are different numbers, and sources report the second one inconsistently. The ARC-AGI-2 paper states that, aggregated across all attempts, humans succeeded on 66% of attempted test pairs, and on 75% of tasks when aggregated by task (Chollet et al., ARC-AGI-2 paper, arXiv:2505.11831). BenchLM.ai instead cites “average individual human performance” at 66% but frames it as a single flat number rather than a per-attempt aggregate, while a secondary write-up citing the same underlying 400-person study puts individual average accuracy closer to 60%. The headline “humans solve X%” figure moves by six points depending on how you slice attempts versus tasks — worth knowing before quoting it as a fixed number.
| Source | Snapshot date | Top model / score | Basis |
|---|---|---|---|
| BenchLM.ai | Sep 10, 2026 | GPT-6 Astra — 95% | Self-reported, unconstrained public leaderboard, 23 models |
| llm-stats.com | Sep 2026 | GPT-6 Astra — 95.0% | Self-reported, 19 models tracked |
| BenchLM.ai (prior snapshot) | Sep 2, 2026 | GPT-5.6 Sol — 92.5% | Self-reported, unconstrained public leaderboard |
| ARC Prize official (Kaggle-constrained) | 2025 contest close | Top submission — 24% | Audited, ≈$50/submission compute cap |
| ARC Prize Grand Prize bar | 2026 contest, ongoing | 85% target, unclaimed as of writing | Private eval set, Kaggle efficiency limits |
What this means for “reasoning” claims
ARC-AGI-1 tells a cautionary tale about what happens after a benchmark gets this kind of attention: it’s now widely described as saturated, with several systems clearing 90%+ once test-time adaptation methods — pioneered by ARC Prize 2024 entrants and by OpenAI’s o3 — arrived in late 2024 (ARC Prize, arc-agi/2). ARC-AGI-2 was built specifically to reopen that gap by removing tasks that were vulnerable to brute-force search and adding the symbolic, compositional, and contextual task types described above. On the evidence above, it’s already showing the same pattern of divergence between self-reported public scores and constrained official ones that ARC-AGI-1 went through — which is presumably why ARC Prize has moved on again. ARC-AGI-3, released alongside the 2026 contest, drops static grids for interactive environments requiring exploration, planning, memory, and goal acquisition; one early report put Gemini 3.1 Pro’s score on it at just 0.37%, suggesting this next version has plenty of room left before the same saturation argument repeats. That interactive framing puts ARC-AGI-3 closer in spirit to agentic evaluations like those covered in our map of AI agent benchmarks and the low, slow-moving scores on OSWorld than to the static puzzle format of ARC-AGI-1 and -2.
None of this settles whether any of these systems “reason” in a meaningful sense — a question our piece on Humanity’s Last Exam runs into from a different angle — but it does mean a bare “X% on ARC-AGI-2” headline number is close to meaningless without a citation to which leaderboard, which date, and which cost constraint produced it. It’s also a reminder that self-reported model scores and hallucinated benchmark claims aren’t unrelated problems; see our breakdown of how hallucination is measured for the parallel issue of unverifiable claims compounding across sources
Is ARC-AGI-2 saturated yet?
Not fully, but two major trackers converged in September 2026. Public trackers disagreed sharply in early September (85–92.5% depending on source); by September 10, BenchLM.ai and llm-stats.com both had GPT-6 Astra on top at 95%. The official Kaggle-constrained leaderboard — the one tied to the $2,000,000 grand prize — had only reached 24% as of the 2025 contest close, far short of the 85% grand-prize bar, and that prize remains unclaimed as of this writing (ARC Prize; ARC Prize 2025 Technical Report).
Why do public leaderboards show much higher scores than the official ARC Prize contest?
Public, self-reported leaderboards let models use arbitrary amounts of compute per task. The official ARC Prize leaderboard caps eligible runs at $10,000 to run, and its Kaggle contest track holds submissions to roughly $50 across all 120 evaluation tasks — a constraint explicitly designed to test efficiency, not just raw capability (ARC Prize).
What’s the difference between ARC-AGI-1, -2, and -3?
ARC-AGI-1 (2019) is now considered largely saturated at the frontier. ARC-AGI-2 (2025) added harder symbolic, compositional, and contextual-rule tasks and expanded each eval set to 120 tasks. ARC-AGI-3, launched alongside ARC Prize 2026, moves from static grid puzzles to interactive environments testing exploration, planning, memory, and goal acquisition — an early report put a leading model’s score at 0.37% (ARC Prize, arc-agi/2).
Last updated September 18, 2026. This page is refreshed as benchmarks and scores move.
Pingback: State of AI Benchmarks: Q3 2026 - Tech Blog