A model can post a 0.7% hallucination rate on one leaderboard and still get the majority of unfamiliar factual questions wrong on another — because “hallucination benchmark” is not one test, it’s four different tests wearing the same name.
Vectara’s grounded-summarization leaderboard, Google’s FACTS Grounding, OpenAI’s SimpleQA, and Artificial Analysis’s AA-Omniscience all claim to measure hallucination. They don’t measure the same thing, they don’t agree on which model is safest, and the gap between them is the most useful finding in this space right now. Here’s what each one actually tests, what the current numbers say, and where they contradict each other.
Three different failure modes, one word
“Hallucination” gets used for at least three distinct behaviors, and conflating them is why leaderboard rankings look inconsistent from one benchmark to the next:
Grounded (RAG) hallucination — the model is given a source document and asked to summarize or answer from it, then generates claims not supported by that source. This is what Vectara’s HHEM and Google’s FACTS Grounding measure.
Parametric (closed-book) hallucination — the model is asked a factual question with no supporting context and has to rely on what it memorized during training. OpenAI’s SimpleQA and the older TruthfulQA test this.
Calibration failure — the model doesn’t just get a fact wrong, it states a wrong answer with confidence instead of admitting uncertainty. Artificial Analysis’s AA-Omniscience is built specifically around this distinction, scoring abstention as neutral rather than penalizing it. For the general methodology behind why these distinctions matter, see our guide to evaluating LLMs.
Grounded summarization: Vectara HHEM and Google FACTS Grounding
Vectara’s Hallucination Leaderboard (GitHub, last updated May 11, 2026) feeds each model a source document and checks whether its summary stays faithful to it, using the HHEM-2.1-open model plus a newer LLM-as-judge layer called FaithJudge, introduced in a May 2025 paper that reported roughly 60% fewer ranking errors than the original HHEM scorer alone. On the original short-document test, Google’s Gemini models and GPT-4o cluster near the top: Gemini-2.0-Flash at 0.7% and GPT-4o at 1.5%, while Claude models range from 4.4% (Sonnet) to 10.1% (Opus), per an April 2026 comparison of the published data. In April 2026 Ant Group’s finix_s1_32b entered the leaderboard at a 1.8% hallucination rate.
That ranking flips on Vectara’s harder, newer test using documents up to 32,000 tokens across law, medicine, finance, and education: reasoning models including GPT-5, Claude Sonnet 4.5, and Grok-4 all cross 10% hallucination rates on the longer documents, roughly two to three times their short-document scores — evidence that extended “thinking” doesn’t reliably reduce ungrounded claims and can amplify them on harder inputs.
Google DeepMind’s FACTS Grounding benchmark asks a similar question a different way: 1,719 examples with source documents up to 32k tokens, scored on whether responses are both accurate and fully grounded. On the current Kaggle leaderboard, Gemini 2.5 Pro Preview 06-05 leads at 0.878, against a 13-model average of 0.702 (llm-stats.com mirrors the same ranking).
Closed-book factuality: SimpleQA and the end of TruthfulQA
OpenAI’s SimpleQA (October 2024) strips away the source document entirely and asks short factual questions the model must answer from memory. On the pricepertoken.com leaderboard as of July 16, 2026, Gemini 2.5 Pro tops the field at 53.0% accuracy, followed by Qwen3 235B A22B Instruct 2507 at 50.6%, across an average score of 18.9 over 34 evaluated models — a reminder that closed-book factual recall remains far from solved even for frontier systems. A separate independent test from AIMon’s “un-leaderboard” put GPT-5 (gpt-5-main) at 46% SimpleQA accuracy and a 47% hallucination rate, underscoring that most wrong answers here are confidently wrong, not “I don’t know.”
TruthfulQA, the benchmark that popularized this category, is now widely reported as saturated: by May 2026, GPT-5.x, Claude Opus 4.7, Gemini 3.x, and Llama 4 all score above 85% on the MC2 metric, leaving little room to distinguish frontier models. It hasn’t been formally retired — it still functions as a seed set for product-specific misconception testing — but it no longer discriminates at the top of the leaderboard the way it did at release. For more on benchmarks that stopped being useful once frontier models cleared them, see our piece on MMLU and MMLU-Pro.
Calibration: AA-Omniscience penalizes confident wrong answers
Artificial Analysis’s AA-Omniscience takes a different approach: 6,000 questions across 42 topics in six domains (business, health, law, software engineering, humanities, and science/engineering/math), scored on an index from -100 to 100 that rewards correct answers, penalizes incorrect ones, and treats abstention as neutral. In the original November 2025 paper, Claude 4.1 Opus posted the top score at just 4.8 — one of only three models to score above zero at all, meaning most frontier models at the time answered incorrectly more often than correctly once penalties for confident hallucination were applied. On the live leaderboard as of August 2026, the top index scores have climbed into the 30s and 40s, with newer Claude models leading; separately, on the accompanying hallucination-rate sub-metric, several smaller open-weight models post very low rates (1–14%) by abstaining heavily rather than by being more knowledgeable — a pattern worth flagging since a low hallucination rate on this metric doesn’t necessarily mean higher accuracy.
Where the trackers disagree
Line these four benchmarks up and the “best” model changes every time: Gemini leads grounded summarization (Vectara, FACTS) and closed-book recall (SimpleQA), while Claude models lead the calibration-weighted AA-Omniscience index. That’s not a contradiction to explain away — it’s the finding. A model tuned to stay faithful to a supplied document isn’t necessarily the same model that best recognizes the edge of its own knowledge when no document is supplied.
| Benchmark | What it measures | Current leader | Score | As of |
|---|---|---|---|---|
| Vectara HHEM Leaderboard | Grounded summarization faithfulness | Gemini-2.0-Flash | 0.7% hallucination rate | May 2026 |
| Google FACTS Grounding | Grounded response accuracy | Gemini 2.5 Pro Preview 06-05 | 0.878 (avg. 0.702) | 2026 leaderboard |
| OpenAI SimpleQA | Closed-book factual recall | Gemini 2.5 Pro | 53.0% accuracy | Jul 16, 2026 |
| TruthfulQA (MC2) | Resistance to common misconceptions | Multiple frontier models | >85% (saturated) | May 2026 |
| AA-Omniscience Index | Knowledge + calibration under uncertainty | Claude models (current); Claude 4.1 Opus (orig. paper) | 4.8 at launch, 30s–40s current | Nov 2025 / Aug 2026 |
The practical takeaway for anyone evaluating a model for production: match the benchmark to the task. If you’re building retrieval-augmented systems where the model only needs to stay faithful to supplied documents, Vectara and FACTS Grounding are the closer proxy. If your use case involves open-ended factual questions with no supporting context, weight SimpleQA and AA-Omniscience more heavily — and specifically favor AA-Omniscience if wrong-but-confident answers are more costly to you than the model occasionally saying “I don’t know.” This is the same reasoning we apply across the LLM evaluation hub and in our Limitations Ledger, which tracks documented failure modes — including hallucination — across individual tools.
It also explains a pattern that shows up repeatedly in tool-specific writeups: our ChatGPT limitations and Gemini limitations pages both document hallucination complaints despite each vendor’s model leading at least one benchmark above. Leading a benchmark measures performance on that benchmark’s specific failure mode — not immunity from hallucination in general.
FAQ
What’s the difference between a “hallucination benchmark” and a “factuality benchmark”?
In practice the terms overlap, but hallucination benchmarks like Vectara’s HHEM specifically test whether a model stays faithful to a given source document (grounded generation), while factuality benchmarks like SimpleQA test whether the model’s own memorized knowledge is correct with no source provided (parametric knowledge). Both are sometimes called “hallucination rate,” which is the source of most cross-benchmark confusion.
Which model hallucinates the least?
It depends entirely on which benchmark you ask. Gemini models currently lead Vectara’s grounded-summarization leaderboard, Google’s FACTS Grounding, and OpenAI’s SimpleQA closed-book test, while Claude models currently lead Artificial Analysis’s AA-Omniscience calibration index. There is no single model that tops all four as of this writing.
Why do reasoning models sometimes hallucinate more than non-reasoning models?
On Vectara’s harder benchmark using documents up to 32,000 tokens, reasoning models including GPT-5, Claude Sonnet 4.5, and Grok-4 all exceeded 10% hallucination rates — two to three times their scores on shorter documents. The leading explanation is that longer chains of reasoning give the model more opportunities to drift from the source text before producing a final answer, and extended “thinking” doesn’t include an automatic fact-check against the original document.
Last updated August 14, 2026. This page is refreshed as benchmarks and scores move.