How Hallucination Is Measured: Benchmarks Behind the Claims

A model can post a sub-2% hallucination rate on one leaderboard and still get the majority of unfamiliar factual questions wrong on another — because “hallucination benchmark” is not one test, it’s four different tests wearing the same name.

Vectara’s grounded-summarization leaderboard, Google’s FACTS Grounding, OpenAI’s SimpleQA, and Artificial Analysis’s AA-Omniscience all claim to measure hallucination. They don’t measure the same thing, they don’t agree on which model is safest, and the gap between them is the most useful finding in this space right now. Here’s what each one actually tests, what the current numbers say, and where they contradict each other.

Three different failure modes, one word

“Hallucination” gets used for at least three distinct behaviors, and conflating them is why leaderboard rankings look inconsistent from one benchmark to the next:

Grounded (RAG) hallucination — the model is given a source document and asked to summarize or answer from it, then generates claims not supported by that source. This is what Vectara’s HHEM and Google’s FACTS Grounding measure.

Parametric (closed-book) hallucination — the model is asked a factual question with no supporting context and has to rely on what it memorized during training. OpenAI’s SimpleQA and the older TruthfulQA test this.

Calibration failure — the model doesn’t just get a fact wrong, it states a wrong answer with confidence instead of admitting uncertainty. Artificial Analysis’s AA-Omniscience is built specifically around this distinction, scoring abstention as neutral rather than penalizing it. For the general methodology behind why these distinctions matter, see our guide to evaluating LLMs.

Grounded summarization: Vectara HHEM and Google FACTS Grounding

Vectara’s Hallucination Leaderboard (GitHub, last updated May 11, 2026) feeds each model a source document and checks whether its summary stays faithful to it, using HHEM-2.3, Vectara’s commercial hallucination evaluation model (an open-source variant, HHEM-2.1-Open, remains on Hugging Face and Kaggle); an LLM-as-judge layer called FaithJudge was introduced in a May 2025 paper reporting roughly 60% fewer ranking errors than the original HHEM scorer alone. The important change for anyone citing this leaderboard: the old short-document table — the one where Gemini-2.0-Flash posted 0.7% and GPT-4o 1.5% — has been retired to an archive branch (hhem-2.3-old-dataset), and the live leaderboard now runs on a single, much larger private corpus of over 7,700 articles spanning news, technology, science, medicine, legal, sports, business and education, from 50 words to 24,000. Numbers moved substantially: on the current table Ant Group’s finix_s1_32b leads at 1.8%, followed by GPT-5.4-nano at 3.1% and Gemini-2.5-flash-lite at 3.3%, while GPT-4o now sits at 9.6%, Claude Sonnet 4 at 10.3% and Claude Opus 4.5 at 10.9%. A “Vectara hallucination rate” quoted without saying which dataset produced it is no longer a meaningful citation. Ant Group’s finix_s1_32b first entered the leaderboard in April 2026 and still holds the top spot.

That ranking flips on Vectara’s harder, newer test using documents up to 32,000 tokens across law, medicine, finance, and education: reasoning models including GPT-5, Claude Sonnet 4.5, and Grok-4 all cross 10% hallucination rates on the longer documents, roughly two to three times their short-document scores — evidence that extended “thinking” doesn’t reliably reduce ungrounded claims and can amplify them on harder inputs.

Google DeepMind’s FACTS Grounding benchmark asks a similar question a different way: 1,719 examples with source documents up to 32k tokens, scored on whether responses are both accurate and fully grounded. On the current Kaggle leaderboard, Gemini 2.5 Pro Preview 06-05 leads at 0.878, against a 13-model average of 0.702 (llm-stats.com mirrors the same ranking).

That single-benchmark framing is now out of date on Google’s side. FACTS Grounding has been absorbed into the FACTS Benchmark Suite, a four-part factuality suite Google DeepMind runs with Kaggle: an updated Grounding v2, plus a Parametric benchmark (closed-book trivia answerable from Wikipedia), a Search benchmark (same web-search tool given to every model, so retrieval setup isn’t a confound), and a Multimodal benchmark (factual answers about input images). The four total 3,513 public examples, with a matching private held-out set, and the headline FACTS Score is the average accuracy across public and private sets on all four. Gemini 3 Pro leads at 68.8%; every one of the 15 leading models DeepMind evaluated scored below 70% overall, with Multimodal the weakest slice across the board. Google reports Gemini 3 Pro cutting the Search error rate 55% and the Parametric error rate 35% relative to Gemini 2.5 Pro. The practical consequence for readers: “FACTS score” now names two different things — the grounding-only number and the four-benchmark composite — and they are not interchangeable.

Closed-book factuality: SimpleQA and the end of TruthfulQA

OpenAI’s SimpleQA (October 2024) strips away the source document entirely and asks short factual questions the model must answer from memory. On the pricepertoken.com leaderboard as of July 16, 2026, Gemini 2.5 Pro tops the field at 53.0% accuracy, followed by Qwen3 235B A22B Instruct 2507 at 50.6%, across an average score of 18.9 over 34 evaluated models — a reminder that closed-book factual recall remains far from solved even for frontier systems. A separate independent test from AIMon’s “un-leaderboard” put GPT-5 (gpt-5-main) at 46% SimpleQA accuracy and a 47% hallucination rate, underscoring that most wrong answers here are confidently wrong, not “I don’t know.”

SimpleQA is also the clearest live example of trackers publishing incompatible numbers under one name. llm-stats.com’s SimpleQA table (47 models, updated September 2026, and flagged by the site itself as 0 verified / 47 self-reported) puts DeepSeek-V3.2-Exp on top at 97.1% and Grok 4 Fast at 95.0%, while placing Gemini 2.5 Pro at 50.8% — in the same band every other source reports. A 97% figure is roughly double anything any independent run has produced on this benchmark and almost certainly reflects a different denominator (accuracy among attempted answers rather than over all questions) rather than a capability gap. Treat any SimpleQA number above the mid-50s as needing its metric definition attached before you cite it.

A cleaner successor now exists. SimpleQA Verified (arXiv:2509.07968) is a curated, reliability-focused subset built to strip out the label noise and redundancy in the original set. Google reports Gemini 2.5 Pro at 54.5% accuracy and Gemini 3 Pro at 72.1% on it — the same generational jump that shows up in the FACTS Suite, measured on questions with verified single answers. It is the closed-book benchmark to migrate to as the original SimpleQA’s label quality becomes the limiting factor.

TruthfulQA, the benchmark that popularized this category, is now widely reported as saturated: by May 2026, GPT-5.x, Claude Opus 4.7, Gemini 3.x, and Llama 4 all score above 85% on the MC2 metric, leaving little room to distinguish frontier models. It hasn’t been formally retired — it still functions as a seed set for product-specific misconception testing — but it no longer discriminates at the top of the leaderboard the way it did at release. For more on benchmarks that stopped being useful once frontier models cleared them, see our piece on MMLU and MMLU-Pro.

Calibration: AA-Omniscience penalizes confident wrong answers

Artificial Analysis’s AA-Omniscience takes a different approach: 6,000 questions across 42 topics in six domains (business, health, law, software engineering, humanities, and science/engineering/math), scored on an index from -100 to 100 that rewards correct answers, penalizes incorrect ones, and treats abstention as neutral. In the original November 2025 paper, Claude 4.1 Opus posted the top score at just 4.8 — one of only three models to score above zero at all, meaning most frontier models at the time answered incorrectly more often than correctly once penalties for confident hallucination were applied. On the live leaderboard as of September 2026 the ceiling has moved a long way from that 4.8: Claude Opus 5.5 (Adaptive Reasoning, Max Effort) leads the index at 46, followed by GPT-6 Astra (high) at 44 and Claude Fable 5.1 at 43, with Claude Fable 5.1 topping the separate raw-accuracy metric at 67%. On the accompanying hallucination-rate sub-metric — defined as incorrect answers as a share of all non-correct responses — the lowest scores belong to small models that abstain heavily rather than to knowledgeable ones: MiniCPM5-1B (non-reasoning) posts 1%, with G9v3-3B at 12%. A low rate here still does not mean higher accuracy.

One caution on secondhand citations of this benchmark: the AA-Omniscience Index is bounded at -100 to 100 by construction, but llm-stats.com currently reports an index of 126.000 for Grok 4.5, a value the metric cannot produce. Where a mirror and the originating lab disagree on a bounded metric, the originating lab’s leaderboard is the one to cite.

Where the trackers disagree

Line these four benchmarks up and the “best” model changes every time: Gemini leads grounded summarization (Vectara, FACTS) and closed-book recall (SimpleQA), while Claude models lead the calibration-weighted AA-Omniscience index. That’s not a contradiction to explain away — it’s the finding. A model tuned to stay faithful to a supplied document isn’t necessarily the same model that best recognizes the edge of its own knowledge when no document is supplied.

Benchmark	What it measures	Current leader	Score	As of
Vectara HHEM Leaderboard (HHEM-2.3, 7,700+ article corpus)	Grounded summarization faithfulness	antgroup/finix_s1_32b	1.8% hallucination rate	May 11, 2026 snapshot
Google FACTS Benchmark Suite (Grounding v2 + Parametric + Search + Multimodal)	Composite factuality accuracy	Gemini 3 Pro	68.8% FACTS Score (all 15 models below 70%)	2026 Kaggle leaderboard
OpenAI SimpleQA	Closed-book factual recall	Gemini 2.5 Pro (trackers disagree; see above)	53.0% accuracy	Jul 16, 2026
SimpleQA Verified	Closed-book recall on de-noised single-answer questions	Gemini 3 Pro	72.1% accuracy (Gemini 2.5 Pro: 54.5%)	2026
TruthfulQA (MC2)	Resistance to common misconceptions	Multiple frontier models	>85% (saturated)	May 2026
AA-Omniscience Index	Knowledge + calibration under uncertainty	Claude Opus 5.5 (current); Claude 4.1 Opus (orig. paper)	46 current; 4.8 at launch	Nov 2025 / Sep 2026

The practical takeaway for anyone evaluating a model for production: match the benchmark to the task. If you’re building retrieval-augmented systems where the model only needs to stay faithful to supplied documents, Vectara and FACTS Grounding are the closer proxy. If your use case involves open-ended factual questions with no supporting context, weight SimpleQA and AA-Omniscience more heavily — and specifically favor AA-Omniscience if wrong-but-confident answers are more costly to you than the model occasionally saying “I don’t know.” This is the same reasoning we apply across the LLM evaluation hub and in our Limitations Ledger, which tracks documented failure modes — including hallucination — across individual tools.

It also explains a pattern that shows up repeatedly in tool-specific writeups: our ChatGPT limitations and Gemini limitations pages both document hallucination complaints despite each vendor’s model leading at least one benchmark above. Leading a benchmark measures performance on that benchmark’s specific failure mode — not immunity from hallucination in general.

FAQ

What’s the difference between a “hallucination benchmark” and a “factuality benchmark”?
In practice the terms overlap, but hallucination benchmarks like Vectara’s HHEM specifically test whether a model stays faithful to a given source document (grounded generation), while factuality benchmarks like SimpleQA test whether the model’s own memorized knowledge is correct with no source provided (parametric knowledge). Both are sometimes called “hallucination rate,” which is the source of most cross-benchmark confusion.

Which model hallucinates the least?
It depends entirely on which benchmark you ask. Gemini models currently lead Vectara’s grounded-summarization leaderboard, Google’s FACTS Grounding, and OpenAI’s SimpleQA closed-book test, while Claude models currently lead Artificial Analysis’s AA-Omniscience calibration index. There is no single model that tops all four as of this writing.

Why do reasoning models sometimes hallucinate more than non-reasoning models?
On Vectara’s harder benchmark using documents up to 32,000 tokens, reasoning models including GPT-5, Claude Sonnet 4.5, and Grok-4 all exceeded 10% hallucination rates — two to three times their scores on shorter documents. The leading explanation is that longer chains of reasoning give the model more opportunities to drift from the source text before producing a final answer, and extended “thinking” doesn’t include an automatic fact-check against the original document.

Last updated September 24, 2026. This page is refreshed as benchmarks and scores move.