Gemini Limitations in 2026: Where Long Context Actually Degrades

Gemini 3 Pro advertises a 10-million-token context window, but on the hardest published long-context retrieval test its accuracy falls to roughly one in four correct answers once documents pass the 1-million-token mark — worse than both Claude Opus 4.6 and GPT-5.4 at the same length, despite Gemini having shipped a 1M-token window two full years before either competitor. This page is a documented, sourced ledger of Gemini’s limitations as they stand in September 2026, following the same rules we apply to Claude and ChatGPT in our Limitations Ledger: every claim carries a model version, a named benchmark, a source, and a date. Where trackers disagree, we report the disagreement instead of averaging it away.

Scope and method

“Gemini” in mid-2026 covers a fast-moving lineage: Gemini 1.5 Pro (the first model to ship a 1M-token window, February 2024), Gemini 2.0 Flash, Gemini 2.5 Pro, Gemini 3 Pro (released November 18, 2025, per Google’s launch coverage), Gemini 3 Flash (December 17, 2025), Gemini 3.1 Pro (February 19, 2026), and a rapid Flash cadence through 2026 — 3.5 Flash (May), 3.6 Flash (July 21), 3.7 Flash (August 13), and 3.8 Flash with its restricted Cyber variant (September 2). A limitation measured on one generation does not automatically transfer to the next — Gemini’s short-context scores have improved sharply across this lineage even where its long-context ceiling has not. Each entry below names the specific model and evaluation, following the same rule we use when evaluating LLMs generally: no score without a named benchmark version and date.

The core finding: a ceiling that hasn’t moved in three generations

The most stringent published long-context retrieval test is MRCR v2 8-needle, developed by OpenAI and adopted by all three frontier labs as an internal evaluation standard. It hides eight distinct “needle” facts inside a haystack of distractor text and asks the model to retrieve a specific one on request. Tracking Gemini’s score on this test across generations exposes a pattern that a single snapshot would miss.

Model	Release date	MRCR v2 8-needle, ≤128K	MRCR v2 8-needle, 1M	Source
Gemini 2.0 Flash	Feb 2025	18.4%	10.2%	yage.ai survey, citing Google Gemini blog
Gemini 2.5 Pro	Jun 2025	58.0%	16.4%	Google Gemini 2.5 technical report
Gemini 3 Pro	Nov 2025	77.0%	26.3% / 24.5%*	Awesome Agents vs. contextarena.ai (via yage.ai)
Gemini 3.1 Pro	Feb 2026	84.9%	26.3%	Gemini 3.1 Pro model card
Gemini 3.5 Flash	May 2026	77.3%	26.6%	Gemini 3.5 Flash launch benchmarks, via independent 2026 reviews
Claude Opus 4.6 (for comparison)	Feb 2026	~90%	76.0%	Anthropic Sonnet/Opus 4.6 system card
GPT-5.4 (for comparison)	Mar 2026	79.3%	36.6%	OpenAI GPT-5.4 announcement

*Two independent trackers report slightly different 1M scores for Gemini 3 Pro — 26.3% from Awesome Agents’ reading of Google’s evaluation card, versus 24.5% from contextarena.ai’s third-party test as compiled by yage.ai. The gap is a reminder that MRCR v2 numbers are not directly comparable across sources: reasoning-effort settings, aggregation method (range-level vs. sample-weighted), and needle count all shift the result. Neither figure is “wrong” — they’re testing under different conditions, and the disagreement itself is the data point worth noting.

Read across the rows and a specific, quotable pattern emerges: Gemini’s ≤128K score has climbed from 18.4% to 84.9% in one year — a genuine, well-documented improvement. But its 1M-token score has stayed essentially flat, moving from 16.4% (2.5 Pro) to 26.3% (3.1 Pro), a far smaller gain that leaves Gemini roughly three times below Claude Opus 4.6’s 76% at the same length. Google was first to ship a 1M window, in February 2024; Anthropic didn’t reach 1M generally available until March 13, 2026, per Cursor’s community forum tracking the rollout — and still ships a model that retrieves more reliably at that length.

Context window size and effective context are different numbers

Industry analysis compiled by Awesome Agents (updated April 8, 2026) puts a model’s typical “effective” context capacity — the range within which retrieval stays reliable — at 60–70% of its advertised maximum. Gemini 3 Pro’s own numbers make the case that its effective ratio is considerably worse than that industry average for multi-needle retrieval: a 10-million-token nominal window with a documented 50-point accuracy drop by the 1-million-token mark (from 77.0% at 128K to 26.3% at 1M) means the useful range for hard retrieval tasks is closer to a tenth of the advertised ceiling, not two-thirds of it.

This isn’t uniformly true of every long-context benchmark, which is exactly why naming the benchmark matters. On LongBench v2, a 503-question long-document comprehension test from Tsinghua University, Gemini 2.5 Pro leads the field at 63.3%, ahead of the 53.7% human-expert baseline under time pressure, and Gemini 3 Pro posts the highest published LongBench v2 score of any model at 68.2%. On NVIDIA’s RULER benchmark at 128K tokens, Gemini 1.5 Pro was among only two models (with Jamba-1.5-large) to stay above 90% accuracy, scoring 94.4% while GPT-4 dropped to 81.2%. Gemini is genuinely strong on document comprehension and moderate-length retrieval — the failure mode is specifically the multi-needle, million-token-plus regime that MRCR v2 8-needle is built to expose.

Hallucination: the number depends entirely on which benchmark you cite

Vectara overhauled its hallucination leaderboard in February 2026, expanding the test set to more than 7,700 long articles — up to 32,000 tokens each — spanning legal, medical, financial, and technical domains, a harder and more realistic dataset than the short-summary version it replaced. On that dataset, per Vectara’s own announcement, Gemini 3 Pro scores a 13.6% hallucination rate — worse than Claude Opus 4.6 (12.2%) and far worse than smaller, non-reasoning models like GPT-5.4-nano (3.1%) or Gemini’s own 2.5 Flash-Lite (3.3%). Every state-of-the-art reasoning model in the update exceeded 10%, which suggests the harder dataset is exposing weaknesses that the old, easier benchmark hid across the board — not a Gemini-specific problem, but Gemini doesn’t escape it either.

Set against that, Google’s own FACTS Benchmark Suite — which tests grounding, parametric knowledge, search-tool use, and multimodal factuality — had Gemini 3 Pro leading all tested models with an overall FACTS Score of 68.8% as of Google’s original write-up, with no model clearing 70% at that time; Google’s FACTS leaderboard on Kaggle has since been updated (last change dated September 4, 2026) to show Gemini 3.7 Flash at 71.2% — the first model on that leaderboard to clear the 70% mark — with 3.8 Flash close behind at 69.4% and 3.1 Pro Preview at 67.4%. So the “no model clears 70%” line no longer holds; it describes the leaderboard as it stood before the Flash-tier gains covered below, not its current state. Both figures are “Gemini’s factuality,” measured nine points apart, on benchmarks designed and in one case published by the model’s own vendor. Treat a self-reported leaderboard win with more caution than a third-party one, and note that DeepMind does report real, measurable gains from 2.5 Pro to 3 Pro: a 55% error-rate reduction on FACTS Search and 35% on FACTS Parametric, per the same DeepMind post. For how Gemini’s numbers stack up against Vectara, OpenAI’s SimpleQA, and Artificial Analysis’s calibration-focused AA-Omniscience side by side, see our breakdown of how hallucination is benchmarked.

The Flash line since has kept the ceiling in place and added a wrinkle. Gemini 3.5 Flash (May 2026) posts 77.3% on MRCR v2 8-needle at 128K — a 7.6-point regression against Gemini 3.1 Pro’s 84.9% on the slice of the window most production workloads actually sit in — while reaching 26.6% at 1M on the pointwise test. That 26.6% is the highest 1M figure in the table above, and it is still roughly a quarter correct. Four model generations in, the 1M number has moved from 16.4% to 26.6%; the 128K number has moved 66 points. The gap is the finding.

Update, September 28, 2026: that ceiling held only through Gemini 3.5 Flash. Independent benchmark trackers now show Gemini 3.7 Flash (released August 13, 2026) scoring 97.0% on MRCR v2 8-needle at 128K — up from 3.6 Flash’s 91.8% — and, on the pointwise 1M-token test, 62.5%: more than double every prior figure in this lineage’s 1M column, ahead of GPT-5.4’s 36.6% comparison point, and closing well over half the gap to Claude Opus 4.6’s 76%. That is the largest single generation-over-generation jump this page has tracked at the 1M mark, and it means the “ceiling hasn’t moved” finding above accurately described Gemini’s Pro tier and its earlier Flash releases, but not the current Flash model. Gemini 3.8 Flash, shipped three weeks later on September 2, has no independently confirmed MRCR v2 score at either length as of this writing — treat any figure circulating for it as unverified until a primary tracker publishes one — and Gemini 3.5 Pro still had not shipped as of this update (llm-stats.com, September 2026; Gemini 3.7 Flash model card).

Update, November 6, 2026: Google shipped a new frontier model on October 1, 2026 — Gemini 4 Argon — which supersedes the 3.x lineage tracked above as Google’s top-tier Gemini; a second model, Gemini 4 Carbon, was announced two weeks later and remains in restricted pre-release testing with government and trusted cyber-defender partners as of this writing, not yet broadly available (Google, “The latest AI news we announced in October 2026,” published Nov 2, 2026). Argon’s one published long-context figure uses a different test than the MRCR v2 8-needle numbers this page otherwise tracks: Google’s own model card reports 84.2% on GraphWalks (BFS, F1) across the 256K–1M token range, against 71.8% for GPT-6 Astra and 66.8% for Claude Opus 5.5 at the same length, and 99.7% at up to 128K (Google DeepMind, Gemini 4 Argon model card, published October 13, 2026). That is a large, genuine jump over anything in the MRCR v2 table above, but GraphWalks is a graph-traversal task, not MRCR v2’s needle-retrieval test, so it does not confirm or contradict the MRCR v2 pattern described there — it is a different benchmark, self-reported by the model’s own vendor, the same caution we apply to the FACTS numbers above. No independent tracker had published an MRCR v2 8-needle score for Gemini 4 Argon as of this writing; one third-party long-context test run reported the model failing to complete several planned 900K-token calls because of an API rate limit, which is a quota issue, not a benchmark result. Neither Vectara nor Google’s own FACTS leaderboard has yet published an Argon hallucination or factuality score. Google’s Gemini 4 Argon Safety & Security Report, published alongside the October 13, 2026 model card, says internal evaluation could not rule out the model reaching capability thresholds for cybersecurity and biological (CBRN) uplift, and that Google applied safeguards it assesses bring residual risk below those thresholds — again a vendor self-assessment, not an independent audit. Separately, an advocate general at the EU’s top court issued a non-binding opinion in October 2026 that chatbots, Gemini included, can infringe publisher rights in some circumstances, and five French publisher groups lodged a further complaint with the European Commission on October 29, 2026; a U.S. class action over Gemini’s default activation in Gmail, Chat, and Meet remains pending, with Google’s motion to dismiss the second amended complaint filed September 29, 2026. None of this changes the MRCR v2 finding above, which still accurately describes the Gemini 3.x lineage — it means that lineage is no longer Google’s frontier Gemini model.

Two structural notes on the lineage as of September 2026. First, cadence: Google shipped three Flash-tier releases in six weeks — 3.6 Flash on July 21, 3.7 Flash on August 13, and 3.8 Flash plus a restricted 3.8 Flash Cyber variant on September 2 — which means any benchmark row on this page is describing a model that may be two revisions old. Gemini 3.5 Pro had not shipped as of this update. Second, and easy to miss: the entire Flash line caps at a 1M-token context window, not the 10M advertised on Gemini 3 Pro, so the headline number at the top of this page describes one tier of the family, not the family. Google’s own model cards add a limitation that no retrieval benchmark captures — the 3.6 Flash card states a March 2026 knowledge cutoff but warns that in some domains “the model’s knowledge is limited to January 2025,” a caveat repeated verbatim on the 3.8 Flash card. A cutoff date that holds for some domains and not others is not a cutoff date; it is a range, and the vendor is the one saying so.

Practical guidance

For document comprehension and moderate multi-document synthesis under roughly 128K tokens, Gemini 3.1 Pro is competitive with anything else on the market — its 84.9% MRCR v2 score at that length ties Claude Sonnet 4.6. For ingesting an entire large codebase or a genuinely massive document set where the working set will exceed a million tokens, the published retrieval numbers say to test carefully at your actual working length before trusting the output, and to prefer Claude Opus 4.6 if multi-fact retrieval accuracy at that scale is what matters most. For cost-sensitive bulk processing where retrieval precision is less critical, Gemini 2.0 Flash’s pricing remains a separate, genuine advantage untouched by any of the numbers above.

FAQ

Does Gemini 3 Pro’s 10-million-token context window actually work at that length?
No independently verified multi-needle retrieval benchmark has tested Gemini at the full 10M-token mark as of this writing; published third-party results (contextarena.ai, via Awesome Agents and yage.ai) stop at 1M, where accuracy is already down to roughly a quarter on the hardest test. Treat the 10M figure as an architectural ceiling, not a performance guarantee.

Is Gemini’s hallucination rate better or worse than GPT and Claude?
It depends on the benchmark. On Vectara’s February 2026 long-document test, Gemini 3 Pro (13.6%) is worse than Claude Opus 4.6 (12.2%) but the entire frontier-model class exceeds 10%. On Google’s own FACTS Benchmark Suite, Gemini 3 Pro leads at 68.8%. Neither number should be quoted without naming the benchmark.

Has Gemini’s long-context performance improved over time?
Sharply, and as of the September 2026 Flash releases, at the 1M mark too. From 18.4% (Gemini 2.0 Flash) to 84.9% (Gemini 3.1 Pro) on MRCR v2 8-needle at ≤128K tokens. At the 1M-token mark, the improvement stayed small for four generations — 16.4% (2.5 Pro) to roughly 26% (3 Pro, 3.1 Pro, 3.5 Flash) — before Gemini 3.7 Flash (August 2026) broke the pattern with 62.5% at 1M, more than double the prior ceiling. Whether that gain carries into a future Gemini 3.x Pro release, rather than staying a Flash-tier result, is not yet established.

Is Gemini 3 Pro still Google’s frontier Gemini model?
No. Google released Gemini 4 Argon on October 1, 2026, which replaces the Gemini 3.x lineage as Google’s top-tier model; Gemini 4 Carbon was announced two weeks later but remains in restricted testing. Argon’s only published long-context figure (84.2% on GraphWalks at 256K–1M, per Google’s own model card) uses a different benchmark than the MRCR v2 numbers in the table above, and no independent tracker has yet published an MRCR v2 score for it, so it cannot be slotted into that table yet.

Last updated November 6, 2026. This page is refreshed as benchmarks and scores move.