Needle-in-a-haystack tests are close to saturated and tell you almost nothing about whether a model can actually reason across a long document; benchmarks that force multi-hop retrieval and reasoning — RULER, MRCR, BABILong, Fiction.liveBench — show every current model’s usable context is far shorter than its advertised window.
Context windows have grown from 4K to 1M-plus tokens in a few years, and vendors lean hard on that number in marketing copy. But “the model accepted 1M tokens” and “the model reasoned correctly over 1M tokens” are different claims, and the benchmarks built to separate them tell a consistent story: simple fact retrieval holds up reasonably well; anything that requires combining multiple facts, tracking order, or reasoning across distributed information degrades well before the context limit is reached.
Why the original needle-in-a-haystack test stopped being useful
The classic needle-in-a-haystack (NIAH) eval drops one fact into a long document and asks the model to retrieve it. It became the default long-context demo because it’s cheap to run and easy to visualize as a heatmap. The problem, documented across several 2025 papers, is that single-needle retrieval is the easiest thing a long context can be asked to do, and it has stopped discriminating between models.
A 2025 analysis (Nakrani, “Why Passing Needle-in-a-Haystack Is Not Enough,” 2025) argues the test measures the easiest retrieval case and predicts almost nothing about harder tasks like combining several facts. Follow-on benchmarks were built specifically to close that gap:
- Sequential-NIAH (arXiv, 2025) requires extracting multiple ordered needles rather than one, and reports that current LLMs still have significant room for improvement on the task (arXiv:2504.04713).
- NeedleChain (2025) chains facts into a causal sequence where every link is necessary to reach the answer, with a parallel single-needle variant for direct comparison — explicitly designed to separate retrieval from reasoning.
- Needle Threading (Cambridge/HKU, arXiv:2411.05000) names saturation directly as a limitation of simple-retrieval NIAH variants.
RULER: NVIDIA’s harder synthetic suite
NVIDIA’s RULER benchmark (arXiv:2404.06654, COLM 2024; code on GitHub) extends NIAH with multi-hop tracing, aggregation, and variable tracking tasks, and found that almost all models show large performance drops as context length increases — even when the same models look fine on vanilla NIAH.
Current leaderboard numbers for RULER are a mess of incompatible harnesses. BenchLM’s October 23, 2026 entry has NVIDIA’s own Nemotron 3.5 Bolt E4B at 86.99% at 64K tokens, run in NVIDIA’s NeMo Gym harness (benchlm.ai). A separate tracker lists Nemotron 3 Super-120B-A12B at 96.3 and Nemotron Nano 9B v2 at 78.9, both at 128K with reasoning off, but without a recorded harness, which makes them hard to verify independently. An older reference point from Codesota, dated December 2024, has Qwen2.5-72B-Instruct at 95.1. Because context length, harness, and reasoning settings differ across these entries, they are not directly comparable — which is itself the point: RULER scores only mean something alongside their exact evaluation settings.
MRCR: retrieval from multiple locations, not one
MRCR (Multi-Round Co-reference Resolution) originated in Google DeepMind’s Gemini 1.5 technical report and the Michelangelo paper (Vodrahalli et al., 2024). It drops a long synthetic multi-turn conversation into context and asks the model to reproduce a specific earlier turn — harder than NIAH because the model must locate and distinguish between multiple similar targets, not just one. The Michelangelo paper’s own results showed every tested model dropping off significantly before 32K tokens, with Gemini models performing best in that original study (Stanford HELM Long Context, Sept 29, 2025; VentureBeat coverage).
Public leaderboards disagree sharply on current scores. One tracker shows Gemini 2.5 Pro leading MRCR at 0.930 across seven evaluated models (llm-stats.com). On the newer MRCR v2 variant, a different tracker lists Gemini 3.6 Flash (self-reported) at 54.0%, while benchmarklist.com ranks Sakana AI’s Fugu-Ultra first at 93.6% with Gemini 3.5 Flash sixth at 77.3%. The variants (2-needle vs. 8-needle, 128K vs. 1M context) are not interchangeable, and several of these figures are self-reported rather than independently run — treat any single MRCR leaderboard number with real skepticism until it’s checked against a model card or the original paper.
BABILong: reasoning over facts scattered through book-length text
BABILong (Kuratov et al., NeurIPS 2024, arXiv:2406.10149) embeds classic bAbI reasoning tasks inside long background text drawn from PG19 books, with published splits up to 10 million tokens. Its central finding has not been contradicted by newer work we could locate: popular models effectively use only 10–20% of their claimed context window, with accuracy declining sharply as both length and reasoning complexity increase. Retrieval-augmented generation tops out around 60% accuracy on single-fact QA in this setup, regardless of context length — a ceiling that has nothing to do with window size and everything to do with retrieval precision. We found no 2025–2026 follow-up leaderboard for BABILong, so current frontier-model scores on it are not publicly tracked the way RULER’s are.
Fiction.liveBench: comprehension over window size
Fiction.liveBench tests deep comprehension of long fiction rather than fact retrieval, and its own trend data is the clearest illustration of the gap between stated and usable context: accuracy starts sliding well before the advertised maximum window, typically around 32K tokens, with only a handful of models (Grok 4 and Gemini have been cited as standouts) holding up past 192K (PromptHub analysis). Reported top scores vary by source and by the context length used for comparison: llmrun.dev’s tracker has O3 Pro at 97.2% and Kimi K2.5 as the top open-weight model at 86.1% (data through January 2026); DataLearner’s August 2026 snapshot independently lists GPT-5 (medium) and o3-pro (medium) tied at 97.2% at 16K tokens specifically — the length at which DataLearner treats the metric as canonical. A separate October 2026 board instead shows o3 at 100%, without specifying context length, which is not directly comparable to the 97.2% figures. The Decoder’s earlier reporting on o3’s launch cites a confirmed 100% at 128K tokens, with Gemini 2.5 Pro as the closest competitor at 90.6%. We found no 2026 entries for Claude or Gemini 3.x on Fiction.liveBench, so current frontier coverage on this particular benchmark is incomplete.
What the trackers agree on, and where they diverge
| Benchmark | What it measures | Known limitation / disagreement |
|---|---|---|
| NIAH (classic) | Single-fact retrieval from a long context | Largely saturated; doesn’t test reasoning or multi-fact combination |
| RULER (NVIDIA) | Multi-hop tracing, aggregation, variable tracking | Leaderboard entries use incompatible context lengths and harnesses; scores aren’t directly comparable across trackers |
| MRCR / MRCR v2 | Retrieving one of several similar targets from multi-turn dialogue | Public rankings conflict sharply (e.g., Fugu-Ultra 93.6% vs. Gemini 3.6 Flash 54.0% on different trackers); several scores are self-reported |
| BABILong | Reasoning over facts distributed through book-length text (up to 10M tokens) | Models use only ~10–20% of claimed context effectively; little current (2025–2026) leaderboard tracking found |
| Fiction.liveBench | Deep comprehension of long fiction, not fact lookup | Comparisons mix 16K, 128K, and unspecified-length scores across sources; figures disagree by up to several points for the same model |
The pattern across all five is the same: wherever a benchmark forces the model to combine, order, or reason over more than one piece of distributed information, scores fall off well before the window closes, and the exact number you see depends heavily on who ran the eval, at what length, and with which harness. This is exactly the measurement-methodology problem covered in general terms in How to Evaluate LLMs, and it’s the same failure mode documented specifically for Gemini’s long-context behavior in Gemini Limitations in 2026. It is also a close cousin of the contamination and saturation problems covered in MMLU and MMLU-Pro and in Humanity’s Last Exam: a benchmark that’s easy enough to saturate stops being a useful measurement instrument, and the field keeps having to build a harder successor.
FAQ
Is a 1-million-token context window the same as being able to reason over 1 million tokens?
No. BABILong’s core finding is that models use roughly 10–20% of their claimed window effectively, and Fiction.liveBench shows comprehension accuracy sliding well before the advertised maximum on most models tested.
Which long-context benchmark should I trust for a single comparable score?
None of them in isolation. RULER, MRCR, and Fiction.liveBench all have leaderboard entries that disagree depending on harness, context length, and whether a score is self-reported, so the right approach is to compare models within the same tracker and same settings, not across trackers.
Why do newer benchmarks like NeedleChain and Sequential-NIAH exist if NIAH already works?
Because classic single-needle NIAH is largely saturated and doesn’t test multi-fact combination or ordered reasoning; both newer benchmarks were built specifically to probe that gap and report that current models still struggle with it.
Last updated November 13, 2026. This page is refreshed as benchmarks and scores move.