On LakeQA, our benchmark for question answering over massive data lakes, Opus 4.5 gets 46% of the questions correct and GPT 5.2 gets 18%. But these numbers tell you almost nothing about why they failed. In this post, we talk about how scale affects difficulty, and how careful ablation can pinpoint the bottlenecks where question answering needs to improve.
Question-answering (QA) agents are agents designed to answer questions based on retrieved knowledge. They answer questions by planning subquestions, where each subquestion may require the agent to search a knowledge base for relevant sources and analyze those sources to extract facts. As an example, we start out with a simple question:
“Which supplier caused the most delivery delays in our Midwest locations last quarter?”
Complexity appears as we scale
- find out which states count as Midwest, run SQL to resolve them to Indiana and Illinois
- use those states to find which suppliers had the most delays by generating more analysis queries
From the carousel above, we can observe that in general, tasks are more difficult as we scale the data lake, increase the steps required to answer the question, and increase the sources required. In LakeQA, our ICML 2026 work, we scale this to the extreme by building a 10TB data lake of 40M files, where each task in the benchmark has about 13 reasoning steps and 7 required sources.
In LakeQA, even the best models fall well short, but we cannot isolate the root causes or see where we need more progress. Scale degrades the quality of planning, search, and analysis, and a failure at any step can cascade. As a result, an improvement to the retrieval system may look bad even if it retrieved the right sources, because the agent failed at planning by missing an important step. As such, we need an evaluation framework to control each bottleneck independently.
SANA: an evaluation framework for question-answering agents
SANA is an evaluation framework that isolates the contribution of each component by keeping the other components at oracle. Specifically, for planning, it gives you the right subquestions to answer the question. Then, within each subquestion, search always retrieves the required sources relevant to the query, and analysis always extracts the correct fact from those sources.
Intent-based ablation
SANA does intent-based ablation. An agent’s intent does not necessarily result in the right answer, as the implementation could be a bottleneck. For instance, even if the agent intended the right query for a source, the relevant sources might not appear from a search tool, as they get pushed out of the top-k by the scale of the lake.
However, if we restrict the data lake to the gold datasets required to answer the question, then the retrieved sources will always be relevant; hence if the agent had the right intent in its query, it will always get the right results.
We can use SANA to answer 2 questions: (i) how big a component’s bottleneck is: by taking the oracle implementation as an upper bound, and making a naive implementation like BM25 for search, the difference in accuracy acts as a proxy for how big a component’s bottleneck is. (ii) how good a component’s implementation is: you can also judge how good your implementation of a component is by calculating how far it is from the oracle and how much better than naive it is.
Experiments & results
Now that we have a framework, we can carefully control the mixture of components to run ablations. The findings we talk about in this post are on a 10k-document subset of LakeQA, mainly evaluated on accuracy with gpt-5-mini. With that in place, we ran 2 modes of ablation:
- Per-component ablation ablates one component while keeping the other two at oracle. For instance on search: keeping plan and analyze at oracle, we change the search modes around, from the naive BM25, to the standard hybrid search, to oracle. Here, standard means an off-the-shelf implementation, standing in for whatever your implementation is.
- End-to-end mode ablation ablates all three axes as one mode: for instance, naive all the way, or standard all the way.
Finding 1: Data analysis and search are major bottlenecks
From per-component ablation, we found that search and analysis are big bottlenecks. The difference between naive and oracle is 9.6% for planning, 13.3% for search, and 18.5% for analysis. Planning looks like a smaller bottleneck than it is: agents deviate from 44% of their own plans, so a better planner often doesn’t get executed.
Finding 2: We are still far away from oracle
From end-to-end mode ablation, we found that we are still far away from oracle. We can see that standard is only slightly better than naive (1.5%), and there is still a huge gap between standard and oracle (18.5%). So there are still improvements that can be made to the implementations.
Takeaways & try it out
- Careful measurement is important. Isolation is necessary, as end-to-end scoring can conflate the contribution of each improvement.
- Bottlenecks are non-generalizable. They are specific to the tasks, benchmark, scale, and models. For instance, LakeQA is bottlenecked by analysis and search, while KramaBench is only bottlenecked by analysis (the gaps between naive and oracle are 3.6% for planning, 6.0% for search, and 14.5% for analysis).
We open-sourced SANA so you can:
- Identify bottlenecks in any question-answering benchmark.
- Evaluate the effectiveness of your tools for planning, search, and analysis in question answering.
Interested in using SANA, measuring the bottlenecks in your own QA benchmark, or working with us on the next set of problems?