Static RAG Misses Agentic Utility

A HotpotQA/ReAct counterfactual study finds that static RAG utility barely tracks which documents causally help a multi-step search agent.

TL;DR — In a ReAct-over-HotpotQA counterfactual replay, Static RAG Utility and Counterfactual Trajectory Utility are nearly independent across 23,322 document observations. Roughly a third of read documents are bridge documents: they look useless to a static reader but causally help the agent, often by supplying discriminative entities for the next query.

Problem: static usefulness misses agent behavior

The abstract describes a gap between static retrieval evaluation and multi-turn agent behavior. In the static setup, usefulness is measured by handing a document and a question to a reader model and checking whether the answer improves. Source anchor: "Retrieval systems are trained and evaluated on a static idea of usefulness: hand a document and a question to a reader model, see whether the answer improves, and score the document accordingly."

That idea can hold when a document is read alone, but the abstract says it breaks for a language-model search agent that issues multiple queries and reasons across turns. The reason is causal rather than merely semantic: a document can matter because it changes what the agent does next. Source anchor: "a document can matter for what it lets the agent do next rather than for what it says about the current question."

Full analysis, extracted claims, numerical results, entity graph, FAQ, related work, applications, and BibTeX are available via x402 micropayment.