Why did GraphRAG lose to simple keyword search?
Graph retrieval missed more evidence than keyword search. I traced irrelevant anchors, narrowed their selection, and recovered baseline recall while preserving relationship paths.
The initial problem
The first graph retrieval policy underperformed simple keyword search in a small benchmark. Required-evidence recall—the share of required evidence retrieved—fell from 82% with the lexical baseline to 45% with the graph policy.
The graph contained useful relationships between sources, but that alone did not make its output useful. I needed to understand why following those relationships was making the input worse.
How I approached it
I traced the irrelevant context back to anchor selection. An anchor is a source selected as a starting point for exploring the graph. The policy filled up to five anchors whenever they cleared an absolute score threshold.
Weaker anchors near the end of that list led through relationship edges to operating tables that were irrelevant to design and interface requests. The issue was how much of the graph the policy pulled into each request.
I used keyword search as the benchmark baseline. To judge the graph's value, I would keep measuring required-evidence recall, irrelevant context injection, relationship-path availability, and usable output separately. Finding the same evidence with an explanation of its relationships is a different benefit from finding more evidence.
How I solved it
I narrowed selection with a relative threshold: an anchor needed at least 85% of the strongest anchor's score. In these cases, that reduced the selection to one or two anchors.
Four of five observed irrelevant-injection cases disappeared, and required-evidence recall recovered to 82%. That matched the lexical baseline; it did not exceed it. The graph's useful addition was relationship paths, which show why a source is connected to the request. The proportion of benchmark cases without paths fell from 27% to zero.
The resulting decision was to adopt the safer anchor policy, preserve useful relationship context, and continue testing against the simpler retrieval method.
What the evidence supports
The run contained 11 test cases, each run once with each retrieval policy. Judge coverage differed between retrieval policies, so their average scores were not directly comparable. The behavior for abstaining when no anchor qualifies did not trigger in this sample, so the run did not verify it.
A measurement issue invalidated an earlier token-saving claim. I did not carry it forward as a result.
This benchmark supported a narrower retrieval-policy change. It did not establish broad GraphRAG superiority or a production impact claim.