LLM Query Expansion Made Our Code Search Worse. Here's What Didn't

We added LLM query expansion to our code search pipeline and watched retrieval quality drop by 32%. Then we tried a bigger model. It dropped by 28.6%. We fine-tuned one specifically for the task. Still down 26.4%. The full four-component pipeline — router, expander, retriever, reranker — degraded MRR@10 by 40.6% and pushed P95 latency to over 20 seconds.

The only thing that helped was a regex classifier that runs in under 1ms.

This is what we found when we ran an 860-query ablation study on Mnemex, our local code search tool for AI coding agents. The results contradicted almost everything the standard RAG literature says about LLM query expansion. Here's the data, the explanation, and what we'd build instead.

What is RAG query expansion and why we expected it to work

The standard assumption in retrieval-augmented generation (RAG) — a technique that augments AI responses by retrieving relevant documents before generating an answer — is that more pipeline stages improve quality. Expanders increase recall on underspecified queries by rewriting them into richer, semantically broader versions. Rerankers improve precision by applying a more capable relevance model after the initial retrieval pass.

Mnemex is MadAppGang's local code search tool built for AI coding agents. It runs on AST-aware analysis, LLM-enriched symbol metadata, a LanceDB vector store, and SQLite BM25 — giving it both semantic and keyword retrieval paths. We had four pipeline components available: a query router, an LLM expander, the hybrid retriever, and a reranker.

The hypothesis was straightforward. Code search queries are often underspecified — a developer asking an agent to "implement rate limiting" doesn't know the exact function name in the codebase. LLM expansion should help bridge that vocabulary gap. We expected it would.

The four-component Mnemex retrieval pipeline — Router improved quality; Expander, Retriever under LLM expansion, and Reranker all degraded it

How we tested retrieval pipeline components across 860 queries and 12 repositories

We ran 860 queries across 12 open-source repositories — fastmcp, tinygrad, smolagents, openai-agents, and eight others — covering a range of AI and systems codebases. Each query was generated using a PageRank-weighted sampling method to ensure coverage across the codebase, not just the most-visited files.

Eight pipeline configurations were tested, from a baseline retriever-only setup through to the full four-component RAG pipeline. We tested three expander models: LFM2-700M, a fine-tuned Qwen3-1.7B, and LFM2-2.6B. The reranker was also Qwen3-1.7B. We measured statistical significance using the Wilcoxon signed-rank test — a paired, non-parametric test suited to bounded score distributions like MRR@10 — with the threshold set at p < 0.05 and effect size |r| > 0.1.

One limitation worth naming upfront: the LLM conditions (expanders and reranker) ran on only two repositories due to LM Studio batch processing overhead. The cross-repository averages for those conditions are directional findings, not a fully representative sample. The router results, by contrast, cover the full 12-repository dataset and are statistically validated.

MRR@10 relative change from baseline across eight pipeline configurations. All LLM-based components degraded retrieval quality; the regex router was the only improvement

The results: every LLM query expansion model hurt search quality

The regex router (configuration B1) improved MRR@10 — Mean Reciprocal Rank at 10, a standard retrieval quality measure — by 21.8% over the baseline. It runs in under 1ms and adds zero infrastructure cost.

Every LLM component went the other direction. The LFM2-700M expander reduced MRR@10 by 32.0%. LFM2-2.6B dropped it by 28.6%. The fine-tuned Qwen3-1.7B, specifically trained for this task, was the best of the three expanders. It still reduced MRR@10 by 26.4%. The reranker alone cut MRR@10 by 32%. The full four-component pipeline was the worst result of all: -40.6% MRR@10 and a P95 latency of 20,656ms — roughly eight times the baseline.

These results are statistically significant. The full pipeline degradation reached p = 0.0004 with an effect size of r = 0.68. The router-plus-expander combination hit p < 0.0001 with r = 0.81. This isn't noise.

A note on the absolute numbers: the baseline MRR@10 differs between our March 11 run (0.438) and our March 16 run (0.309), likely due to a partially corrupted vector store after a tool rename. The absolute figures across runs aren't directly comparable — but the relative rankings are consistent across both. The router helps. Every LLM component hurts.

860 queries across 12 open-source Python repositories

Does LLM query expansion improve RAG retrieval quality?

Not for code search, and possibly not for any query distribution that skews toward exact-match lookups. An 860-query study across 12 open-source repositories found that every LLM-based expander reduced MRR@10 by 26–32% relative to an unaugmented hybrid retrieval baseline. A 1ms regex classifier routing symbol queries to BM25-only search was the only component that consistently improved quality, adding +21.8% MRR@10. The root cause: LLM expanders are tuned for natural-language queries, and when the majority of queries are identifier lookups, expansion destroys the precise keyword signal BM25 already handles well.

The router improved MRR@10 in 8 of 9 repositories tested. The largest single gain was smolagents at +0.342 (+66% relative). The only regression was openai-agents at -0.081 (-15%). We're still investigating why that one went the other way.

8/9 repos improved

How an LLM expander transforms a symbol-name query — the rewritten output has high semantic relevance but low lexical overlap with the actual class definition

Why query expansion fails for code search

The core issue is query distribution mismatch. The majority of code search queries in AI agent sessions are symbol-name lookups: CamelCase class names, snake_case function names, backtick-quoted identifiers. Things like FastMCP, ConnectionPool, parse_requirements.

BM25 handles these directly. The symbol name appears verbatim in the source file, in the function signature, and in Mnemex's LLM-generated description metadata. A keyword search finds it cleanly.

LLM expanders weren't built for this. They were built for underspecified natural-language queries where the vocabulary gap between what the user says and what's in the document is real. Feed an expander FastMCP and it rewrites the query as something like "server implementation for the Model Context Protocol, handling tool registration and request routing." That paraphrase has high semantic relevance to MCP documentation and related comments across the codebase. It has low lexical overlap with the actual FastMCP class definition. Vector search retrieves plausible-sounding results. The right one often isn't there.

Why BM25 beats vector search for symbol-name queries

Symbol names are exact-match queries by nature. The identifier ConnectionPool appears in the source file, in the function signature, in import statements, and in Mnemex's generated metadata description — all verbatim. BM25's term frequency weighting finds it directly, with high precision.

Vector similarity works differently — it retrieves documents semantically close to the query regardless of exact token overlap. For a symbol lookup that means connection pooling concepts, related utilities, and similar patterns elsewhere in the codebase. The right class rarely appears.

Hybrid retrieval — the combination of vector and BM25 — dilutes the BM25 signal rather than amplifying it. You're blending a high-precision exact-match result with a set of semantically related but lexically wrong candidates. The blend produces worse precision than BM25 alone on this query type. The router's fix is simple: detect the symbol lookup pattern and skip vector search entirely.

BM25-only search returns the exact class definition as the top result for a symbol-name query. Hybrid retrieval dilutes this signal with semantically related but lexically incorrect candidates

How Cursor, Cody, and Aider handle code search retrieval

None of them use real-time LLM query expansion. Not Cursor. Not Sourcegraph Cody. Not Aider.

All three rely on deterministic routing and keyword or hybrid retrieval as the primary mechanism. They don't run a language model on the query before retrieval. They classify the query type and route it to the appropriate retrieval strategy.

Our study didn't inspire that choice — they made it independently, presumably after running into the same wall we did. What our data does is give that choice empirical grounding. The reason production code search tools don't use LLM query expansion isn't because nobody thought of it. It's because it doesn't work for the query distribution that actually shows up in agent sessions. At MadAppGang, we built Mnemex from the ground up for AI agent workflows and ran this study specifically to understand why the tools we were building weren't performing as expected — and the answer matched what the production tools had already figured out.

The right RAG pipeline for code search: no LLM at query time

The optimal pipeline is simpler than most RAG literature would suggest:

• incoming query → regex classifier (under 1ms)
• symbol_lookup → BM25-only keyword search
• semantic or exploratory → hybrid (vector + BM25) retrieval

No expander. No reranker. No LLM at query time.

Full pipeline is 8x slower than baseline

That said, this isn't a verdict against LLM components everywhere. Expansion may still help for semantic and exploratory queries, where the vocabulary gap between natural-language intent and source code is genuine. Our mixed-query dataset showed the router's advantage compresses when these query types make up a larger share of the workload. The answer isn't to remove vector search from the hybrid retrieval path — it's to route to it selectively.

The practical starting point: add the regex router first, measure the improvement on your actual query distribution, and only reach for LLM components if you've confirmed the query types they're designed for are actually present in your workload. Don't add pipeline stages to solve problems your queries don't have.

The optimal Mnemex pipeline — a regex classifier routes symbol lookups to BM25-only search and all other query types to hybrid retrieval. No LLM at query time

Frequently Asked Questions

What is query expansion in RAG and when does it help?

Query expansion rewrites an incoming query into a richer or broader version before retrieval, with the goal of increasing recall on underspecified searches. It's most useful when there's a genuine vocabulary gap between how a user phrases a query and how the relevant content is written. For natural-language search over documents, it can help. For code search where most queries are exact symbol names, it consistently hurts.

Why does BM25 outperform vector search for code symbol lookups?

Symbol names appear verbatim in source files, function signatures, and metadata. BM25's term frequency weighting finds exact matches directly with high precision. Vector similarity retrieves semantically related content regardless of exact token overlap — for identifier lookups, this produces plausible but incorrect results. Hybrid retrieval blends the two signals, which dilutes precision rather than improving it.

Does model size affect how much query expansion hurts retrieval quality?

In our study, larger models degraded quality slightly less — but all three expander models still reduced MRR@10 significantly. The fine-tuned Qwen3-1.7B was the best performer at -26.4%, versus -32.0% for LFM2-700M and -28.6% for LFM2-2.6B. Model size doesn't fix a query distribution mismatch problem.

What types of code search queries still benefit from semantic retrieval?

Semantic and exploratory queries — things like "how does authentication work" or "implement rate limiting" — have a real vocabulary gap between intent and source code where vector search adds value. The optimal approach is to route these query types to hybrid retrieval while routing symbol lookups to BM25-only search.

How is Mnemex different from tools like Cursor or Cody?

Mnemex is a local code search tool built specifically for AI coding agents, not for interactive developer search. It runs on AST-aware analysis, LLM-enriched symbol metadata, and a dual-retrieval backend combining LanceDB vector storage with SQLite BM25. It runs on-device on Apple Silicon with no cloud dependency, which is also why latency characteristics matter more for Mnemex than for server-side tools — an agent waiting on a search call blocks its whole task loop.

The conclusion the data forces

Every LLM component we added to the pipeline made retrieval worse. The only improvement came from a regex classifier that adds no meaningful overhead and routes symbol queries away from the vector search path that was degrading them.

If you're designing a retrieval pipeline for code search, start with the query distribution you actually have. If it skews toward identifier lookups — and in AI agent sessions, it almost certainly does — build the router first and measure everything else against that baseline. Don't add an expander because the literature says it should help. Talk to an engineer who has run the numbers on a real codebase.