Query Routing in RAG: The Fix That Recovered 4x Retrieval Quality

The ablation study didn't go the way I expected. [INTERNAL LINK: our ablation study on LLM query expansion] We tested eight pipeline configurations on 860 queries across 12 open-source repositories, and every LLM-based component made retrieval worse. Expanders reduced MRR@10 by 26–32%. The reranker cut it by 32%. The full four-component pipeline degraded quality by 41% while running 8x slower than the baseline.

That result pointed straight at a follow-up question: is the damage uniform, or is it concentrated in specific query types?

It was concentrated. Almost entirely in one type. And once we understood that, fixing the query routing in our RAG pipeline took about 15 lines of regex.

Why the same retrieval strategy fails different query types

Query distributions in AI agent code search sessions are not what most retrieval benchmarks assume.

In AI agent code search sessions — Claude Code, Cursor, Aider navigating a real codebase — roughly 60–80% of queries are symbol-name lookups. Not "how does authentication work." Not "implement rate limiting." Just: FastMCP. ConnectionPool. parse_requirements. An agent encounters a class name in one file and searches for its definition.

This matters because BM25 (a keyword-matching retrieval algorithm that scores documents based on term frequency and inverse document frequency) is extremely good at these queries. The symbol name appears verbatim in the source file, in the function signature, and in the enriched description Mnemex generates during indexing. Exact keyword match finds it immediately.

LLM expanders were not designed for this case. They were designed for the opposite case: natural-language queries where the vocabulary gap between intent and source code is the problem. "Authentication middleware for FastAPI" needs expansion. "FastMCP" does not.

What the expander does with a symbol name is the root of the damage. For FastMCP, the expander rewrites the query into something like "server implementation for the Model Context Protocol, handling tool registration and request routing." That paraphrase has high semantic relevance to MCP documentation and comments in the codebase — but low lexical overlap with the class definition itself. The expander destroys the precise BM25 signal that would have found it directly, and then the error compounds.

The hybrid retriever scores candidates against the expanded description, surfacing documentation and comments instead of the class definition. The reranker re-evaluates those already-degraded candidates and confirms they match the expanded query. Each stage amplifies the mistake made by the one before it. The result is a -41% MRR@10 (Mean Reciprocal Rank at 10 results — a standard retrieval quality metric) at 8x the baseline latency. More pipeline, worse results.

The fix is not to remove the expander entirely. The fix is to not call it when the query is a symbol lookup.

Two queries, two outcomes: a symbol lookup routed directly to BM25 finds the class definition in milliseconds; the same query processed through blind LLM expansion returns documentation instead.

How to classify code search queries with regex

Mnemex uses a four-label query classifier that runs before anything else in the pipeline. The labels are symbol_lookup, structural, semantic_search, and exploratory. Classification determines which retrieval path the query takes.

The full ruleset from Appendix B of our ablation study, executed in priority order:

// Rule 1: Symbol lookup — backtick-quoted, CamelCase, or snake_case identifier
const SYMBOL_PATTERNS = [
  /`[^`]+`/,                         // Backtick-quoted: `FastMCP`, `parse_args`
  /\b[A-Z][a-zA-Z0-9]{2,}\b/,       // CamelCase: FastMCP, ConnectionPool
  /\b[a-z][a-z0-9_]{2,}\b(?=\s*$)/, // snake_case standalone: parse_requirements
];

// Rule 2: Structural — relationship queries
const STRUCTURAL_PATTERNS = [
  /callers?\s+of\b/i,
  /where\s+is\s+.+\s+called/i,
  /\bimports?\s+/i,
  /depends?\s+on\b/i,
];

// Rule 3: Semantic — error and behavior descriptions
const SEMANTIC_PATTERNS = [
  /\braises?\b/i,
  /\breturns?\s+wrong\b/i,
  /\bdoes(n'?t|not)\s+work\b/i,
  /\bbug\b/i,
  /\berror\b/i,
  /\bfail(s|ing|ed)?\b/i,
];

// Rule 4: Exploratory — intent-based queries
const EXPLORATORY_PATTERNS = [
  /\bhow\s+(to|does|do)\b/i,
  /\bimplement(s|ation)?\b/i,
  /\badd\s+support\b/i,
  /\bwhat\s+(is|does|are)\b/i,
];

// Default: semantic_search

Classification takes under 1ms. No ML model, no LLM call, no added latency. When the router labels a query as symbol_lookup, the retriever bypasses vector search entirely and uses keyword-only BM25 search. Every other label goes to hybrid retrieval — vector plus BM25 — unchanged.

The routing decision is deliberately minimal. BM25 handles exact-match lookups without help. Hybrid already handles semantic and exploratory intent well. The only intervention needed is preventing the expander from running on queries it will damage.

The Mnemex query classifier evaluates each query against four rule sets in priority order. The first matching rule determines the retrieval path. Unmatched queries default to semantic search and take the hybrid path.

How do you implement query routing in a RAG pipeline?

To implement query routing in a RAG pipeline for code search:

1. Classify each incoming query using deterministic regex rules that identify symbol lookups (CamelCase identifiers, snake_case tokens, backtick-quoted names), structural queries, and semantic or exploratory queries.
2. Route symbol lookups to BM25-only keyword search, bypassing vector retrieval.
3. Route all other query types to hybrid retrieval (vector + BM25 with reciprocal rank fusion).
4. Optionally add a reranker for non-symbol queries where your latency budget permits. The classifier runs in under 1ms and requires no ML model or LLM call at query time.

Query routing RAG results: 4x quality recovery at a fraction of the latency

In MadAppGang's ablation study, I ran route-aware variants alongside the router-only and blind-expansion conditions on fastmcp (n=30 symbol queries, March 16 run):

E-RA (0.477) versus blind E (0.118) is a 4x improvement from a single change: skipping the expander when the router identifies a symbol lookup. F-RA versus blind F shows the same pattern at 3.6x. Notably, the regex router alone (B1, 0.442) outperforms F-RA while running at 1.1s P95 — adding a route-aware expander on top of the router provides marginal quality gain at the cost of nearly doubling latency.

These results are directional, not statistically validated at scale. I ran the route-aware variants on a single repository with n=30 symbol queries. The evidence is consistent and the margin is large, but I would not call it proven across all codebases and query distributions. What I can say is that the direction held across every condition we tested, and the mechanism is clear enough that the result is not surprising.

One note on the baseline: the March 16 run shows A = 0.309, lower than the March 11 baseline of 0.438. The likely cause is a partially corrupted vector store after a tool rename during that period — the claudemem → mnemex rename left the vector store with partially corrupted metadata. Relative rankings across conditions are consistent between runs. The absolute numbers are not directly comparable.

For the cross-repository router-only results, the picture is cleaner: the router improved MRR@10 in 8 of 9 repositories across 860 queries, with an aggregate +21.8% improvement over baseline. The largest gain was smolagents (+66% relative, from 0.521 to 0.863). The only regression was openai-agents (-15%, from 0.549 to 0.468).

The variation across repositories reflects differences in codebase structure and query distribution. smolagents' unusually large gain (+0.342) suggests its symbol graph is particularly well-suited to keyword-only retrieval; openai-agents' regression (-0.081) is the one case where the routing heuristic appears to misclassify enough queries to hurt.

MRR@10 by pipeline configuration — route-aware variants and the regex router alone all outperform their blind-expansion counterparts by a wide margin.

Here are the full results with details:

Reranker or no reranker: choosing between E-RA and F-RA

The choice between E-RA and F-RA is a latency tradeoff.

E-RA hits 0.477 MRR@10 — the highest quality result in the study. But its P95 tail latency is 35.4 seconds. The reranker scores candidates sequentially, and latency scales with the number and length of retrieved candidates. Under load, with long files returning many candidates, the tail is unpredictable.

F-RA achieves 0.427 MRR@10 at 1.9s P95. The regex router alone (B1) achieves 0.442 at 1.1s — marginally better quality than F-RA at lower latency, with no expander at all. The latency cost of each component is unambiguous:

The reranker's wide range (+3,000–10,000ms) reflects sequential candidate scoring — short files resolve faster, long files with many candidates drive the tail. Full pipeline P95 of +18,000ms means one in twenty searches adds 18 seconds on top of baseline. For interactive AI agent sessions, that is not a latency budget — it is a broken experience.

For interactive use, the decision tree is simple: start with the regex router alone (B1). If you need marginally higher quality and can absorb ~1.9s P95, add route-aware expansion (F-RA). Do not add the reranker unless you are running offline batch retrieval where the latency tail is acceptable and the 0.05 MRR gain over F-RA justifies it.

Quality versus latency across six conditions. The route-aware and router-only variants cluster in the top-left quadrant — high MRR, low latency. Blind expansion variants collapse to the bottom-right.

Where query expansion still adds value

The finding from this study is not "never use LLM expansion." It is "do not use it blindly across all query types."

The mixed-query dataset for fastmcp — 10 symbol queries, 10 semantic queries, 10 exploratory queries — shows the router's advantage compresses when the query set is more evenly distributed. When symbol lookups no longer dominate, the harm from blind expansion is less concentrated, and the gap between route-aware and baseline narrows.

Expansion may still provide value for semantic and exploratory queries, where the vocabulary gap between natural-language intent and source code is genuine. "How does authentication work" is exactly the case the expander was designed for. A targeted evaluation of expansion on semantic-only queries has not been done — that is an acknowledged gap in the research. Before enabling expansion on semantic queries, I would want to see that evaluation on your specific codebase and query distribution.

The practical starting point: disable expansion for symbol lookups first. That change is low-risk and the evidence for it is strong. Then evaluate separately on your semantic query distribution before enabling expansion there. Do not add components until you have tested whether they help on your actual traffic.

The production RAG pipeline architecture for code search

Based on this study, the optimal Mnemex pipeline is:

query → regex classifier → {
  symbol_lookup  → keyword-only BM25 search
  semantic / exploratory → hybrid (vector + BM25) search
}

The optimal Mnemex pipeline. One classifier, two paths, no LLM at query time.

No expander. No reranker. No LLM at query time.

This is also what production code search tools already do. Sourcegraph Cody, Cursor, and Aider all use deterministic routing and rely on keyword or hybrid retrieval as the primary mechanism. They did not arrive at this through compromise — it is the correct architecture for a query distribution dominated by symbol lookups. Our study provides the empirical grounding for that choice.

The principle generalises beyond code search. Any retrieval system with meaningfully distinct query type distributions — identifier lookups versus natural-language intent, for example — should evaluate whether routing before LLM components improves quality. Adding pipeline stages only helps when those stages are matched to the queries they will actually receive.

At MadAppGang, we apply this routing architecture in the production AI systems we build for clients — classifying query intent before any LLM component runs, and routing to the retrieval strategy that matches it.

We build retrieval pipelines that perform in production, not just in benchmarks. If you are designing a search or RAG system and want honest tradeoff analysis before you commit to an architecture, talk to our team.

Frequently asked questions

What is the difference between query routing and query expansion in RAG?

Query routing classifies the incoming query and directs it to a different retrieval strategy based on its type — for example, sending symbol lookups to BM25-only search and semantic queries to hybrid retrieval. Query expansion rewrites the query into multiple variants before retrieval to improve recall. They address different problems: routing handles query type mismatch, expansion handles vocabulary gap. In a well-designed pipeline, routing runs first and can prevent expansion from running on queries it will damage.

How do you identify a symbol-name query automatically?

Symbol-name queries follow predictable lexical patterns: CamelCase identifiers (FastMCP, ConnectionPool), snake_case tokens used standalone (parse_requirements), or backtick-quoted names. A regex classifier matching these patterns catches the vast majority of symbol lookups in under 1ms, with no ML model or LLM required. Mnemex's full ruleset is in Appendix B of our ablation study and is reproduced in full above.

When should I use hybrid retrieval vs. BM25-only for code search?

Use BM25-only for symbol-name queries — exact identifier lookups where the symbol name appears verbatim in the source. Use hybrid retrieval (vector + BM25 with reciprocal rank fusion) for semantic and exploratory queries where the vocabulary gap between the user's intent and the source code is the problem. Hybrid adds latency for symbol queries without quality benefit; BM25-only fails on underspecified natural-language queries. A query classifier routes each type to the right strategy.

Does route-aware query expansion work for non-code retrieval systems?

The core principle applies to any retrieval system with a bimodal query distribution — some queries are exact lookups, others are underspecified natural-language intent. If your system receives a significant proportion of exact-match queries, blind LLM expansion will damage quality on those queries in the same way it does in code search. Classify first, expand only on queries where expansion helps. The specific regex rules are code-specific, but the routing logic is not.

What is reciprocal rank fusion and how does it interact with query routing?

Reciprocal rank fusion (RRF) combines results from multiple retrieval signals — in this case, BM25 and vector search — by ranking each candidate based on its position in each signal's result list and summing the reciprocal ranks. It is used in the hybrid retrieval path for semantic and exploratory queries. For symbol lookups, the router bypasses vector search entirely, so RRF does not apply — there is only one signal (BM25) and no fusion needed. RRF adds value precisely where multiple signals are complementary; for exact-match queries, a single strong signal is better than fused weaker ones.