Why pure vector search fails in enterprise RAG — and how RRF fixes it

Click to expand

Dense vector search is great for semantic queries like "How do I reset my password?" But when an enterprise user searches for "ISO-27001-A.9.2" or "SKU-4821-B", pure vector similarity fails. Embeddings map exact string identifiers to semantically adjacent terms, returning documents about general security policies or unrelated product SKUs instead of the exact match.

In production RAG systems across 5,000+ enterprise documents, relying solely on dense vector retrieval produced a Hit Rate @ 5 of less than 68%. Switching to a hybrid retrieval pipeline powered by Reciprocal Rank Fusion (RRF) and cross-encoder reranking boosted retrieval precision by +23%.

Here is why single-signal vector search breaks down in enterprise environments, how RRF mathematical score fusion works, and how to implement it on Azure AI Search.

What BM25 Catches That Dense Embeddings Miss

Dense embedding models (like text-embedding-3-large or bge-large-en-v1.5) project text into a continuous vector space. They excel at capturing conceptual intent, synonymy, and paraphrase. However, they suffer from three distinct failure modes in technical enterprise corpora:

1. Exact Identifiers: Terms like CVE-2024-21412 or error codes like ERR_SOCKET_TIMEOUT_10054 are flattened into embedding clusters representing general security vulnerabilities or network issues.
2. Domain Acronyms: In a corporate setting, MFA might mean Multi-Factor Authentication, Master of Fine Arts, or Ministry of Foreign Affairs. Without exact lexical matching, vector search frequently surfaces out-of-domain matches.
3. Low-Frequency Jargon: Specialized part numbers, chemical formulas, and internal codenames rarely have strong representation in pre-trained embedding vocabularies.

Conversely, BM25 (Best Matching 25) is a probabilistic term-frequency/inverse-document-frequency (TF-IDF) ranking algorithm. It does not care about semantic meaning; it measures exact term frequency penalized by document length.

Combining both signals ensures that whether a user searches by natural concept or exact SKU, relevant chunks surface to the top.

How Reciprocal Rank Fusion (RRF) Works

When combining BM25 keyword results with dense vector results, you cannot simply add their raw scores together. BM25 scores are unbounded positive numbers (e.g., 12.45), while cosine similarity scores range from -1.0 to 1.0 (or 0.0 to 1.0).

Normalizing raw scores requires tuning arbitrary min-max scalars that break whenever the document index changes.

Reciprocal Rank Fusion (RRF) solves this by ignoring raw score magnitudes entirely and operating exclusively on ranks.

The RRF Formula

For a document d appearing in rank positions r_m(d) across multiple retrieval lists M:

1RRF_score(d) = sum( 1 / ( k + r_m(d) ) )  for m in M

Where:

• M is the set of retrieval systems (e.g., BM25 and Dense Vector).
• r_m(d) is the 1-indexed rank of document d in system m.
• k is a constant parameter (typically k = 60).

Why k = 60?

The smoothing constant k = 60 prevents top-ranked items from dominating the fused score excessively. A document ranked #1 in BM25 contributes 1 / (60 + 1) = 0.01639. A document ranked #2 contributes 1 / (60 + 2) = 0.01612. The difference is subtle enough that a document appearing at rank #3 in both BM25 and Vector search will outrank a document appearing at #1 in BM25 but missing entirely from Vector search.

1def reciprocal_rank_fusion(
2    vector_results: list[str],
3    bm25_results: list[str],
4    k: int = 60
5) -> list[tuple[str, float]]:
6    """
7    Combines two ranked lists of document IDs using Reciprocal Rank Fusion.
8    """
9    scores: dict[str, float] = {}
10
11    for rank, doc_id in enumerate(vector_results, start=1):
12        scores[doc_id] = scores.get(doc_id, 0.0) + (1.0 / (k + rank))
13
14    for rank, doc_id in enumerate(bm25_results, start=1):
15        scores[doc_id] = scores.get(doc_id, 0.0) + (1.0 / (k + rank))
16
17    # Sort documents by descending fused score
18    sorted_docs = sorted(scores.items(), key=lambda item: item[1], reverse=True)
19    return sorted_docs

Azure AI Search Implementation

Azure AI Search provides native hybrid search with built-in RRF. When you send a query containing both vectorQueries and a text search string, Azure AI Search executes both searches in parallel and applies RRF automatically before returning results.

Here is how to configure hybrid search with semantic reranking using the Python SDK:

1from azure.core.credentials import AzureKeyCredential
2from azure.search.documents import SearchClient
3from azure.search.documents.models import VectorizedQuery
4
5def execute_hybrid_search(
6    query_text: str,
7    query_vector: list[float],
8    service_endpoint: str,
9    index_name: str,
10    api_key: str,
11    top_k: int = 5
12):
13    client = SearchClient(
14        endpoint=service_endpoint,
15        index_name=index_name,
16        credential=AzureKeyCredential(api_key)
17    )
18
19    vector_query = VectorizedQuery(
20        vector=query_vector,
21        k_nearest_neighbors=50,
22        fields="vector_content"
23    )
24
25    # Hybrid Search: search parameter executes BM25, vector_queries executes Dense Vector
26    # Azure AI Search automatically applies RRF across both result sets.
27    results = client.search(
28        search_text=query_text,
29        vector_queries=[vector_query],
30        query_type="semantic",
31        semantic_configuration_name="default-semantic-config",
32        top=top_k
33    )
34
35    return [
36        {
37            "id": doc["id"],
38            "content": doc["content"],
39            "reranker_score": doc.get("@search.reranker_score")
40        }
41        for doc in results
42    ]

Adding a Cross-Encoder Reranking Pass

RRF produces a strong candidates list (e.g., top 50 chunks). However, because RRF operates solely on rank positions, it does not evaluate deep contextual alignment between the user query and the retrieved text.

Adding a second pass with a cross-encoder reranker (such as bge-reranker-large or Azure's Semantic Reranker) analyzes the query and candidate chunk simultaneously through multi-head self-attention.

While bi-encoders generate separate embeddings for queries and documents (fast retrieval), cross-encoders process query + document together (high accuracy, higher latency). By limiting the cross-encoder to the top 50 RRF candidates, total pipeline latency remains under 200ms while retrieval accuracy reaches 90%+.

Benchmark Results

Testing on an internal enterprise dataset of 5,000+ technical PDF documents produced the following precision metrics across 250 evaluation queries:

The combination of BM25 exact keyword matching, dense vector semantic search, RRF score fusion, and cross-encoder reranking represents the baseline architecture for production enterprise RAG.

Full implementation details and test benchmarks are available in the Document Intelligence RAG case study.