Hybrid Search and Reranking

You wire up a vector store, embed the corpus, retrieve the top-5. The demo works. Three weeks into customer usage, someone searches for "P1 SLA" and the answer is wrong — because the policy document uses the phrase "priority-one" and never the token "P1". The embedding model knows the two are close; your single-retriever pipeline never gave it the chance to prove it, because the candidate list for that query was empty in the right neighbourhood.

This lesson is why real RAG systems run two retrievers and a reranker.

Dense and sparse, and why both

Two retrievers, two different failure modes. Running them together is not a luxury; it is how you avoid being blind to half the queries.

The two retrievers are not interchangeable: they fail on different queries. If you ship a system with only one, you ship a system that is silently bad for half of your customers' phrasing.

Reciprocal rank fusion, or: why you throw away the scores

The temptation when combining two retrievers is to add up their scores. Resist it. BM25 scores and cosine similarities live on different scales, and normalising one to the other is a knob that will haunt you.

The standard move is reciprocal rank fusion. For each document, add up 1/(k + rank) across every retriever that returned it. Rank-only; scores thrown away.

1. Step 1

   Retrieve twice

   Run dense and sparse retrievers independently. Each returns its own top-100, ranked.

2. Step 2

   Score by rank

   For each document in either list, compute 1/(k + rank_dense) + 1/(k + rank_sparse). k is typically 60. Documents only in one list still score, just lower.

3. Step 3

   Fuse and cut

   Sort by the fused score, take the top-100. This is the candidate pool handed to the reranker.

The virtue of RRF is that it gets simpler as the number of retrievers grows. Add a third retriever — a title-only match, say, or a metadata filter — and nothing in the formula changes. It is score-agnostic by construction, which means a tuning change in one retriever does not quietly shift the balance of the whole system.

Rerankers and what they actually do

Retrieval is fast and approximate. The retriever asks "which passages are about this query" and uses a representation it computed once. A reranker asks "given this exact query and this exact passage, which one actually answers it" and recomputes a score for each pair.

The mechanism is called a cross-encoder. Where an embedding model produces a vector for the query and a vector for each document and compares them, a cross-encoder takes query and document together as a single input and emits a relevance score. It is more accurate and it is exactly as expensive as that sounds: one forward pass per (query, document) pair.

Cohere Rerank, bge-reranker and the open rerankers from Jina and Mixedbread all share this shape. The differences are latency, cost per 100 candidates, and how they fare on short versus long passages. The numbers move on a release cadence; the pattern does not.

What to quote a customer honestly. Reranking 100 candidates per query adds roughly 100ms on a hosted reranker and a cost per query that you should put in the TCO slide, not hide in the appendix. If the customer's latency budget is 500ms end-to-end, the reranker takes a fifth of it before the model has written a token. That is a design constraint, and the FDE call is to either cut the candidate pool, batch across queries, or pick a lighter reranker.

Recall@k as the metric that tells you the truth

Each stage of the pipeline has its own failure mode and its own metric. For retrieval, the one that matters is recall@k: of the queries in your eval set, what fraction retrieved the correct passage somewhere in the top-k?

Why not answer accuracy? Because answer accuracy mixes the retrieval failure with the generation failure, and when it drops you cannot tell which half to fix. Recall@k isolates retrieval: if it is high and answers are still wrong, the problem lives downstream; if it is low, nothing downstream can save you.

The eval set itself is the expensive part. For the retrieval metric you need (query, correct-chunk-id) pairs, and chunk ids only survive if your chunking is deterministic and versioned — which is the previous lesson, now earning its keep.

Pick a k that matches what you actually pass to the model. If the reranker's top-5 go to the LLM, measure recall@5. Measuring recall@50 when only 5 reach the model is a lie you tell yourself.

The pipeline you will actually ship

Three stages, each measurable.

1. Step 1

   Hybrid retrieval

   Dense and sparse, each returning their own top-100. Measure recall@100 — if you are not retrieving the right chunk in a hundred, no reranker helps.

2. Step 2

   RRF fusion

   Combine by rank. Output: a single top-100 candidate pool. This step is free; you can skip it if you only have one retriever, and you should not.

3. Step 3

   Cross-encoder reranking

   Score each of the 100 candidates against the query. Take the top-5 or top-10. Measure recall@5 — this is the metric the model sees.

4. Step 4

   Ground and generate

   Hand the top-5 to the model with grounding instructions. Measure groundedness and answer accuracy — the next lesson covers how.

What to take away

Dense and sparse retrievers fail on different queries, so hybrid search is the default rather than the optimisation. Reciprocal rank fusion combines them by rank rather than score, which is what makes it robust across tuning changes and extra retrievers. A cross-encoder reranker over the top-100 is the step that turns "the right chunk is in here somewhere" into "the right chunk is at position one"; the honest cost is one model call per candidate, and the honest latency ceiling is your customer's. Recall@k is the metric that keeps the retrieval failure visible on its own, and it is only measurable if your chunks have stable ids.

Next: grounding and citation — binding the answer back to the passages, so the customer can click through to the one sentence that decided their outcome.