Reranking: the retrieval step most finance RAG skips

Reranking is a second retrieval pass that reads each candidate passage against the actual question and reorders the list, moving the true answer to rank 1. Most finance RAG skips it, so the correct passage sits at rank 14 where the generator never reads it. That gap between retrieved and ranked is where accuracy quietly leaks.

First-pass retrieval, whether vector, BM25 or a hybrid of both, is tuned for recall. It casts a wide net and returns a few dozen passages that all look plausibly relevant. The generation model then reads only the top three to five chunks. If the passage that names the right revenue segment or the right covenant threshold ranks eighth, the model answers from whatever did make the cut. The answer reads fine, and it is wrong, or it is grounded in the wrong entity, and nothing downstream flags it because the retrieval log looks healthy. Recall was high. Precision at the point of use was not.

What a first-pass retriever gets wrong

A bi-encoder embeds the query and the passages separately and compares them with a dot product. It has to, because that is what makes the index searchable in milliseconds across millions of chunks. But it means the model never sees the query and the passage together. It approximates relevance from two vectors computed in isolation, and on financial text that approximation slips in predictable ways.

• Two passages about the same metric for different entities look nearly identical in embedding space. Ask about one issuer’s leverage ratio and the top hits mix in a peer’s, because “net leverage” dominates the vector and the entity token barely moves it.
• Point-in-time distinctions collapse. The Q2 and Q3 versions of the same disclosure are semantically almost the same sentence. A bi-encoder has no reason to prefer the one that matches the as-of date in the question.
• Negations and conditions get flattened. “The covenant does not apply below the threshold” and “the covenant applies above the threshold” sit close together, and the retriever cannot tell which one the analyst needs.

A cross-encoder reranker fixes exactly this class of error. It takes the query and one candidate passage as a single input and runs full attention across both, so the entity, the date and the condition are read in context instead of averaged into a vector. It is far too slow to run over a whole corpus, which is why it never replaces first-pass retrieval. It runs over the 30 or 50 candidates the first pass already narrowed to, and that is a job it can finish inside a normal request.

Adding the reranker without blowing latency

The naive fear is that a second model in the path doubles your latency. In practice the reranker is bounded work, and you control the bound. The pattern we use in most finance builds:

• First pass returns a fixed candidate set. We usually pull 40 to 60 from hybrid retrieval, not more. Beyond that the reranker cost climbs and recall gains flatten.
• The cross-encoder scores all candidates against the query in one batched forward pass. Batching matters. Sending 50 individual requests is slow for no reason.
• Truncate the passage side. If your chunks are large, the reranker only needs enough of each to judge relevance. Capping input length is the single biggest latency lever after candidate count.
• Keep the reranker on the same box as the retriever, or close to it. A network hop per candidate defeats the point.

There is a cheaper variant worth knowing. You do not always have to rerank. If the first-pass score gap between rank 1 and rank 2 is wide, the retrieval is confident and reranking rarely changes the order, so you can skip it and save the budget for the ambiguous queries where it actually moves rank 1. Gate the reranker on the score margin and you spend compute where it changes the answer.

Model choice is less exotic than it sounds. A small cross-encoder in the 100 to 300 million parameter range, fine-tuned or off the shelf, handles most finance corpora. The domain adaptation that pays off comes from training pairs drawn from your own corpus, not from a bigger model. Feed it entity-mismatched hard negatives and wrong-quarter hard negatives, the confusable cases the bi-encoder keeps getting wrong. A few thousand of those teach the reranker the distinctions that matter here.

Prove it moved the number, then keep watching it

A reranker is only worth its latency if you can show it changed retrieval quality on cases you care about, and the only way to know is a held-out eval set with labelled correct passages. Measure the metric that maps to how the passages get used. Since the generator reads the top few chunks, recall@3 and MRR are the numbers that matter, not recall@50, which the first pass already maxed out. If recall@3 barely moves after reranking, the reranker is not earning its place and you should find out in eval, not in production.

Build the eval set from real queries, with the answer passage identified by someone who knows the corpus, and freeze it. When you swap the reranker, change the candidate count, or re-chunk the documents, rerun the same set and compare. A reranker also drifts as the corpus grows: new document types and new entities its training pairs never covered. The confusable pairs it handles today are not the ones next quarter’s filings will produce. Treat the reranker as a component you re-measure on a schedule, with its scores and its inputs written to the same audit trail as the rest of the retrieval path, so that when an answer is questioned you can show which passages were ranked and why the top one won.