Question

When combining keyword-based search with vector-based search in a hybrid system, which mathematical technique is used to normalize scores from two different retrieval methods so they can be merged effectively?

Accepted Answer

The mathematical technique used to normalize scores from two different retrieval methods is Reciprocal Rank Fusion, also known as RRF. Keyword-based search typically produces scores based on term frequency, such as BM25, which can range from zero to very high numbers, while vector-based search produces cosine similarity scores that typically range between negative one and one. Because these two systems use different scales, their scores cannot be added or compared directly. Reciprocal Rank Fusion solves this by ignoring the raw scores entirely and focusing on the rank position of each result. RRF assigns a score to each document based on the formula of one divided by the sum of a constant value, usually sixty, and the rank of the document in the result list. By calculating this value for every document across both search methods and then summing the results, the system creates a unified score that reflects the consensus rank of the document. This method effectively merges the two systems because it treats a high rank in either the keyword or vector search as a strong signal, regardless of the original underlying numerical scale.

Home → All Courses → Engineering and Technology Courses → Generative AI Application Development → Flashcard

When combining keyword-based search with vector-based search in a hybrid system, which mathematical technique is used to normalize scores from two different retrieval methods so they can be merged effectively?