Question

When evaluating an automated RAG system, which specific metric measures the ability of the system to rank the single most relevant document at the very top of the retrieved list?

Accepted Answer

The specific metric that measures the ability of a system to place the single most relevant document at the top of the retrieved list is Mean Reciprocal Rank, commonly abbreviated as MRR. This metric evaluates a retrieval system by looking at the rank position of the first relevant document. The Reciprocal Rank is calculated as the multiplicative inverse of the rank of the first correct answer; for example, if the most relevant document appears at the first position, the score is one divided by one, equaling one. If it appears at the second position, the score is one divided by two, equaling zero point five. If it appears at the third position, the score is one divided by three, or approximately zero point three three. The Mean Reciprocal Rank is then determined by averaging these individual reciprocal rank scores across a set of multiple test queries. A higher MRR indicates that the system is more consistently placing the best possible result at the highest possible position in the list.

Home → All Courses → Engineering and Technology Courses → Generative AI Application Development → Flashcard

When evaluating an automated RAG system, which specific metric measures the ability of the system to rank the single most relevant document at the very top of the retrieved list?