RAG Retrieval Quality: Recall, MRR and NDCG

When a RAG answer is wrong, the problem may start before the language model sees the prompt. The retriever may miss a useful document, rank it too low or return near-duplicates that crowd out better evidence. Measuring retrieval separately from answer generation helps you find those failures before changing prompts or switching models. Three useful metrics are Recall@k, Mean Reciprocal Rank (MRR) and Normalized Discounted Cumulative Gain (NDCG).

Create a labeled query set

Collect representative questions from the tasks your system must support. For each question, label which documents or chunks are relevant. If some documents are more useful than others, use graded labels such as highly relevant, partly relevant and not relevant. Keep these labels independent from the retriever being tested; otherwise the evaluation can reward the current system for finding the results it already returns.

Include common questions, long-tail queries, ambiguous wording, recent documents, rare entities and cases where the right answer is “not found.” Split results by task or source type so you can see whether a change helps one group while hurting another. A small, carefully labeled set is more useful than a large set with no reliable relevance judgments.

Recall@k measures whether evidence was found

Recall@k asks what fraction of known relevant documents appear in the first k results. If a question has five relevant documents and the retriever returns three of them in its top five, Recall@5 is 3/5, or 0.60. Calculate the score per query, then aggregate results consistently.

Recall is useful when missing evidence is costly. A low score can point to chunking problems, an embedding mismatch, a narrow filter or a query that needs rewriting. Increasing k may retrieve more relevant material, but it can also add noise and use more context tokens. Do not increase k without checking which extra documents are introduced.

MRR measures how early the first relevant result appears

Mean Reciprocal Rank focuses on the rank of the first relevant result. For each query, take the reciprocal of that rank: a relevant document in position one contributes 1.0, position two contributes 0.5 and position four contributes 0.25. MRR is the average across the query set.

This metric helps when one strong source near the top is enough to answer the question or when the application relies heavily on the first few chunks. MRR does not measure all relevant documents, so it should not replace Recall@k. A system can rank one relevant document first while still missing other evidence needed for a multi-part question.

NDCG measures the quality of the ranking

Normalized Discounted Cumulative Gain supports graded relevance. It gives more weight to highly relevant documents and discounts documents that appear lower in the list. The score is normalized against an ideal ranking for the same query, which helps compare ranking quality across queries with different relevance labels.

NDCG is useful when the top few results matter more than the tail and when “partly relevant” is meaningfully different from “directly answers the question.” Choose a cutoff such as NDCG@5 or NDCG@10 based on how many chunks the application actually sends to the model. Record the cutoff with the score; NDCG@5 and NDCG@10 answer different questions.

Separate retrieval from answer evaluation

Retrieval metrics do not tell you whether the final answer is correct. After retrieval, evaluate the generated answer for groundedness, completeness, relevance and correctness. A response can be grounded in retrieved text but still draw the wrong conclusion; it can also be factually correct while ignoring the user's actual question.

Microsoft's RAG evaluation guidance recommends examining retrieval and generation separately, then choosing metrics that fit the workload. NVIDIA's RAG evaluation documentation similarly separates document retrieval measures from answer-generation measures such as faithfulness, answer relevance and correctness. Use a combination rather than reducing the whole system to one score.

Compare retrieval strategies fairly

1. Set a baseline. Save the current retriever, chunk size, overlap, top-k, filters and reranking settings.
2. Change one variable. Compare keyword search, dense retrieval or a hybrid approach using the same query set and relevance labels.
3. Use the same cutoff. Record Recall@k, MRR and NDCG@k for the same k values the application uses.
4. Inspect failures. Review queries with low recall or poor ranking and identify whether the issue is chunking, query formulation, metadata filtering or ranking.
5. Run end-to-end tests. Check whether improved retrieval helps answer completeness and groundedness without adding too much latency or irrelevant context.

Do not compare two retrievers using different test questions or relevance labels. Record the index version and source corpus too. A changing document collection can make a score rise or fall even when the retrieval algorithm stays the same.

Keep evaluation useful after launch

Save a stable regression set and add cases when users report a missed answer, stale document or misleading citation. Keep a holdout set for decisions that could overfit the same queries. If you use an LLM as a judge for answer quality, review a sample manually and track disagreement; judge scores are estimates, not ground truth.

If key documents are missing, inspect Recall@k. If one relevant result is buried, inspect MRR and NDCG. If retrieval looks healthy but answers remain poor, examine the prompt, grounding and generation stage. Our model evaluation workflow, structured-output validation guide and LLM latency diagnostic guide cover adjacent checks without replacing retrieval-specific evaluation.

Sources

• Microsoft Learn: Large language model end-to-end evaluation for RAG.
• NVIDIA RAG Blueprint: Evaluate your RAG system.