How Do You Grade a RAG System?

The previous experiment graded one prediction (is this spam?). But a RAG retriever hands back a ranked list of passages — and the tempting hypothesis is “if the right chunk is in there, retrieval worked.” FAR FROM IT. A gold chunk at rank 80 is a chunk the model never reads. I built the retrieval-evaluation toolkit from first principles — and the two metrics that matter, MRR and NDCG, fall out of asking “what should a good ranking score reward?“

RAG eval has two layers — retrieval (did we fetch the right context) and generation (is the answer good given it). This post is retrieval. And retrieval fails in exactly two ways, both names you already know:

• ①Missed a needed chunk → context recall — plain: of all the info the answer needed, how much did we fetch? (Comparing HyDE vs Step-Back needs both chunks; miss one and the answer can’t be written.)
• ②Grabbed junk → context precision — plain: of what we fetched, how much was actually relevant? Distractor chunks don’t just waste space — they pull the model’s attention and degrade the answer.

And the lever between them is familiar: the similarity-score threshold is the same dial as a classifier’s — slide it and you trade context precision against context recall, exactly like Bouncer Babu’s strictness knob.

Here’s why “is it in the set?” isn’t enough. LLMs suffer from “lost in the middle”: they attend most to the start and end of a long prompt, so a relevant chunk buried deep gets under-read even though it’s technically in the context. So a set-based metric like recall@k (is a good chunk in the top k) gives full marks whether the gold chunk sits at rank 1 or rank 19 — and those are wildly different outcomes for the model.

The fix in practice is a reranker (plain: a cheap first pass casts a wide net, then a heavier scorer — a cross-encoder or hybrid BM25+dense — reorders the catch so the best chunks sit on top). But to even measure this, we need a metric that rewards relevant chunks for being high, not just present.

Simplest case: one query, one right answer. Score it by where the gold chunk landed — and you want rank 1 to be great, rank 20 nearly worthless, with a big penalty at the top and a flat tail. Put the rank in the denominator:

$\text{MRR} = \frac{1}{Q}\sum_{q=1}^{Q} \frac{1}{\text{rank of the first relevant chunk}}$

↑ Slide where the gold chunk lands. 1/rank halves from rank 1→2 (the penalty you want for falling off the top) and barely moves 19→20 (already buried). The dashed line is a linear score — notice it can’t tell rank 1 from rank 3.

Average that over all your queries and you’ve built MRR — mean reciprocal rank. It’s perfect for one-right-answer retrieval, but it’s blind to everything after the first hit: if a query needs the HyDE chunk and the Step-Back chunk, MRR scores a smug 1.0 the moment it finds HyDE — even with Step-Back rotting at rank 80.

To fix MRR’s blind spot, count every relevant chunk, and let chunks be graded (perfect=3, decent=2, weak=1, useless=0), not just yes/no. Each chunk’s contribution is two factors multiplied — how relevant it is times how high it sits:

$\text{DCG} = \sum_i \frac{rel_i}{\log_2(1 + \text{rank}_i)} \qquad \text{NDCG} = \frac{\text{DCG}}{\text{IDCG}} \in [0,1]$

IDCG is the score of the ideal ordering (highest grades on top); dividing by it normalizes to [0,1] so queries are comparable. (Neat: the log base cancels in the ratio.)

↑ Reorder the chunks, or hit “bury the gold.” Watch IDCG never move while DCG — and so NDCG — collapses.

We’ve graded the fetch. But perfect retrieval doesn’t guarantee a good answer — the model can have every right chunk in hand and still hallucinate or wander off the question. Next: faithfulness and answer relevancy — when the retrieval is perfect and the answer still lies (EXP-013).