CRAG: Grade Your Retrieval Before You Trust It

RAG systems assume that if retrieval returns something, it returned something relevant. That assumption is wrong often enough to matter. Dense retrieval produces results for every query — it has no way to say “nothing useful was found.” The cosine similarity score tells you the best match found, not whether that match is actually good.

When irrelevant context reaches the LLM, the LLM doesn’t say “I don’t know”. It reasons from the bad context and produces a confident, hallucinated answer. Prompt guardrails (“only answer from the provided context”) reduce this — but they don’t eliminate it, and they don’t fix the root cause.

CRAG (Corrective Retrieval Augmented Generation) adds an evaluator between the retriever and the generator. Each retrieved chunk gets a score: CORRECT (clearly relevant), AMBIGUOUS (partially relevant), or INCORRECT (not relevant). The system routes based on those verdicts — using only CORRECT chunks for generation, refining ambiguous results, or triggering a fallback entirely.

Think of a researcher preparing a briefing. A naive approach: hand them the first five search results and tell them to write the report. They’ll use whatever they get, even if two of the five results are irrelevant — the brief will sound confident even where it’s wrong.

CRAG is the editorial review step. Before the researcher writes, an editor reads the five sources and marks each one: solid reference, questionable, unrelated. The researcher only uses the solid references. If none pass, they go find better sources rather than writing from bad ones.

The evaluator in CRAG is itself an LLM, acting as a lightweight judge. It receives the query and each chunk and outputs a relevance verdict. The cost is one additional pass per retrieval call — but it’s a much cheaper model than the generator, and it prevents the expensive failure mode of confident hallucination.

Type any query below. CRAG evaluates a set of hypothetical retrieved chunks and shows the verdict for each — CORRECT, AMBIGUOUS, or INCORRECT — with the routing decision at the end.

↑ Notice that the evaluator doesn’t just score — it routes. No CORRECT chunks means no generation from bad context.

Standard RAG passes all retrieved chunks directly to the generator:

$\text{Answer} = \text{LLM}_\text{gen}(q, C) \quad \text{where } C = \text{Retrieve}(q, k)$

CRAG inserts an evaluation step. For each chunk $c_i$, the evaluator assigns a score and verdict:

$\text{score}(c_i) = \text{LLM}_\text{eval}(q, c_i) \in [0, 1]$

$\text{verdict}(c_i) = \begin{cases} \text{CORRECT} & \text{if } \text{score}(c_i) \geq \theta \\ \text{AMBIGUOUS} & \text{if } \theta - 0.3 \leq \text{score}(c_i) < \theta \\ \text{INCORRECT} & \text{otherwise} \end{cases}$

The routing decision depends on the evaluation outcome. The confidence threshold $\theta$ is the key engineering parameter — it controls the precision-recall tradeoff:

↑ Low threshold: more chunks pass, fewer fallbacks triggered, lower precision. High threshold: fewer chunks pass, more fallbacks, higher precision. Set $\theta$ based on your knowledge base quality and how expensive fallbacks are.

CRAG evaluates retrieval at query time and routes to a fallback when needed. The next evolution: FLARE (Forward-Looking Active Retrieval) makes retrieval itself dynamic — triggering additional retrieval passes mid-generation whenever the model detects its own uncertainty, rather than evaluating upfront.