Evaluate a RAG Pipeline Before Your Users Do

You will build a small, repeatable evaluation harness for a retrieval-augmented generation (RAG) pipeline: a golden dataset of questions with known-good source passages, retrieval metrics that tell you whether search is the problem, generation checks that tell you whether the model is inventing details, and a scoring report you can run in CI before any change to chunking, embeddings, prompts, or models reaches your users.

The uncomfortable truth about RAG is that the demo always works. You ask three questions you already know the answers to, the answers look right, and you ship. Then a user asks a question phrased differently, about a passage your chunker split in half, and the model confidently answers from the wrong document or from nothing at all. Evaluation is how you find that out in a test run instead of in a support ticket.

What you are evaluating, exactly

A RAG pipeline fails in two independent places, and your harness has to measure them separately, or you will fix the wrong one:

1. Retrieval failure. The passages needed to answer the question never reach the model. No prompt engineering fixes this.
2. Generation failure. The right passages arrive, but the model ignores them, contradicts them, or pads the answer with plausible detail that is not in the sources.

Every metric below belongs to one of these two buckets. When a score drops, you should be able to say which stage regressed.

Step 1: Build a golden dataset

A golden dataset is a list of questions, each paired with the passages that should be retrieved and a reference answer. You do not need hundreds of rows to start. A few dozen carefully chosen questions will catch most regressions; you can grow the set every time a real user finds a failure, by adding that exact question to the set so it can never silently recur.

Write your questions from your actual documents, and deliberately include the awkward cases:

• Questions whose answer spans two passages in different documents.
• Questions phrased with synonyms your documents do not use.
• Questions about information that appears in a table or a list, not prose.
• Questions with no answer in the corpus at all. The correct behaviour here is a refusal or an honest “I don’t know”, and your harness should grade for that instead of rewarding a confident guess.

from dataclasses import dataclass, field
 
 
@dataclass
class GoldenCase:
    question: str
    relevant_ids: set[str]          # chunk ids that contain the answer
    reference_answer: str           # short, factual, written by a human
    should_abstain: bool = False    # True when the corpus has no answer
 
 
CASES = [
    GoldenCase(
        question="How long do deleted projects stay recoverable?",
        relevant_ids={"retention-policy#3"},
        reference_answer="Deleted projects stay recoverable for 30 days.",
    ),
    GoldenCase(
        question="Can I export my data to a competitor's format?",
        relevant_ids=set(),
        reference_answer="",
        should_abstain=True,
    ),
]

Keep the dataset in version control, next to the code. It is a test suite, and it deserves the same treatment: reviewed changes, no silent edits to make a score pass.

Step 2: Score retrieval on its own

Run each question through your retriever without generating an answer, and compare the returned chunk ids against relevant_ids. Three metrics cover almost everything you need:

• Recall@k: of the chunks that should be found, what fraction appear in the top k results? This is the ceiling on answer quality. If recall@k is low, generation scores are meaningless.
• Hit rate: for how many questions does at least one relevant chunk appear in the top k? A simple, brutal summary number.
• Mean reciprocal rank (MRR): how high does the first relevant chunk rank? Rank matters because models attend unevenly to long contexts; a relevant passage buried at position 38 of 40 behaves a lot like a missing passage.

def recall_at_k(retrieved: list[str], relevant: set[str]) -> float:
    if not relevant:
        return 1.0
    return len(set(retrieved) & relevant) / len(relevant)
 
 
def reciprocal_rank(retrieved: list[str], relevant: set[str]) -> float:
    for i, chunk_id in enumerate(retrieved, start=1):
        if chunk_id in relevant:
            return 1.0 / i
    return 0.0
 
 
def evaluate_retrieval(cases, retrieve, k=10):
    recalls, ranks, hits = [], [], []
    for case in cases:
        if case.should_abstain:
            continue
        top = retrieve(case.question, k=k)
        recalls.append(recall_at_k(top, case.relevant_ids))
        ranks.append(reciprocal_rank(top, case.relevant_ids))
        hits.append(1.0 if set(top) & case.relevant_ids else 0.0)
    n = len(cases)
    return {
        "recall@k": sum(recalls) / n,
        "mrr": sum(ranks) / n,
        "hit_rate": sum(hits) / n,
    }

Two failure patterns to watch for. First, recall that is fine at k=20 but poor at k=5: your reranker or your similarity threshold is doing damage, or your chunks are near-duplicates crowding out the one chunk you need. Second, a single document dominating results across unrelated questions: usually a chunking artefact, such as a giant FAQ chunk whose embedding sits close to everything.

Step 3: Score the generated answer

With retrieval measured, generate answers with the retrieved context and grade three things.

Faithfulness. Every factual claim in the answer should be supported by the retrieved passages. The practical way to check this at scale is claim decomposition: split the answer into atomic claims, then verify each claim against the context. You can do the verification with a model call, but read the next section before trusting it.

Answer relevance. Does the answer address the question that was asked, including its constraints? A faithful answer to a slightly different question is still a failure your users will feel.

Abstention behaviour. For your should_abstain cases, grade whether the system declined instead of answering. Score this as pass/fail, and treat a failure as severe: a fabricated answer to an unanswerable question is the worst output a RAG system produces.

CLAIM_CHECK_PROMPT = """You are verifying an answer against source passages.
 
Passages:
{context}
 
Claim: {claim}
 
Answer with exactly one word: SUPPORTED if the passages state or clearly
imply the claim, CONTRADICTED if they state the opposite, or UNSUPPORTED
if the passages say nothing about it."""
 
 
def check_claims(claims, context, judge):
    verdicts = [judge(CLAIM_CHECK_PROMPT.format(context=context, claim=c))
                for c in claims]
    supported = verdicts.count("SUPPORTED")
    return {
        "faithfulness": supported / len(verdicts) if verdicts else 1.0,
        "contradicted": verdicts.count("CONTRADICTED"),
        "unsupported": verdicts.count("UNSUPPORTED"),
    }

Track contradicted separately from unsupported. They are different bugs: unsupported claims usually mean your prompt permits general knowledge; contradicted claims mean your model is misreading the passage, and that points at context formatting, chunk boundaries, or a model that is too small for the reading comprehension you are asking of it.

The judge problem

Using a model to grade a model is convenient and genuinely useful, but it has known failure modes you have to design around:

• Self-preference. A judge tends to rate outputs from its own model family more leniently. Where you can, grade with a different model than the one generating answers.
• Verbosity bias. Longer answers score higher on relevance-style rubrics even when the extra text is filler. Grade faithfulness per claim, not per answer, and keep the rubric about support, not style.
• Position and ordering effects. When a judge compares two answers, swapping their order can flip the verdict. For regression testing, absolute rubrics against your reference answer are more stable than pairwise comparisons.
• Rubric drift. A judge prompt is code. Version it, and re-run your whole golden set when you change it, or your score history stops being comparable.

The pragmatic setup: use the judge for breadth (every case, every run), and hand-grade a small rotating sample yourself to check the judge is not quietly going soft.

A failure taxonomy you can act on

When a case fails, classify it before you fix anything. Most RAG debugging goes wrong because a retrieval fix is applied to a generation problem.

The “right answer, wrong document” row deserves emphasis in multi-tenant systems: if your retriever is not scoped by tenant or permissions metadata, evaluation will not catch it unless your golden set includes questions whose correct passages differ per tenant. Add those cases deliberately; a cross-tenant leak that produces a correct-looking answer is the failure your users cannot see and you cannot afford.

Step 4: Gate changes in CI

An evaluation you run by hand when you remember is a demo, not a harness. Wire it into the change process:

1. Run the harness on every change to chunking, embedding model, index, prompts, or generation model.
2. Compare against the stored baseline scores, not against perfection. Fail the build on a meaningful drop in recall@k, faithfulness, or abstention pass rate.
3. Store the per-case results, not just the averages. When a score drops, you want to open the exact failing questions and read what changed.
4. Keep the generation settings used during evaluation identical to production: same model, same temperature, same context size. Evaluating a configuration you do not ship measures a system your users never meet.

Expect the full run to cost real money and time once your golden set grows. That cost is the honest price of changing a system whose failures are otherwise invisible, and it is almost always smaller than the cost of one confident wrong answer reaching a customer who acts on it.

Verdict

Start with a hand-written golden set, measure retrieval and generation separately, and grade abstention as a first-class behaviour. Those three decisions give you a harness that tells you what broke and where, which is the entire point. Add a model judge for scale, but audit it against your own reading on a rotating sample, and never let an average score hide the individual failures: read the failing cases, classify them with the taxonomy, and fix the stage that actually broke. Do that, and your pipeline’s quality becomes a number you control instead of a surprise your users report.