A small cheat sheet: evaluating a RAG pipeline

In the order they’re most likely to reveal a problem:

1. Retrieval, before generation. Over a sample of questions, what share brings back at least one genuinely relevant document? Without that number, there’s no way to know whether a bad answer comes from the retriever or the model.
2. Out-of-scope questions. What does the system answer when the right answer isn’t in the corpus? This is often the most revealing case, and the most neglected.
3. Consistency over time. Asked twice, does the same question get the same answer? If not, that’s not necessarily a bug — but it needs to be known before someone else discovers it.