Building a RAG agent that knows how to say "I don't know"

Most RAG (Retrieval-Augmented Generation) demos work fine on the easy cases: the question resembles a passage in the corpus, retrieval returns the right document, and the model answers correctly. The problem shows up on the remaining 20% of questions — the ones where the corpus doesn’t actually contain the answer.

The real problem isn’t generation

We tend to treat hallucinations as a language-model problem. In practice, in a RAG pipeline, it’s often a silent retrieval problem: the retriever always returns documents, even when none are relevant, and the model does its best with whatever it’s handed.

An LLM given three irrelevant passages will (almost) never answer “I don’t know” — it will try to be helpful, and so it invents a bridge between the question and the passages.

Three levers that actually change the behaviour

1. An explicit relevance threshold, not just a top-k

Retrieving the k nearest documents isn’t the same as retrieving relevant documents. Adding a minimum score threshold — and accepting that it can return zero documents — is the simplest change with the biggest impact:

def retrieve(query: str, k: int = 5, min_score: float = 0.72):
    hits = vector_store.similarity_search_with_score(query, k=k)
    return [doc for doc, score in hits if score >= min_score]

2. Make an empty context a normal case, not an exception

If retrieve() can return an empty list, the prompt should handle that explicitly rather than treating it as an upstream error:

Context: {context or "No relevant document found."}

If the context doesn't support a confident answer,
say explicitly that you don't know rather than guessing.

3. A verification step that checks the answer against the sources

A separate agent, whose only job is to check that every claim in the answer is actually backed by at least one retrieved passage, catches a good chunk of the remaining cases — at the cost of an extra round trip, usually acceptable for anything that isn’t strictly real-time.

What this changes in practice

The result isn’t a perfect system — there are still false negatives (the system says “I don’t know” even though the answer existed, just phrased differently in the query). But it’s the right trade-off for most production use cases: better an agent that admits its limits than one that confidently makes things up.