RAG Evaluation Set: How to Build One Before You Ship

A RAG evaluation set is a fixed list of real questions, each paired with the source passages that answer it and a short reference answer. You grade retrieval and generation separately: did the right passages come back, and did the answer stay inside them, cite them, and refuse when the sources are silent? Then you rerun it on every change.

TL;DR: Write around fifty real questions that cover every way RAG breaks, including out-of-scope and permission-scoped ones. Record gold passages and a reference answer for each. Grade retrieval first (hit rate at k), then answers (correct, grounded, cited, refused when it should). Store the set in the repo and run it whenever chunking, prompts, or models change.

This is a method, not a product. The parts we can point to in our own systems are the retrieval scoping rule in NoteTube and the quote check in SigmaCPM. The rest is how we'd test any RAG system before trusting it with users.

Why isn't "it looks good in the demo" enough?

Because RAG fails quietly. An answer can read fluently and be wrong, be right but drawn from the wrong document, or answer a question the sources never covered. Demo questions are the ones the builder already knows work, so spot checks miss exactly the failures that matter.

There are also a lot of moving parts. Chunk size, overlap, the embedding model, how many passages you retrieve, reranking, filters, the prompt, and the model all change results. A change that helps contract questions can hurt table questions. You'll only see that if you ask the same questions every time and compare.

Where do the questions in a RAG evaluation set come from?

From the people who'll use the system, not from your imagination. Three sources work:

1. Support tickets, emails, and chat logs, if the system replaces a person answering questions.
2. Interviews with future users: "what do you look up most often, and where?"
3. The documents themselves. Headings, tables, and appendices hint at what people come to find.

Keep the phrasing real: typos, shorthand, half-sentences. OpenAI's evaluation guide makes the same point, recommending a mix of production data and test cases that include "typical cases, edge cases, and adversarial cases" (OpenAI docs).

How many? Anthropic's testing guide says to prioritise volume, because more automatically graded questions beat a few hand-graded ones (Anthropic docs). For RAG the bottleneck is gold passages, which a person has to find. So start with about fifty carefully labelled questions, automate the grading, and grow from there.

Which question types break RAG?

Each of these fails in its own way, so the set needs all of them:

Type	Example shape	What it tests
Single fact	"What is the notice period in the vendor contract?"	Basic retrieval and quoting
Multi-passage	"Compare the warranty terms in the two contracts."	Retrieving and combining chunks
Table or number	"What was the bond amount on line 14?"	Table chunking, exact numbers
Out of scope	A question the documents don't answer	Refusing instead of inventing
Ambiguous	"What is the deadline?" with several deadlines	Asking back or listing options
Wrong premise	A question that assumes something false	Correcting rather than agreeing
Multi-turn	"And what about the second site?" after an earlier question	Carrying context into retrieval
Permission-scoped	An answer that sits only in another user's documents	Access control in retrieval

The out-of-scope and permission-scoped rows matter most. A system that answers everything is the dangerous one.

What goes into each gold record?

For every question, store five things: the question exactly as typed, the gold passages (document, page or section, and the text), a short reference answer from someone who knows the material, the expected behaviour for edge cases (answer, refuse, ask, or list), and tags for document type, question type, and difficulty. Here is an illustrative row (a made-up tender, not a client document):

Field	Example
Question	"whats the retention money on the civil works tender"
Gold passages	tender_0412.pdf, page 37, clause 9.2
Reference answer	"Retention is held from each running bill until the defects period ends, at the rate stated in clause 9.2."
Expected behaviour	Answer, citing page 37
Tags	tender, single-fact, lowercase-typo

Have a domain expert review the gold data before you trust a single score. If a reference answer is wrong, every number built on it is wrong too.

How do you grade retrieval on its own?

Run retrieval without generating anything, and check whether the gold passages come back. Most RAG problems are retrieval problems: if the right passage never reaches the model, no prompt will rescue the answer. So measure this layer first.

1. For each question, run retrieval and take the top k passages.
2. Check whether the gold passages appear in that set.
3. Record the rank of the first correct passage.

That gives you a hit rate at k and an average rank, which you track per tag. If table questions retrieve badly, look at how tables are chunked. If one document type is always missed, look at its parsing. Frameworks formalise the same idea: Ragas defines context recall as the share of reference contexts that the retriever actually returned, which needs exactly the gold passages you recorded (Ragas docs).

During my ML internship at Lamina Solutions I built RAG with ChromaDB and Pinecone, and the lesson that stuck is that the vector store matters much less than whether you can measure what it returns.

What does a permission-scoped test look like?

The correct result is that another user's passage never appears, at any k. Test it with two real accounts, not one account and a mocked filter. In NoteTube, retrieval in Qdrant is scoped per user and per session, so a student's chat can only draw on their own sources. Qdrant applies that kind of rule as a payload filter at search time (Qdrant docs). An eval set should prove the rule holds; our post on tenant isolation in vector search covers how to build it.

How should the answers be graded?

Generate answers for every question and grade each on four checks:

1. Correct? Does it agree with the reference answer?
2. Grounded? Is every claim supported by the retrieved passages?
3. Cited? Does it point to the passages it used, and do those passages contain the claim?
4. Right refusal? For out-of-scope questions, did it say the sources don't cover this?

For the citation check, don't rely on a judge's opinion. Require a quote and verify it in code; the method is in citation checking for AI extraction, and SigmaCPM uses it to drop any value whose quote isn't on the cited page.

Can a model grade the answers?

Yes, with care. OpenAI's guide lists position and verbosity bias as known problems with model judges and says a judge should show it "consistently agrees with human annotations" before you scale it (OpenAI docs). Anthropic's examples suggest using a different model to grade than the one that generated the answer (Anthropic docs). Keep the grading prompt fixed so runs compare, and review a sample by hand each time.

What should a run report show?

One row per metric, split by tag, compared with the last accepted run. A report like this is enough to make a decision:

Metric	Layer	How it's computed
Hit rate at k	Retrieval	Questions where a gold passage is in the top k, divided by all questions
First correct rank	Retrieval	Average rank of the first gold passage
Leak count	Retrieval	Permission-scoped questions where another user's passage appeared (target: zero)
Correct	Answer	Answers matching the reference
Citation pass	Answer	Answers whose quotes are found in the cited passage
Refusal	Answer	Out-of-scope questions correctly declined
Tokens and latency	Cost	Per question, per run

Notice the cost row. A change that improves answers but triples the tokens is a decision for a person, not an automatic win.

How do you keep the set running on every change?

Treat it like a test suite. Store the questions and gold data in the repository, versioned with the code. Run the set in CI, or on a schedule, whenever chunking, embeddings, prompts, retrieval settings, or the model change. Save each run's results and compare against the last accepted run.

Then decide what blocks a release. A drop in hit rate on a key tag, or any leak at all, should stop the deploy or at least require a named person to sign off. Without that rule, the report becomes a chart nobody reads.

How does the set stay useful over time?

Every bad answer a user reports becomes a new test. Add it with gold passages and a reference answer, then fix the system until it passes. Over months the set turns into a record of every way the system has failed, and a guard against those failures coming back.

Review it every quarter. Remove questions about documents that no longer exist, and add new document types as they arrive. Is the set still shaped like real traffic? If users now ask mostly about one document type, the tags should show it. And if you're still deciding between retrieval and training a model, our RAG vs fine-tuning page covers that choice.

FAQ

How long does it take to build a first RAG evaluation set?

Labelling is the slow part. A domain expert who knows the documents can usually find gold passages and write reference answers for fifty questions in a few focused sessions. Grading code takes less time than the labelling, and the set then runs automatically on every change, so most of the cost is paid once.

Should the evaluation set use synthetic questions?

Use them to fill gaps, not as the base. Generated questions tend to mirror the document's wording, which makes retrieval look better than it is with real users. Start from tickets, logs, and interviews, then add synthetic questions for document types or edge cases that real traffic hasn't reached yet.

What's a good hit rate at k?

There's no universal number. It depends on k, your documents, and how strict your gold passages are. What matters is the trend per tag between runs. Set your own release threshold from the first accepted baseline, and treat any permission leak as a failure regardless of the overall score.

Do we need a separate set for each language?

If users ask in more than one language, yes, tag questions by language at minimum. Embedding models and chunking can behave differently across languages, and OpenAI's guide lists non-English input as an edge case to plan for. Keep the gold passages the same where the source document is shared.

Planning a RAG system, or not sure whether yours answers correctly? See our RAG development work and start with an evaluation of the setup you already have.