Evaluating Retrieval Quality in RAG Systems
An internal research note on why similarity search alone is a weak proxy for retrieval quality, and what we're exploring instead.
The Default Metric Is a Proxy, Not the Goal
Most retrieval-augmented generation systems are built around a single metric: cosine similarity between a query embedding and a set of document embeddings. That metric is convenient and cheap to compute, but it's a proxy for what actually matters — whether the retrieved content lets the model produce a correct, complete answer. Those two things correlate, but not as tightly as the simplicity of the metric suggests.
Where Similarity Search Quietly Fails
Similarity search struggles with queries that require combining information from multiple documents, with questions phrased differently than the source material, and with distinguishing a document that's topically similar from one that actually answers the question. A document about a company's general refund policy can be highly similar, embedding-wise, to a query about a specific refund exception — while being the wrong document to retrieve.
What We're Exploring Instead
Hybrid retrieval — combining semantic similarity with keyword-based search — recovers cases where an exact term match matters more than semantic closeness. Re-ranking a larger initial candidate set with a separate, more expensive model narrows a broad recall-oriented first pass down to genuinely relevant results. Structured metadata filtering, applied before semantic search runs, cuts out an entire category of same-topic-wrong-document failures without needing better embeddings at all.
Evaluation Has to Measure the Right Thing
Evaluating retrieval quality separately from generation quality is a discipline in its own right: building a labeled set of queries with known correct source documents, and measuring whether retrieval surfaces them — independent of how well the language model writes from what it's given. Conflating the two makes it impossible to tell whether a bad answer came from bad retrieval or a bad generation step, which makes the failure much harder to fix.
An Open Question We're Still Working Through
The harder unsolved problem is measuring retrieval quality on a knowledge base that changes constantly, where yesterday's labeled evaluation set may not reflect today's content. This is an active area of exploration rather than a solved one — closer to ongoing research than a shipped best practice, which is exactly why it's a research note and not a how-to.