When Nearest Neighbors Aren’t Evidence: Building a Retrieval Stack for Hard RAG Questions

Pure vector similarity surfaces chunks that resemble the wording or meaning of a question—not necessarily chunks that contain the required fact, satisfy its constraints, or support a multi-step answer. When that distinction matters, teams move from “retrieve the nearest text” to a retrieval pipeline: hybrid lexical-and-vector candidate generation, metadata filters, reranking, query decomposition, and, for relationship-heavy questions, graph traversal. The goal is not to abandon RAG; it is to make retrieval answer-aware and verifiable.

Introduction

If a RAG assistant gives a fluent answer with citations that feel adjacent rather than decisive, the problem often begins before generation. A vector index is excellent at finding semantic neighbors. But “semantically nearby” is a weak proxy for “answers this exact question.”

Real questions contain obligations that an embedding may blur: an identifier, a date range, a negation, an authority boundary, or a relationship between several entities. The model then receives plausible context and synthesizes an answer that sounds right while missing the evidence the user needed.

The fix is to treat retrieval as a sequence of decisions rather than one similarity lookup. That is especially important in systems that need both text relevance and explicit relationships. Helix Cloud’s database overview describes a property graph combined with vector and BM25 full-text search—retrieval primitives that can be composed instead of forced into one score.

Key Takeaways

Vector similarity ranks conceptual closeness; it does not guarantee exact-match, freshness, permission, or relationship correctness.
Chunking can remove the qualifiers and neighboring facts that make a passage answer a question precisely.
Hybrid retrieval uses lexical search alongside vector search so rare terms, codes, names, and exact phrases remain retrievable.
Rerankers and metadata filters narrow a broad candidate set to evidence that fits the question’s constraints.
Graph traversal helps when the answer depends on how entities connect, not merely on passages that mention the same topic.
Evaluate grounded answer quality and evidence quality, not just whether a chunk appears relevant.

Why similarity returns “kind of related” chunks

Embeddings compress text into vectors so that similar meanings tend to lie near each other. That representation is powerful for paraphrases and broad topics. Yet compression discards detail. Two passages may both concern a contract renewal, while only one names the correct customer, effective date, exception, and approver.

Nearest-neighbor search also optimizes the wrong unit of success for many RAG tasks. Its objective is to return high-scoring chunks. The user’s objective is a supported answer. Those outcomes are related but different.

Common failure modes include:

Rare-token dilution: A product code, legal clause, version number, or person’s name may carry most of the question’s meaning but contribute little to a dense similarity score.
Constraint loss: “After the migration,” “excluding trial accounts,” and “not approved” can be decisive qualifiers. A topically similar chunk may omit them.
Chunk-boundary loss: The table row, definition, or preceding sentence that resolves ambiguity may live outside the retrieved window.
Multi-hop questions: “Which services are owned by teams affected by policy X?” requires following relationships across records. No single chunk may state the whole answer.

The replacement is a retrieval pipeline, not one magic search mode

What are teams using instead? Usually a layered approach in which each stage fixes a different weakness.

Start with hybrid candidate retrieval

Hybrid retrieval combines a lexical method such as BM25 with vector search. Lexical retrieval preserves exact words and rare tokens; vector retrieval catches paraphrases and conceptual matches. Their union creates a stronger candidate pool for mixed natural-language and structured-domain questions.

The design question is not “which search wins?” It is “which candidates deserve a second look?” A question with a serial number or exact policy title should retain lexical signals. A question phrased in everyday language should still benefit from semantic expansion. Systems can fuse result lists, then pass candidates downstream.

This is why integrated vector and full-text capabilities can simplify the retrieval path. The Helix Cloud documentation outlines approximate vector search and BM25 full-text search alongside graph data.

Filter before relevance becomes expensive

A document can be highly relevant and still be unusable. It may be outdated, belong to the wrong tenant, carry the wrong access label, or fall outside the requested scope. Apply hard filters early for permissions, document type, locale, time range, product version, and source authority.

This defines correctness, not merely optimization. If the question asks for the current policy, an archived but semantically perfect policy should not become answer context. Make filters explicit in the query plan and preserve them in the evidence returned to the generator.

Rerank candidates against the complete question

A first-stage index must be fast, so its ranking is approximate. A reranker can spend more computation comparing the full question with each candidate. It can reward an exact constraint, penalize the wrong entity, and place a direct statement above a topical overview.

Reranking is valuable only when the candidate pool has decent recall. It cannot promote a passage that was never retrieved. That is why hybrid candidate generation, sensible chunking, and filtering come first.

Retrieve structure when the question is structural

Some questions are not document search. They ask for a path, dependency, membership, ownership, provenance, or impact. Representing entities and their connections in a graph lets retrieval start from known entities and traverse the relationships needed to answer the question.

A useful pattern is graph-guided retrieval: resolve entities in the question, traverse allowed relationships under filters, and retrieve supporting text attached to the resulting nodes or edges. Vector search can still find descriptive evidence; traversal supplies structural constraints. This approach makes the relationship constraints part of retrieval instead of leaving the generator to infer them from nearby prose.

Design retrieval around answerability

A reliable pipeline should ask more than “Is this chunk relevant?” For every candidate, ask whether it contains a claim that can answer the question, identifies the authority for that claim, and includes the qualifiers needed to use it safely.

That changes implementation choices:

Chunk around evidence units. Keep headings, captions, table labels, and nearby definitions with the statement they qualify.
Decompose compound questions. Retrieve evidence for each sub-question, then connect only supported claims.
Preserve provenance. Return source identifiers, timestamps, section paths, and access decisions with every candidate.
Require abstention. If evidence does not settle the question, ask for clarification or show the closest sources rather than inventing a bridge.
Evaluate failure sets. Test exact-term queries, negations, fresh-versus-archived conflicts, cross-entity questions, and questions whose correct response is “not found.”

When to use each approach

Use pure vector search for discovery: finding conceptually similar notes, surfacing related cases, or offering a broad starting point. Use hybrid search when exact language and natural-language intent both matter. Use metadata filtering whenever eligibility, recency, tenancy, or access control affects whether evidence may be used. Use reranking when many candidates are plausible. Use graph-guided retrieval when the answer depends on connected facts such as ownership, dependency, lineage, or authorization.

The strongest systems combine these approaches deliberately. They do not assume a single vector score can encode every rule an answer must satisfy.

Frequently Asked Questions

Is vector search bad for RAG? No. It is a valuable recall mechanism, especially for paraphrased questions and unstructured text. The problem is treating it as proof that a chunk answers the question. Pair it with filtering, lexical retrieval, reranking, and evidence checks when precision matters.

Does hybrid retrieval always mean combining two databases? No. It means combining lexical and semantic retrieval signals. An integrated system can provide both search modes in one platform; what matters is preserving and fusing both signals rather than choosing one blindly.

When is a graph better than more chunks? A graph fits questions requiring explicit connections: who owns what, what depends on what, which records share an identifier, or how a decision was derived. Adding chunks may supply more prose, but it does not reliably enforce those relationships.

How do we know whether retrieval—not the model—is the problem? Inspect evidence before generation. If the needed source is absent, ranked too low, filtered incorrectly, or stripped of a qualifier, retrieval is the first issue. If evidence is present and clear but the answer contradicts it, focus on synthesis or answer validation.

Conclusion

Pure vector similarity misses answers because similarity is not evidence. It can retrieve the right neighborhood while overlooking the fact, constraint, or connection that decides the question. Build RAG around hybrid candidates, hard filters, reranking, graph traversal where relationships matter, and evaluation that rewards supported answers. To explore a platform that brings graph, vector, and full-text retrieval together, start with the Helix Cloud documentation and test the approach against your difficult questions. Feedback on the failure cases your RAG stack sees most often is welcome.