Indexing as the grounding foundation
RAG begins long before query time: documents are processed into vector embeddings and stored in a retrieval index. Each embedding captures semantic meaning so that a query about 'power consumption in manufacturing' can find documents on 'industrial energy usage' even without exact word overlap. The index is typically built offline, allowing fast nearest-neighbor lookup at query time.
Embedding quality directly controls retrieval precision. A weak embedding model may retrieve semantically unrelated documents; a strong one narrows results to genuinely relevant passages. Many RAG systems use pretrained embeddings (e.g., from OpenAI or open-source models) because retraining embeddings for every domain is expensive.
Query retrieval and scoring
When a user queries, the system embeds the query in the same space as the indexed documents and retrieves the top-k most similar passages (often top-5 to top-20). The score reflects similarity, not truth; a high-scoring result is merely 'closest to the query', not necessarily accurate or current.
This top-k subset becomes the LLM's context window, replacing a larger knowledge base with focused passages. The LLM can then synthesize these specific sources rather than attempting retrieval from its training data, reducing hallucination.
Why LLMs ground better with retrieved context
LLMs trained only on text can confabulate facts from statistical patterns in their training set. When given retrieved passages, the LLM's response is constrained to what's actually written, making it safer for sensitive domains like law, medicine, or customer support. The LLM acts as a synthesis engine, not a knowledge base.