Overview
Retrieval-Augmented Generation (RAG) combines information retrieval with language model generation to produce responses grounded in specific source documents. Instead of relying solely on a model’s parametric knowledge (which can be outdated or hallucinated), RAG retrieves relevant documents from an external corpus and includes them in the model’s context window. This architecture reduces hallucination, enables real-time knowledge updates without retraining, and provides citations for generated claims. RAG has become the dominant pattern for building knowledge-intensive AI applications over proprietary or frequently changing data.
Key Concepts
- Indexing pipeline — Documents are chunked, embedded into vector representations, and stored in a vector database for semantic search.
- Retrieval — Given a query, the system finds the most semantically similar document chunks using embedding similarity (cosine distance or dot product).
- Chunking strategies — How documents are split affects retrieval quality. Options include fixed-size, sentence-based, semantic, and recursive splitting.
- Reranking — A second-pass model scores retrieved chunks for relevance, improving precision over raw embedding similarity.
- Hybrid search — Combining semantic (vector) search with keyword (BM25) search for better recall across different query types.
- Context window stuffing — Inserting retrieved chunks into the model prompt. Order and formatting affect how well the model uses the information.
Trade-offs
- Retrieval quality directly limits generation quality (garbage in, garbage out)
- Chunking granularity trades recall (smaller chunks) against context (larger chunks)
- Adding retrieval increases latency compared to direct generation
- Embedding models must be chosen to match the domain vocabulary
Related
- Vector Databases provide the storage and search layer for RAG systems
- Embeddings are the representation format that enables semantic retrieval
- Context Windows constrain how much retrieved content can be used