What is RAG (Retrieval-Augmented Generation)?

RAG (Retrieval-Augmented Generation) is an architecture that connects a language model to an external data source —documents, databases, or APIs— so its responses rely not only on what it learned during training, but on up-to-date, domain-specific information. Instead of asking "what does the model know?", you ask "what does our knowledge base say?"

Pro Tip: RAG is the most practical way to bring an LLM to production with private or current information without retraining the model. If your company has manuals, FAQs, or internal documentation, RAG is your best starting point.

What is it

RAG describes a two-phase flow:

1. Retrieval: when a user asks a question, the system searches a document collection for the most relevant fragments using similarity search (embeddings).
2. Generation: those fragments are injected as additional context into the LLM prompt, which generates an answer grounded in that evidence.

The key idea is that the model does not need to store the answer. It only needs to read and reason about the information you just gave it.

Note: RAG does not modify the model. It is not fine-tuning. The LLM stays the same; what changes is what you put in the prompt.

Mental model

Imagine you are a student taking an open-book exam. Without RAG, the student answers only from memory (bare model). With RAG, the student has the book right there and can consult it for each answer. Same student, but now answers are far more accurate because they cite the source.

The components of a RAG system:

• Source documents: PDFs, web pages, wikis, databases, any text.
• Chunker: splits documents into manageable fragments (chunks).
• Embedding model: converts each chunk into a numerical vector representing its meaning.
• Vector database: stores the vectors and enables similarity search (e.g., Pinecone, Chroma, Qdrant, pgvector).
• LLM: the language model that receives the question + retrieved chunks and generates the final answer.

How it's used

Implementing RAG follows two phases: indexing (one-time setup) and query (each interaction).

Phase 1: Indexing (setup)

1. Collect the documents you want to use as your knowledge source.
2. Split each document into chunks of 256–1024 tokens, depending on content type. Use 20–50 token overlap between chunks to avoid losing context at boundaries.
3. Generate an embedding for each chunk using a model like text-embedding-3-small (OpenAI) or BGE (open source).
4. Store the vectors in a vector database alongside the original text and metadata (source, page, date).

Warning: Chunk size matters. Large chunks dilute relevance; small chunks lose context. Start with 512 tokens and adjust based on results.

Phase 2: Query (inference)

1. The user writes a question.
2. The system generates the embedding of the question using the same model used for indexing.
3. It searches the vector database for the top_k most similar chunks (typically 3–10).
4. It builds a prompt with the question and the retrieved chunks as context.
5. It sends the prompt to the LLM and returns the answer to the user.

Example RAG prompt:
 
Context:
---start fragment 1---
[retrieved document 1]
---end fragment 1---
---start fragment 2---
[retrieved document 2]
---end fragment 2---
 
Instruction: Answer the question using ONLY the information in the context provided. If the context does not contain the answer, say "I don't have enough information to answer."
 
Question: What is the refund procedure?

Pro Tip: Always include a "don't hallucinate" instruction in the prompt. Without it, the model may ignore the context and answer from its training knowledge, which is exactly what RAG aims to prevent.

When to use it / when not to

Use it when:

• You need answers grounded in private or frequently updated documentation
• Your users ask questions about a specific domain (legal, medical, technical, products)
• You want to reduce hallucinations in a production system
• You cannot or do not want to fine-tune the model
• Source data changes often and you need to reflect changes without retraining

Do not use it when:

• The question is always the same and a static answer suffices (a simple FAQ solves the problem with less complexity)
• You have a single small document that fits entirely in the prompt (no retrieval needed)
• Latency is critical and you cannot afford the extra vector search time
• The model already knows the answer with sufficient accuracy (e.g., very general questions about pop culture)

Pro Tip: RAG and fine-tuning are not mutually exclusive. Many teams use fine-tuning to teach the model the desired format and tone, and RAG to provide the specific factual context. They are complementary.