Retrieval‑Augmented Generation (RAG)

Retrieval‑augmented generation (RAG) is a technique that enables large language models (LLMs) to retrieve and incorporate new information from external data sources.^[1] With RAG, LLMs first refer to a specified set of documents, then respond to user queries. These documents supplement information from the LLM's pre‑existing training data.^[2] This allows LLMs to use domain‑specific and/or updated information that is not available in the training data.^[2] For example, this enables LLM‑based chatbots to access internal company data or generate responses based on authoritative sources.

Why RAG improves LLMs

RAG improves LLMs by incorporating information retrieval before generating responses.^[3] Unlike LLMs that rely on static training data, RAG pulls relevant text from databases, uploaded documents, or web sources.^[1] According to Ars Technica, "RAG is a way of improving LLM performance, in essence by blending the LLM process with a web search or other document look‑up process to help LLMs stick to the facts." This method helps reduce AI hallucinations,^[3] which have caused chatbots to describe policies that don't exist or recommend nonexistent legal cases to lawyers seeking citations.^[4]

Limitations of RAG

While RAG reduces the need to retrain LLMs with new data, saving computational and financial costs,^[1] it does not solve all problems. LLMs can still generate misinformation even when pulling from factually correct sources if they misinterpret the context. MIT Technology Review gives the example of an AI‑generated response stating, "The United States has had one Muslim president, Barack Hussein Obama," a misinterpretation of a title.^[2] Additionally, RAG does not guarantee that models will recognize when they lack sufficient information, leading to confident but incorrect answers.^[1]

Process

Retrieval‑augmented generation enhances LLMs by adding an information‑retrieval mechanism. First, raw data is converted into embeddings and stored in a vector database. Given a user query, a retriever selects the most relevant documents, which are then used to augment the original query in the prompt. The LLM generates its answer based on both the query and the retrieved documents. Optional steps such as re‑ranking, context selection, and fine‑tuning can further improve output quality.^[2][3]

Key Stages

• Encoding: Convert text to dense or sparse vectors and store them in a vector store.
• Retrieval: Use similarity search (e.g., ANN) to fetch relevant documents.
• Reranking: Refine results with models like RankGPT.
• Generation: Combine retrieved passages with the user query and feed them to the LLM.

Applications

RAG is used wherever generated responses need to be grounded in up‑to‑date information, such as internal company chatbots, healthcare assistants, and legal advisory tools. By grounding answers in external sources, RAG improves factual accuracy and allows citations for verification.