Your LLM doesn’t know what happened yesterday.
It doesn’t know your company’s product catalog. It doesn’t know last month’s pricing changes. And when you ask, it won’t always tell you that, it’ll just make something up that sounds right.
You could fine-tune the model on your data. But fine-tuning is expensive, slow, and goes stale every time something changes.
RAG (Retrieval-Augmented Generation) solves this differently. Instead of teaching the model new facts, you give it the right facts at the right time, on every request.
This is the version of “what is RAG” I wish someone had given me when I started building AI features into real applications.
The Problem RAG Solves
Two things break LLMs in production.
The first is the knowledge cutoff.
The model only knows what existed in its training data. Anything that happened after that is invisible to it. Ask about a product launched last month and you get silence, or worse, a polite invention.
The second is hallucination. When asked about something it doesn’t know, the model doesn’t shrug. It fills the gap with plausible-sounding text. The answer reads well. The facts are wrong. And users tend to trust whatever sounds confident.
For tasks like “summarize today’s news” or “what’s our refund policy”, a vanilla LLM fails one way or another. It either refuses, guesses, or invents.
RAG fixes both problems with a single idea: don’t make the model remember anything. Let it look things up.
What Is RAG?
RAG is an architecture pattern, not a model.
You keep the LLM exactly as it is. Around it, you build a retrieval layer that fetches relevant context from your own data, documents, databases, wikis, APIs, and ships that context to the model alongside the user’s question.
The model now answers using your facts, not its memory.
The clearest analogy I’ve heard: closed-book exam vs. open-book exam. Same student, same brain. But with the textbook open on the desk, the answers stop being a memory test and start being a reasoning task. That’s exactly what RAG does to the LLM.
Where RAG Came From
The term was coined in 2020, in a paper called Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Patrick Lewis and a team at Facebook AI Research, University College London, and NYU. The paper described a model that combined two kinds of memory: the parametric memory baked into the LLM’s weights, and a non-parametric memory, an external index the model could query at runtime.
That distinction is the whole insight.
Parametric memory is everything the model “knows” because of how it was trained. It’s compressed, frozen, and impossible to update without retraining. Non-parametric memory is anything you can change, search, and swap out without touching the model at all. RAG is what happens when you connect the two.
Five years later, Lewis’s name for the technique stuck, even though, as he’s joked, “RAG” is not a great acronym. The pattern is now used in hundreds of papers and almost every commercial AI system that needs to be both fluent and factual.
How RAG Works
A RAG pipeline has three stages:
indexing, retrieval, and generation.
Almost everything you’ll ever read about RAG is some variation of these three.
Indexing
Indexing happens before any user asks anything. You take your knowledge base, PDFs, articles, database rows, support tickets, anything textual, and break it into smaller pieces called chunks.
Each chunk is then converted into a vector through an embedding model like OpenAI’s text-embedding-3, Cohere’s embed-v3, or open-source options like the sentence-transformers family. A vector is just a list of numbers, usually somewhere between 384 and 3,072 of them, that represents the meaning of the chunk in a way machines can compare.
These vectors live in a vector database like Qdrant, Pinecone, or a normal dg using extensions or vector columns. You only run indexing when your data changes. The output is a searchable index where things that mean similar things are physically close to each other.
Retrieval
When a user asks a question, you embed the question with the same model and search the vector database for the chunks closest in meaning. Closeness is usually measured by cosine similarity.
This is the part that surprises people coming from keyword search. The user can ask “Easter” and the system can return a chunk that talks about “rabbit”, because the meanings are close, even with no overlapping words.
That sounds magical until you realize it’s also the most common point of failure. If the wrong chunks come back, no LLM in the world can save the answer. Retrieval is where most RAG systems quietly break.
That’s why production teams almost never rely on vector search alone. The current best practice is hybrid search, running vector search alongside a classic keyword algorithm like BM25 and merging the results, often with a step called reciprocal rank fusion. Vector search catches semantic matches. BM25 catches exact terms, error codes, product IDs, version numbers, that embeddings tend to miss. Combined, they cover each other’s blind spots. Many teams add a reranker (a small cross-encoder model) on top to reorder the final shortlist.
Generation
You take the retrieved chunks, paste them into the prompt as context, and call the LLM. The instruction is usually “answer the question using only the material below”. Some teams add stricter rules, “if the context doesn’t contain the answer, say so“, to keep the model from filling gaps with imagination.
The model now answers grounded in your data. Hallucinations drop. Updates are instant, change the source documents, and the next query reflects the new reality. And because every answer can cite the chunks it came from, you finally get something LLMs almost never give you: traceability.
That’s the whole loop.
Where People Are Actually Using This
Financial services use RAG for compliance and regulatory analysis. Rules change constantly. Fine-tuning a model every time a regulator updates a document is impossible. Indexing the new document into a vector store and letting the LLM reason over it takes minutes.
Legal and healthcare are the highest-stakes domains. Lawyers use RAG to query case law and internal contracts. Hospitals use it for evidence-based clinical recommendations grounded in medical literature.
Customer support is the most obvious case. Support documentation before generating any reply, combined RAG with a knowledge graph to built from historical issue tickets.
Marketing and Media, generating automated posts, metrics analysis, creating sales pages.
There are many more…
Conclusion
RAG is the most practical way to ground an LLM in your own data without retraining anything.
The core pattern is simple:
- Index your knowledge base into a vector store.
- Retrieve the most relevant chunks for each user query.
- Generate the answer with the LLM using those chunks as context.
Get the indexing right, the retrieval right, and the prompt right, in that order. The LLM is the easy part.
That’s all you need to start shipping RAG to production.
