RAG explained: how an AI assistant answers from your own documents

Why a plain chatbot is not enough

Large language models learn from public text. They write fluently, but they have never seen your price list, your leave policy or last month's product update. Ask them about those and they will often produce a confident answer that is simply wrong. For a business, a wrong answer delivered confidently is worse than no answer.

RAG fixes this by giving the model the right material at the moment a question is asked, and telling it to use only that material.

How RAG works, in four steps

1. Split your documents into passages. Web pages, PDFs and manuals are cut into short pieces of a few paragraphs each, and each piece keeps a note of where it came from.
2. Turn each passage into numbers. An embedding model converts each passage into a list of numbers that captures its meaning, so passages about the same thing end up close together. These are stored in a vector database.
3. Find the closest passages. When someone asks a question, the question is converted the same way, and the system pulls out the handful of passages closest in meaning.
4. Answer from those passages only. The model receives the question plus those passages, with a firm instruction: answer from this material, cite where it came from, and say so if the answer is not there.

What we have learned building them

Clean sources matter more than the model

Website pages repeat the same menu, footer and cookie notice on every page. If you feed that in unfiltered, the search keeps finding the menu instead of the real content. Stripping repeated text before indexing made a bigger difference to answer quality in our projects than switching models.

Show the source with every answer

Every answer should link to the page, or give the page number in a PDF, that it came from. Users trust answers they can check, and your team can spot when a source document itself is out of date.

Teach it to say "I don't know"

A good assistant hands over to a person when the documents do not cover a question, for example with a WhatsApp or callback button. One honest "I don't know" protects trust far more than ten clever answers build it.

Keep the index fresh

Content changes. Rebuild the index on a schedule, such as every night, so new pages and edits flow into the answers automatically.

Size the passages to the content

Passages that are too short lose context, and passages that are too long bury the answer. A few paragraphs, with a small overlap between neighbouring passages, works well for most business documents.

What it costs to run

The embedding step can run on an ordinary server using open-source models, so it costs little beyond hosting. A few thousand pages of documents fit comfortably in a small vector database. The main running cost is the language model call for each question, which depends on the model you choose and how many questions you get. Open-source models served by low-cost providers keep this small for most businesses.

When RAG is not the answer

• Live numbers and calculations. Stock levels, account balances or prices that change by the minute should come from your systems through a direct connection, not from documents.
• A handful of fixed answers. If you only have ten common questions, a well-written FAQ page may serve customers better than an assistant.
• Decisions. RAG helps someone find and understand information. Approving, rejecting or paying should stay with rules and people.

We have built RAG assistants over websites with hundreds of articles and over product manuals running to thousands of pages. You can read about them in our case studies.