RAG (Retrieval-Augmented Generation) is a method in which an AI assistant first searches for information in the company's documents and only then formulates its answer, citing the source. This way the assistant answers from the company's procedures, contracts or technical data sheets, not from the model's general knowledge.
Many managers want an AI assistant that can answer questions about the company's procedures or products. An ordinary language model doesn't know your internal documents and, when it doesn't know, it may produce an answer that sounds convincing but is wrong. RAG is one of the most widely used approaches to this problem. Below we explain how it works, what you can realistically expect from it and what to prepare before a project.
In short
RAG = searching your documents + an answer generated from the passages found + citing the source. It works best with up-to-date, well-organised documents, with permissions respected and with a set of test questions. It reduces errors, but for important decisions a person checks the cited source.
How RAG works: search first, then answer
RAG stands for “Retrieval-Augmented Generation”, meaning answer generation supported by search. Instead of answering from its general “memory”, the assistant first searches for the relevant passages in the company's documents and only then formulates the answer based on them.
It resembles the way a conscientious colleague works. When they get a question they are not sure about, they open the procedure, find the right paragraph and base their answer on it.
- The documents are prepared: split into passages and indexed so they can be found quickly by meaning, not just by exact words.
- The user asks a question in natural language.
- The system searches for the passages most relevant to the question.
- The language model receives the question together with the passages found and formulates the answer.
- The answer is displayed with references to the documents used.
Why not simply give the model all the documents at once? Because, in a company, the volume of documents is usually too large for a single question, and cost and response time would rise needlessly. Also, the more irrelevant text it receives, the greater the risk that the model latches onto the wrong information. Searching first gives it only what matters for that question.
Why citations matter in a RAG assistant
A good RAG assistant doesn't just say “here is the answer”, but also “here is where I got it from”. The citation can be the document name, the section or a direct link to the relevant page.
For a business, the citation is the difference between an answer you have to take on trust and one you can check in a few seconds. It is especially useful when the answer underpins a decision, a communication to a customer or a financial operation.
A conceptual example: a salesperson asks what the warranty period is for a particular product. The assistant answers and points to the technical data sheet and the section it took the information from. If the salesperson has any doubt, they open the document and confirm before passing the answer on to the customer.
The limits of RAG: it reduces errors but doesn't eliminate them
RAG significantly reduces the risk of the assistant making things up, because it forces it to start from real sources. Even so, it doesn't make the system infallible, and this needs to be made clear to anyone who uses it.
- The search may return an unsuitable passage, and the answer will be wrong even though it has a citation.
- The model may misinterpret a correct passage or combine information from different contexts.
- If two documents contradict each other, the system cannot tell on its own which version is valid.
- Complex tables, scanned documents or images may be read incompletely.
Important limitation
An answer with a citation is not automatically a correct answer. For important legal, financial or medical decisions, a person checks the cited source directly before acting.
Document quality determines answer quality
A RAG assistant is only as good as the documents it receives. If procedures are old, duplicated or contradictory, the assistant will reflect exactly that disorder.
- Delete or archive outdated versions before indexing.
- Decide which document is the official source when there are several versions.
- Prefer documents with clear titles, sections and update dates.
- Check scanned documents: the text must be extractable correctly.
- Note who is responsible for each category of documents.
The good news is that this effort pays off beyond the AI project too. A well-organised document base helps people, not just the assistant.
Permissions: who sees which documents
In a company, not everyone has access to every document. Contracts, salaries or customer data are usually restricted. A RAG assistant must follow the same rules.
In concrete terms, the system checks who is asking the question and searches only the documents that person already has access to. Otherwise the assistant becomes a shortcut through which someone can learn information they shouldn't see. This is settled at the start of the project, not added at the end.
It is also worth deciding what happens to the questions put to the assistant. They may contain sensitive information themselves, so it must be clear where they are stored, for how long and who can view them. For some categories of documents it may be wiser not to include them at all, at least in the first phase.
Updating sources and missing information
Company documents change: new procedures appear, prices change, contracts are updated. The assistant's index must be updated along with them, automatically or on a clear schedule, and withdrawn documents must be removed from the index.
Just as important is what the assistant does when it can't find the information. The right behaviour is to say openly that it has no source for that question and to point the user to someone they can ask. An assistant that “fills in” gaps from imagination is more dangerous than one that admits it doesn't know.
This behaviour doesn't happen by itself. It is configured explicitly through the instructions given to the model and tested with questions you know for certain the documents cannot answer.
Practical tip
Keep a list of the questions the assistant couldn't find an answer to. It shows you exactly which documents are missing or need clarifying and, over time, becomes a tool for improving your internal documentation.
How to evaluate a RAG assistant
Evaluation isn't based on impressions after a few questions asked during a demo. You need a set of real questions with known correct answers, which you can run again after every change.
- Accuracy: does the answer match the official document?
- Citation: does the cited source really contain the information in the answer?
- Coverage: how many questions in the test set get a useful answer?
- Correct refusal: does the assistant admit when it doesn't have the information, instead of making it up?
- Permissions: does a user without access get no information from restricted documents?
- Feedback: can users easily flag an answer as wrong, and are the flags reviewed regularly?
The test set should be built together with people who know the subject, because they know which questions are common and where the pitfalls are. Run the set every time you add new documents, change the search method or switch to another language model, so you notice early if an improvement in one area has broken something else.
How we start a RAG project
At WEB-AZ we start with a small set of well-defined documents and the team's real questions. Only once the assistant answers correctly and with citations on this set do we extend the sources. If you want to see whether your documents are ready for such an assistant, an initial assessment is a good starting point.
Next steps