What happens under the hood
In RAG, a system retrieves relevant passages from an external collection and passes them to the model with the question. An assistant can thus use company documents without necessarily retraining. Answer quality depends both on retrieval and on how the model interprets the material.
Why retrieval matters more than elegant prose
Suppose an employee asks whether leave can be moved after approval. The collection contains the current policy, last year's version and discussion of proposed changes. If search retrieves an obsolete document, the model may produce a logical, polished, incorrect answer. Evaluate RAG in stages: available documents, retrieved passages and conclusions drawn from them.
Organise sources before connecting a model: assign document owners, retain effective dates and distinguish approved text from drafts. Then test searches with ordinary user wording. People may use informal terms for leave or internal forms. Good retrieval must recognise meaning while preserving the precision of numbers, names and exceptions.
Hidden sources of error
Documents are normally divided into passages so the model receives relevant context. If a condition is in one passage and its exception in the next, an answer may lose a critical qualification. Test questions needing several parts of a text: who may act, under which conditions and who confirms it. One correct keyword match is insufficient.
Access rights form a separate boundary. Search must respect which documents the particular user may see. Hiding a link does not solve the problem if its content already appears in the answer. Test with two roles: the same question should reveal only permitted information. If no authorised source exists, acknowledging insufficient data can be correct.
How to accept such a system
Prepare questions with known supporting documents. Check the answer, retrieved passages, version currency and fidelity to the cited meaning. Include unanswered questions and conflicting sources. The system should expose uncertainty instead of filling gaps with plausible explanations. This is part of quality, not evidence of a failed implementation.
