How do you trust an assistant that might confidently invent facts? To solve this, we need to change how the AI accesses information. We need to move from a memory test to an open-book exam.
The Hallucination Problem: The Closed-Book Exam
To understand why standard AI fails in a business context, it helps to look at where its answers come from. A standard AI generates answers from patterns learned in its training data. Because it relies purely on its memory, it acts like a student taking a closed-book exam.
A closed-book exam tests what you memorized, and you'll naturally guess the answers you don't know. This creates a significant business risk because the AI is confident even when it is wrong, and this is exactly where hallucinations come from. Furthermore, an out-of-the-box model has never seen your private contracts or specific prices.
The RAG Solution: The Open-Book Exam
RAG stands for Retrieval-Augmented Generation. The RAG mental model is an open-book exam.
An open-book exam lets you find the answers in your books rather than relying on memorization. RAG works by handing the AI your specific books before it attempts to answer your question.
The simple version of what happens when you ask a RAG-enabled assistant a question breaks down into a four-step loop:
Ask: You ask a question in plain language.
Retrieve: The system finds the most relevant pages in your files.
Link: Those pages are attached to your question.
Answer: The AI replies using the attached pages.
Why RAG Matters for Business Outputs
When an AI reads the relevant files first and then answers, the results are fundamentally different. It pulls figures straight from the source, reducing human errors, and ensures every number is traceable to the file it came from. Because it can cite the exact source for every figure, review becomes much easier.
Furthermore, RAG minimizes hallucination by allowing the AI to admit when it lacks information. As shared in Red Airship’s recent AI workshop, many times AI models hallucinate because you are forcing them to give you something, but you are not telling them how. With a proper RAG prompt, you are giving it an avenue to say, "I'm not sure".
The Limits of the Open-Book Exam: Context Budget
While an open-book exam is a massive upgrade, there is a fundamental physical limitation: your desk only has so much space. In AI terminology, this is your "context budget".
If it can fit, it doesn't mean it should fit. Even if you are lucky and your documents fit into a 1 million token context window, the AI will still lose attention. Attention is the strongest at the start and at the end. The middle 40%, some say 60%, is prone to hallucination or not finding info at all.
If you put a whole document into the context when you don't actually need to, you are introducing noise. By giving it one line of important data, and everything else is noise, the signal-to-noise ratio becomes much less, and you are confusing the AI. Furthermore, you don't want to just burn tokens by filling up the whole document in your context. You want to only feed the relevant information, relevant parts of the document, not the whole document.
How Data Ingestion Solves the Problem
To ensure we only feed the AI relevant information, we use a process called data ingestion. During ingestion, we read the documents, we break them down into small chunks, we convert them into numbers, and then we store them into a database.
1. Chunking to Reduce Noise
Why do you want to break a document down into pieces? Let's say you are doing e-commerce and you have a price list with 10,000 SKUs. Your query is just checking the price of one product. Without chunking, the retrieval mechanism will put the entire 10,000 SKU document into the context window. Immediately, a bulk of your context window is gone, and all those extra SKUs are noise that will confuse your AI. This is why we want to break documents down into smaller sections.
2. Embedding to Create a Semantic Space
After chunking, we perform embedding. Embedding basically means taking a chunk of text and converting it to meaning in terms of numbers. Embedding is converting text into numbers in a semantic space.
To form a mental model of what it means to be "semantically similar," let's reduce this to a semantic space described with two numbers: calories and protein.
Where is KFC fried chicken? High calorie, high protein.
What other drinks have zero sugar? Pepsi Black also has zero calories and zero protein.
100 Plus Zero is also low calorie, low protein.
If we want to find out if a new word is closer in meaning to 100 Plus Zero or KFC, we find out its calories and protein. An unknown fried chicken will be high calories, high protein. It must be somewhere near KFC. This is how AI can tell whether two words are the same, or whether they have the same meaning.
3. Live Fetch vs. Indexed Retrieval (The Librarian)
Ultimately, how the AI retrieves this data comes down to two models: Live Fetch and Index Retrieval.
If you need a mental model, doing live fetch is a librarian. When you ask him to find a book, he will walk the whole library to find a book and come back.
When doing index retrieval, the librarian keeps a catalog, which he updates every month. When you ask him to find a book, it tells you straight away from the catalog. He doesn't have to go and walk through the library.
By building this catalog (index) beforehand, the system can instantly find the semantically matching chunks and hand the AI exactly what it needs to ace the open-book exam.
Ready to bring RAG into your own workflows?
The open-book exam only works if the retrieval is actually accurate. Get the chunking or indexing wrong and the AI still shows up to the exam confidently, just with the wrong pages in front of it.
If you're deciding whether RAG is the right approach for a specific workflow, or you've built something and the retrieval isn't holding up the way it should, get in touch. Email us at hello@redairship.com and tell us what you're working with. We'll give you a straight read on where the problem actually sits.
Rate this insight
Was this article useful?
Click to rate
