When building an AI assistant to read your internal documents, you have to decide how it accesses that data. Behind the scenes, there are two primary ways to read your drive: Live Fetch and Indexed Retrieval.
Choosing the right approach is a business question, not just a technical one. Here is a breakdown of how both techniques actually work under the hood, using the mental models and exact scenarios discussed during our workshop.
Method 1: Live Fetch (Direct Retrieval)
Live fetch operates in real-time. Every time a user asks a question, the system goes and fetches the raw data.
If you need a mental model, doing live fetch is a librarian. When you ask him to find a book, he will walk the whole library to find a book and come back.
The Pros of Live Fetch
Freshness: When you do live fetch, every time you ask, you'll find the raw data, which means that you access the latest version of the data.
Simplicity: It's really simple to set up. If you point it to a folder, you can do it already.
The Limits of Live Fetch
Platform Caching: As a caveat, it is still subject to platform-level caching. If Google Drive decides to cache, then it's too bad; you need to wait for 15 minutes before it refreshes the cache.
Context Budget and Attention: The scope is kind of smaller compared to index retrieval because you can only feed so much data into your context. Even if you are lucky and it fits into your 1 million token context, you'll still lose attention. Attention is the strongest at the start and at the end. The middle 40%, some say 60%, is prone to hallucination or not finding info at all.
Cost: You don't want to just burn tokens by filling up the whole document in your context. You want to only feed the relevant information.
Method 2: Indexed Retrieval
With index retrieval, it indexes everything. When you read, it reads the index rather than the data source.
When doing index retrieval, the librarian keeps a catalog, which he updates every month. When you ask him to find a book, it tells you straight away from the catalog. He doesn't have to go and walk through the library.
The Pros of Indexed Retrieval
Scale: Since you're building a catalog, not in real time, not on demand, it can scale really well. Thousands, millions is not an issue.
Semantic Search: It searches faster and you can search by meaning instead of just keywords. For example, when you go to Google Drive and say search by "pricing", at most it can match "prices" or "price" or some typo that looks like price. It can never give you "cost", which is a totally different keyword but with the same meaning. But with index retrieval, we can.
The Limits of Indexed Retrieval
Lag Risk: The downside is, if there are any books written or loaned out after the refresh, then you wouldn't know about it. It also means that it won't be seeing the latest; you'll be slightly outdated.
The Read/Write Problem: You have to wait for the refresh, and while refreshing your whole index, it actually burns tokens as well. So it's a classic "do you read more or write more" problem. If your organization does a lot of writing or very little reading, maybe this is not for you. But if you do very little writing but you do thousands of reads, maybe this is for you.
Batch Processing: To optimize the tokens burned on refreshing the index, you can use batch processing. The batch pricing is half of the standard on-demand pricing.
The Deciding Factor: User Needs and Data Structure
Ultimately, the choice between these two methods depends entirely on the end user and the specific reality of their workflow. When making this decision, you must evaluate three key variables:
Data Volume and Structure: Are you dealing with a handful of targeted, highly structured files, or do you need to search across a massive, unstructured historical archive spanning multiple clients and years?
Refresh Frequency: Does your data change constantly (daily or weekly), requiring up-to-the-minute freshness to avoid quoting an outdated figure? Or is your data relatively fixed, such as master service agreements or standard pricing sheets that only need to be refreshed quarterly or yearly?
Context Budget: You must consider what actually needs to be in your AI's context and how much it can comfortably fit without degrading its attention span.
Our Recommendation
Start simple and graduate when it hurts.
Let's say you want to do a Proof of Concept (POC). You can probably start with Live Fetch.
We use Live Fetch because it's easy. You don't have to deal with the infrastructure on how to ingest your data or refresh your index. You do that so that you can prove, is AI the right solution?. If you use AI and do Live Fetch and it works, and your users say it is giving them the input they need, then it's a solution and problem fit.
Then you go into optimising the solution and deciding if you should do indexed retrieval. You don't need to jump straight into implementing full indexed retrieval from the start. Start small, and graduate when you outgrow it.
Rate this insight
Was this article useful?
Click to rate
