RAG or long context — paste your documents in or search through them

Not long ago, to make a bot answer from your documents, RAG was mandatory: the context window was tiny, the whole text wouldn't fit. Now models have windows of hundreds of thousands — even up to a million — tokens, so you can just paste the whole document straight into the prompt. The logical question: is RAG dead? Here's the answer that'll save you money and headaches: no. They solve different problems. Let's sort out when to use which.

Two ways to give a model knowledge

Both are about "make the model know your text," but they work differently.

Long context — you paste the whole document (or all of them) straight into the prompt. The model sees everything at once and answers. Simple as copy-paste: paste it, ask.

RAG — you don't paste everything. The system first finds the relevant chunks for the specific question and puts only those into the prompt — like a cheat sheet. The model reads three on-topic pages, not the whole book.

Comparison on axes that matter

| Criterion | Long context | RAG (chunk search) | |---|---|---| | What you do | paste the whole text into the prompt | find what's needed, paste only that | | Cost | you pay for all tokens every time | you pay only for the found chunk | | Data volume | up to the window limit | gigabytes — it needn't all fit the window | | Freshness | you have to resend it | edit the document in the base, done | | Reliability | "gets lost" in the middle of long text | you feed only what's needed — less noise | | Complexity | dead simple: paste and ask | needs a vector database and search |

Who should use which — no fence-sitting

Use long context if the task is one-off, there's a single document that fits the window, and simplicity matters. "Analyze this contract," "summarize this article," "find the bug in this file." Paste the text — get the answer. No point building search here.

Use RAG if there's a lot of data, it changes often, and running it all every time is expensive. A company knowledge base, support across hundreds of articles, your personal wiki. RAG feeds the model only what's relevant — cheaper, faster, and scales past the window limit.

Is it true the model "loses" the middle?

Yes, and it's an important caveat against "just paste everything." Long context has a "lost in the middle" effect: the model best remembers the start and end of the pasted text and recalls the middle of a long wall worse. On top of that, answer quality drops noticeably at very large volumes. So "a million tokens in the window" ≠ "the model remembers all million equally well." RAG sidesteps this by feeding little and on-point.

Will long context kill RAG?

No. It took the simple "one document, one-off question" cases away from RAG — you used to need search even there, now you can paste directly. But at scale (lots of data, frequent updates, saving on tokens) RAG stays cheaper. More likely they coexist: small stuff via context, volume via RAG.

Which is cheaper?

Almost always RAG — if there's a lot of data and you query often. Long context makes you pay for the whole pasted text on every request; paste a book, pay for the book again and again. RAG pays only for the found chunks. On a single one-off question the difference is pennies, but at volume it decides everything. And don't confuse this choice with RAG versus fine-tuning — that's a different question: how to give a model knowledge permanently.