Why my RAG returns wrong answers — 3 causes and how to check each

You built a bot over your own documents. The answer is definitely in there — you put it there yourself. Yet the bot answers off-target, or claims it found nothing. The first instinct is to rewrite the prompt or reach for a smarter model. That's almost always the wrong end: in most cases the right chunk of text simply never reached the model. The model answered honestly from what it was given. It was given the wrong thing.
Good news: you can identify the guilty stage in two minutes.
First, the one check that saves hours
RAG is two separate operations: first retrieval of relevant chunks, then generation of an answer from them. The first one breaks almost every time — and for some reason people fix the second.
Before changing anything, log the chunks that were retrieved — the exact text going into the model:
chunks = retriever.search(question, k=5)
for c in chunks:
print("---", c.score, c.text[:200])
Then look with your own eyes:
- The right chunk isn't in the list → retrieval is broken. Causes 1 and 2.
- The chunk is there and the answer is still wrong → generation is broken. Cause 3.
That's it. Half of all «why is my RAG dumb» questions die to that printout.
Cause 1 — retrieval doesn't find the right chunk
The most common one. You ask «how much is delivery to Kazan», the document says «nationwide freight rates». Different words — and the vector landed in the wrong neighbourhood too.
How to check. Take an exact phrase from the target document and ask that as the question. If it's found, retrieval works and your phrasing is the gap. If it's still not found, the problem is deeper — the chunk probably never made it into the index.
How to fix:
- Add keyword search alongside vector search. This is the big lever. SKUs, error codes, contract numbers and rare names are exactly what pure vector search handles badly — the embeddings vs keyword search comparison explains why. Hybrid closes that hole.
- Raise
k. If you hand the model 3 chunks, try 10. Cheap, and often an instant fix. - Check the index is current. A document was updated but its embeddings weren't recomputed — the bot answers from the old version and looks like a liar.
Cause 2 — the chunk is cut in the wrong place
Chunking is the quiet quality killer. Slice text mechanically every 500 characters and the cut lands wherever it lands.
The classic wreck: the heading «Return policy» sits at the end of one chunk, the actual policy at the start of the next. Searching for «return» finds the first chunk, which contains nothing but the heading.
How to check. Open 5–10 chunks from the index and read them as standalone texts. Is each one comprehensible without its neighbours? If a chunk opens with «...in which case a 15% rate applies», there's your answer.
How to fix:
- Cut on meaning boundaries — headings, paragraphs, sections. Not on character counts.
- Add 10–15% overlap so an idea sitting on a seam lands in both chunks.
- Prepend the section heading to every chunk. Cheap and remarkably effective: the chunk stops being anonymous and becomes findable by its section's topic.
- Never cut tables and lists in half. Half a table is useless to both the model and the search.
Cause 3 — the model ignored what it was given
Rarer, but real: the right chunk is in context and the answer is still wrong.
How to check. Ask the model to state which chunk it based its answer on. If it cites the wrong one or none at all, the problem is here.
Three typical failures:
- The model answered from its own knowledge instead of your documents. It has a general sense of the topic and it overrode your text. Fix it with a hard instruction: «Answer only from the provided chunks. If the answer isn't there, say you couldn't find it». Without that line the model will politely fill in the gaps — that is, hallucinate.
- Too many chunks. Dump 30 in and the right one drowns in the middle. Models see the beginning and end of a context noticeably better than its middle. Five precise chunks beat thirty «just in case».
- Chunks contradict each other. An old and a new revision of the same document both sit in the index, and the model picks at random. Clean duplicates and store document dates.
What if none of that helped?
Assemble 20 questions with known-correct answers and run them after every change. Without that mini test set you're fixing blind — and breaking one thing while repairing another.
Would a bigger model help?
Usually no. If the right chunk wasn't retrieved, the most powerful model available will miss just as badly — only in a more confident tone. Fix retrieval first.
How do I know it's chunking and not retrieval?
Ask the question using exact wording from the document. If the right chunk surfaces but the answer in it cuts off mid-thought, chunking is to blame.
The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.





