Why your vector search returns garbage — 3 causes and the fixes

You built search over your own documents. You ask "what's our return window" — and back come a chunk about shipping and a paragraph from an office lease. The model confidently invents an answer on top of that.
First thought: "the model is dumb." Almost always: no. The garbage arrived before the model, from search.
And here's the main cause nobody says out loud.
Cause 1. Vector search can't say "not found"
Vector search returns the k nearest chunks. Always exactly k. Ask for five, get five — even when the nearest "neighbour" is from another universe entirely.
Ordinary text search will honestly say "nothing found." Vector search never will: every query has a nearest point, the only question is how far away it sits.
Then the worst part happens: those five chunks go into the prompt labelled "context from your documents." The model treats them as material for the answer. That's where confidently wrong answers come from — a RAG classic.
How to check. Log not just the retrieved text but its similarity score. Run a control query for something definitely not in your data ("when is our Sochi company retreat"). Look at the numbers — that's your noise floor.
How to fix. Set a threshold: drop results below a certain similarity. If nothing survives the filter, answer accordingly: "that isn't in the documents." Say it to the model too, in the system prompt: "answer only from the provided fragments; if they aren't enough, say you don't know."
Cause 2. Chunks are the wrong size
An embedding compresses a text's meaning into a single point. Which leads to simple logic: the more text you squeeze into one point, the blurrier that "meaning" gets.
Two typical extremes:
Chunks too big. You indexed a whole page — its vector became an average of everything: a bit about returns, a bit about shipping, a bit about the legal entity. Such a vector is equally weakly similar to every query at once.
Chunks too small. You cut at 100 characters — every fragment lost its context. "It is 14 days" — and what "it" refers to lives in the neighbouring chunk, which didn't get retrieved.
How to fix. The working middle is roughly 300–800 tokens per chunk with a small overlap (10–15%) so a thought doesn't get cut at the seam. But more important than size is where you cut: on headings and paragraphs, not blindly by a character counter. One section, one chunk.
And prepend the document's heading to each chunk's text. The fragment "14 days from receipt" is useless; "Return policy → Window: 14 days from receipt" retrieves beautifully.
Cause 3. The embeddings don't speak your language (or there are two of them)
The most infuriating bug, because it breaks everything at once: documents were indexed with one embedding model and queries are embedded with another. You updated a library, switched to a newer model — and the old index turned into noise. The numbers look fine; the similarity is random.
How to check. Take a chunk of text from your database and search for it verbatim. If it doesn't come back first, it isn't your chunking or your threshold. Your index and your queries live in different coordinate systems.
The second flavour of the same problem is language. If the embedding model was trained mostly on English, a non-English query maps into its space crudely, and semantic search works at half strength. For non-English documents, pick a model that advertises multilingual support explicitly.
How to fix. One model for indexing and for queries, strictly. Change the model and you reindex everything — partial updates don't work here. Write the model name down next to the index so you're not guessing six months later.
And check that it's the right tool at all
Vector search finds things similar in meaning. It's great at matching "how do I return an item" to "I'd like to arrange a refund" — and noticeably worse at exact strings: an SKU like SKU-4471, an error number, a surname, a date.
For those queries there's no meaning to match — there's an exact hit. Hence the standard answer: hybrid search. Regular full-text search finds exact matches, vector search finds meaning, and the results get merged. Nearly every mature document-search system works this way, and it fixes more cases than any threshold tuning.
If you're short on time, fix in this order: check the models match (cause 3 — fatal), then add a threshold and logs (cause 1 — cheap), and only then re-chunk your documents (cause 2 — slow).
What similarity threshold counts as normal?
There's no universal number: the scale depends on the model and the metric. To find yours, run a dozen obviously good queries and a dozen obviously nonsensical ones, look at the score distribution, and draw the line between the two clusters.
How many chunks should I give the model?
Usually 3–7. More isn't better: extra fragments dilute the context and raise the odds the model latches onto the wrong one. If an answer needs dozens of chunks, the task probably isn't search — it's a summary over the whole corpus.
Would a model with a bigger context window just solve it?
Sometimes — if you have few documents, you can drop them all into the prompt and skip search entirely. But as the corpus grows this runs into cost and speed: every query pays for the whole volume. Search earns its place exactly where "just include everything" stops being reasonable.
The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.





