Vector search or keyword search — what to pick for AI search over your own data

You built search over your documents with embeddings. It works beautifully: ask "how do I get my money back" and it finds the refunds section, even though the word "back" never appears there.
Then a user searches for SKU-4471. And search confidently returns five lovely, completely wrong paragraphs.
Here's the nasty part: nothing broke. No error, clean logs, and the model cheerfully writes an answer from the wrong chunks. A vector-search miss is silent — which is the main reason it's almost never left alone in production.
Two different sieves
BM25 is keyword search, and it isn't a neural network. It's a formula from the nineties: count how often a query word appears in a document, weighted by how rare that word is. "The" is everywhere, so it means almost nothing. SKU-4471 appears in one document out of ten thousand, so matching it is worth a lot. No training, just statistics over your own texts.
Vector search compares meanings. Text becomes an embedding — a list of numbers — and texts close in meaning end up close in that space. That's how it finds "refund" for "get my money back": different letters, same direction. The full mechanics live in the piece on semantic search.
The difference in one line: BM25 looks for the same words, vectors look for the same idea.
| Criterion | Keyword search (BM25) | Vector search | |---|---|---| | Finds best | SKUs, error codes, names, function names, exact phrases | questions in the user's own words, synonyms, paraphrases | | Misses on | user phrased it differently — zero results | a rare token dissolves into the general meaning | | What a miss looks like | an empty list — instantly visible | five confident wrong results — invisible | | What you must set up | nothing beyond a database index | an embedding model plus vector storage | | Cost | computed in place | you pay to embed documents and queries | | New data | index it and it works | embeddings must be recomputed | | Language handling | needs word processing (stemming, stop words) | works out of the box, often across languages |
Why one doesn't replace the other
The temptation is obvious: if vector search is "smarter," just use that. Benchmarks disagree. In BEIR — a large multi-domain test suite — plain BM25 remains a strong contender and beats dense models on text unlike what the model was trained on: medicine, legal documents, narrow catalogues full of codes and part numbers.
The reason is clear if you remember the mechanics. An embedding is compression. It pulls toward the overall gist of a phrase, and a rare token like a part number is precisely the detail compression drops first. To a human, SKU-4471 and SKU-4472 are different things; to a meaning model they're nearly the same point.
The reverse holds too: BM25 is helpless when the document says "return processing" and the person asks "how do I get my money back for an item." No shared words, no matches.
How they're fused: by rank, not by averaging
The first instinct is to add up the scores. It doesn't work, and it's worth knowing why up front.
The scales aren't comparable. BM25 returns an unbounded number that depends on your collection's statistics: 3 here, 18 there. Cosine similarity sits in a narrow band near one. Adding them gives a number with no meaning, and normalizing per query is brittle: one outlier and the ranking falls apart.
So the standard became RRF (Reciprocal Rank Fusion) — a trick from a 2009 paper that simply throws the scores away and keeps the ranks. Each document gets 1 / (k + rank) in each list, and those fractions are summed; k is usually 60. A document that lands in both top tens beats one that's first in a single list and missing from the other.
Nothing to train, the formula fits on one line — and it's what sits under hybrid search in Elasticsearch, OpenSearch, Azure AI Search, Weaviate and Qdrant. If your database does both full-text and vector search, hybrid is usually one query rather than a separate service.
Who should pick what
- BM25 only — when people search for short exact things: products by SKU, logs by error code, documents by number. Cheap, predictable, zero dependencies. Start here: every decent database ships full-text search.
- Vectors only — when queries are long and human and the data is uniform: a support knowledge base, notes, conversations. And when you're fine paying to embed.
- Hybrid — as soon as your data mixes both: names, codes and file names next to human prose. For RAG that's the default state — the model can't answer from a document retrieval never fetched.
Practical order: BM25 first, because it's already there; add vectors when you see paraphrase failures; then fuse with RRF. Not the other way around — otherwise the very first SKU turns into that silent miss.
Isn't this the same as full-text search in my database?
Practically, yes: what Postgres calls full-text search is a family of algorithms that includes BM25. A dedicated search engine becomes necessary when you hit scale or need fine ranking control.
Is hybrid more expensive than plain search?
It costs one extra database query and one query embedding — usually tens of milliseconds. Next to a model response measured in seconds that's barely visible, while the miss rate drops noticeably.
How do I tell whether my search is missing things?
Collect twenty real user queries and look at what came back with your own eyes. Deliberately include SKUs, error codes and proper names — that's exactly where the silent miss shows up.
The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.





