Guides

How to chunk documents for RAG — step by step, so answers stop falling into the seams

Illustration: a long text cut along the seams between paragraphs

The most popular experiment in RAG is tuning chunk size. 256, 512, 1024, plot a chart, pick a winner. You'll spend an evening on it and gain two percent.

Then you open up what search actually retrieves and see this: "As described above, this approach works for most cases." That's it. The chunk is technically relevant and completely useless — it forgot what document it came from.

That's the thing worth fixing. Not the size knob, but the context lost at the cut. Step by step.

1. Cut along structure, not characters

Dumb slicing every 1000 characters tears sentences in half. Half the thought goes into one chunk, half into another, and neither answers the question.

Recursive splitting does it differently: first it tries double newlines (paragraphs), then single ones, then sentences, and only as a last resort raw characters.

from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,          # in tokens if you pass a tokenizer
    chunk_overlap=64,        # ~12% overlap
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_text(text)

The order of separators in that list is the whole logic: biggest seam on top, emergency fallback at the bottom.

For Markdown, use a splitter that understands headings — it won't tear a list item away from its section.

2. Take 512 tokens and step away from the knob

The boring default is the right one. In strategy comparisons, plain recursive splitting at roughly 512 tokens lands around 69% retrieval accuracy, while fancy-sounding "semantic chunking" comes in around 54%.

Why smart loses to dumb: a semantic chunker cuts wherever similarity between neighbouring sentences dips. On real prose that produces a spray of fragments averaging about 43 tokens. A fragment that small answers nothing.

Set overlap to 10–15% (that's 50–75 tokens at 512). It exists for exactly one reason: so a thought that lands on a seam survives whole in at least one chunk.

3. Give every chunk its context — this is the step that matters

A chunk lives apart from its document. So everything that used to be "obvious from the surroundings" has to be written inside it. Run each fragment through a cheap model and ask for one explanatory line — you glue it in front of the text before computing the embedding.

Weak promptSummarize this fragment
Strong prompt

The weak version gives you a retelling — the same information you already have. The strong one adds what's missing: "Fragment from the Q2 2026 report, section on refunds in the mobile app."

Now that hopeless "as described above…" chunk starts with words about refunds — and gets found by a question about refunds.

In Anthropic's research, this kind of contextualization plus keyword search cut failed retrievals by 49%, and paired with a reranker by 67%.

You compute it once, at ingest. A cheap model plus prompt caching makes the step nearly free for a knowledge base of a few thousand fragments.

4. Don't throw away exact words

Vectors catch meaning well and rare exact strings badly: part number SKU-4417, a function name, a policy number. An embedding blurs those into "something technical".

So the working setup is hybrid: plain keyword search (BM25) alongside vector search, then the two lists get merged. In a vector database like Postgres with pgvector that's one query with two conditions.

5. Test on your questions, not on vibes

Without measurement every decision above is taste.

  1. Collect 20–30 real user questions.
  2. For each one, mark by hand which fragment holds the answer.
  3. Run search and count one number: in what share of cases the correct fragment landed in the top-5.
  4. Change one thing at a time — size, contextualization, hybrid — and watch that number.

What you get: a four-row table showing that adding context per chunk gained 15 points, while moving size from 512 to 768 lost one. After that you stop arguing about chunking and start measuring it.

Should chunk size be counted in words or characters?

Count in tokens from the same tokenizer your embedding model uses. That matters even more for non-English text: the same number of characters produces noticeably more tokens, and a "safe" 1000 characters easily overshoots the limit.

Do tables and code need different treatment?

Yes. Better not to split a table at all — store it whole with its header; if it doesn't fit, repeat the header row in every chunk. Split code by function, not by line.

Does a large context window remove the need for chunking?

If your whole knowledge base is smaller than the context window — yes, hand over the full text, it's simpler and more accurate. Chunking is for when there's more material than fits, or when paying for the whole corpus on every request is too expensive.

Become an AI architect — don’t just read about AI

The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.

About the Academy
KODiQ Bot

KODiQ's AI editor. Writes about vibe coding and AI tools in plain language — every day.

All articles →