What is a tokenizer — why the model never sees letters, and Russian costs more

It feels like a language model reads your text the way you do — letters, words. It doesn't: the model never sees letters at all. Between you and the model sits a separate part — the tokenizer. It slices text into chunks before the model. And here's the surprising consequence: because of how it slices, the same sentence in Russian costs more than in English. Let's see why.

What a tokenizer is, in plain words

A tokenizer is a program that slices your text into chunks (tokens) before the model sees it. The model works not with letters or words, but with these chunks and their numbers. The tokenizer is a translator from "human text" into "numbers the model understands," and back again.

Analogy: a barcode at the checkout. The cashier (the model) doesn't read the product name — they see a code. The tokenizer is the scanner that turns your sentence into a stream of codes.

How it slices — not by words, not by letters

The surprise: the chunks are not words. The tokenizer slices by statistics: common chunks (whole words, endings, spaces) become one token; rare ones get broken into parts. "token" is one token, but "tokenization" might become "token" + "ization." The technique is called BPE — gluing frequently-seen pairs of characters into a single chunk.

So one token is, on average, not a letter and not a word, but a chunk: sometimes a whole word, sometimes a couple of letters. Spaces and punctuation count as tokens too.

Why Russian costs more than English

Here's the main point, and it's about money. Tokenizer vocabularies were built mostly on English — common English words got "fat" single tokens. Cyrillic got almost none: Russian words are more often chopped into small pieces, down to individual characters.

The result: the same thought in Russian comes out 2–3× more tokens than in English. And you pay — and measure the context window limit — in tokens. So Russian text eats more money and more room in the model's memory for the same thought. This is also why a token is the unit everything is measured in: both price and request length.

Why this matters to you

Three practical consequences:

• The count is in tokens, not characters. When estimating the cost of a request, count chunks, not letters — especially in a non-English language.
• The window limit is in tokens too. A long non-English document takes up more room than the word count suggests.
• Odd truncations and typos from the model sometimes come from a word being cut in the wrong place. Once you get the tokenizer, you stop being surprised.

Are a tokenizer and embeddings the same thing?

No, they're two different steps. The tokenizer splits text into chunks and gives each a number. Embeddings are the next stage: each number turns into a list of figures that capture the chunk's meaning. First we slice (tokenizer), then we make sense of it (embeddings).

Can I see how my text gets sliced?

Yes. Many models have online "tokenizers": paste a phrase and see how many chunks it broke into. It's worth running the same thought in two languages once and seeing the difference in the count with your own eyes. After that, "why non-English costs more" stops being abstract.

Do all models use the same tokenizer?

No — different models have different tokenizers and different vocabularies. So the same text can break into a slightly different number of tokens across two models. The general principle is the same (common = big chunks, rare = small chunks), but the exact boundaries and the price in tokens are specific to each model.