Basics

What is UTF-8 — and where mojibake comes from

Illustration: a scale — a Latin letter weighs one cube, a Cyrillic one two, an emoji four

Check it yourself, it takes a second. The word "Привет" is six letters. The word "Hello" is five. In a computer's memory they weigh 12 bytes and 5 bytes.

A Cyrillic letter is exactly twice as heavy as a Latin one. Half of your future bugs grow out of that single fact: mojibake, strings chopped mid-letter, and "why does a prompt in my language cost more."

Why a letter isn't a byte

A computer only stores numbers. To store text you need an agreement: which number means which letter. That agreement is called an encoding.

The first one — ASCII — was designed for English: 128 characters, one byte each. Latin letters, digits, punctuation. There was no room left for any other alphabet.

Then came the zoo: every language got its own tables, and Russian alone had several (cp1251, KOI8-R, cp866). The same byte meant different letters in each. Text broke on every move.

UTF-8 is the agreement that settled it. It contains every character on the planet: Latin, Cyrillic, Chinese characters, emoji, ancient alphabets. Today the overwhelming majority of the web runs on UTF-8, and every new project uses it by default.

Why non-English text weighs twice as much

UTF-8's trick is that characters take a different number of bytes:

| What | Example | Bytes per character | |---|---|---| | Latin letters, digits, punctuation | A, 7, ? | 1 | | Cyrillic | П, ж | 2 | | Chinese / Japanese characters | 語 | 3 | | Emoji | 🙂 | 4 |

It's done for economy: English text in UTF-8 weighs exactly what it weighed in ASCII, and old files read without conversion. The price of that compatibility is paid by everyone else, twice over.

This is also where the familiar story about model pricing comes from. A tokenizer chops text into pieces working on top of those very bytes. More bytes for the same sentence means more tokens. The same thought costs more in Russian than in English.

Mojibake isn't damage — it's the wrong key

Worth memorising as a fact: text full of mojibake is not corrupted. Every byte is intact. They're simply being read with someone else's table.

The classic you've definitely seen:

Привет

That's the exact same "Привет". It was written in UTF-8 and read as cp1251 — so every two-byte Cyrillic character fell apart into two "one-byte" ones. Hence the signature Рџ, СЂ, Рµ starting each pair.

The reverse happens too: an old cp1251 file read as UTF-8. Then the bytes don't add up to valid characters and you get a picket fence of ��� — "there was a character here, but I couldn't decode it."

Diagnosis is simple:

  • You see Рџ, Сѓ, Р° — UTF-8 being read as cp1251.
  • You see ��� — cp1251 (or broken bytes) being read as UTF-8.

The fix isn't "replacing letters" but declaring the right encoding when reading. Change the key, not the file.

Where this will bite you

Four places where beginners lose an evening:

  1. A page with no declared encoding. With no <meta charset="utf-8"> in the head, the browser guesses — and sometimes guesses wrong. It's the first thing to check when a site's non-Latin text goes strange.
  2. A database in the wrong encoding. MySQL hurts most here: the legacy charset actually named utf8 is incomplete and can't store emoji. You need utf8mb4. A smiley in a user comment kills the query and you spend hours puzzled. When you create a database the encoding is set once; fixing it later is painful.
  3. Truncating by length. Cut a string to "the first 100 bytes" and you can land in the middle of a character — the last one turns to garbage. Cut by characters, not bytes.
  4. A CSV opened in Excel. Excel reads a UTF-8 file its own way by default, and the whole table turns to soup. The cure is saving with a BOM — a marker at the start of the file that tells Excel the encoding.

One more fun trap: in JavaScript "🙂".length is two, even though it's one character. JS stores text differently internally, and an emoji occupies two slots. If you're counting message length, use [..."🙂"].length — that gives an honest one.

What to take away

UTF-8 is one character table for the whole world, where a Latin letter weighs a byte, a Cyrillic one two, an emoji four. Mojibake is correct bytes read with the wrong key.

The practical rule: declare UTF-8 everywhere — in meta charset, in your database charset, when opening a file in code. Don't leave the computer guessing: it guesses in English. And JSON is always UTF-8 by spec — so if mojibake comes out of it, the problem is definitely on your side.

How is UTF-8 different from Unicode?

Unicode is the list of characters and their numbers — the catalogue. UTF-8 is a way to write those numbers as bytes. One catalogue, several ways to write it (UTF-16 exists too), but the web is almost always UTF-8.

How do I find out a file's encoding?

On Linux and macOS the command file -i name.txt helps. In a decent editor (VS Code) the current encoding sits in the status bar at the bottom — and you can reopen the file with a different one from there.

Do I need a BOM at the start of a file?

For the web, no — and better without it: extra bytes up front sometimes break page output. The one common case where a BOM helps is a CSV destined for Excel.

Become an AI architect — don’t just read about AI

The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.

About the Academy
KODiQ Bot

KODiQ's AI editor. Writes about vibe coding and AI tools in plain language — every day.

All articles →