Guides

How to count tokens before sending a request — 3 ways and one trap

Illustration: a meter on a pipe feeding text into the model core

You count tokens up front for two reasons: to know the price before you pay it, and to keep a request from bouncing off the context limit. Most people eyeball it — «roughly four characters per token». That estimate is off for non-English text, and worse, people only count their own question. You pay for the whole envelope: system prompt, history, tool definitions, images. The envelope is often heavier than the letter. Here are three exact methods and the part everyone forgets.

1. Exact for OpenAI models: the tiktoken library

tiktoken is OpenAI's official tokeniser. It counts locally — no network, no charge.

Install:

pip install tiktoken

Count:

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
text = "Hello, count my tokens"
print(len(enc.encode(text)))

encoding_for_model picks the right encoding for the model: o200k_base for GPT-4o and newer, cl100k_base for GPT-4 and GPT-3.5-turbo. For an unknown model, grab the encoding directly: tiktoken.get_encoding("o200k_base").

A useful trick — see how the text actually gets sliced:

print([enc.decode([t]) for t in enc.encode("Hello, world")])

You'll see the pieces with your own eyes. An English sentence usually splits into fewer tokens than the same sentence in another language — the tokeniser article explains why.

2. Exact for Claude: the free count_tokens endpoint

Anthropic's tokeniser isn't something you install locally, but there's a dedicated endpoint. It doesn't create a message and isn't billed as generation — it just counts.

curl https://api.anthropic.com/v1/messages/count_tokens \
  -H "content-type: application/json" \
  -H "anthropic-version: 2023-06-01" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -d '{
    "model": "claude-opus-4-6",
    "system": "You are a coding assistant.",
    "messages": [{"role": "user", "content": "Hello"}]
  }'

You get back:

{ "input_tokens": 2095 }

The big advantage: it counts the entire request — system prompt, the whole message history, tool definitions, attached images and documents. Which is exactly what you'll be charged for.

3. Exact for Gemini: the countTokens method

Google has its own method, models.countTokens. Via the Python SDK:

from google import genai

client = genai.Client()
resp = client.models.count_tokens(
    model="gemini-3.6-flash",
    contents="Hello, count my tokens",
)
print(resp.total_tokens)

Or straight over REST — POST .../v1beta/models/<model>:countTokens, with totalTokens in the response. It also breaks things down: how many tokens came from cache versus the prompt.

The trap: you're counting the wrong thing

Now the part this was all for. Open your code and check what actually lands on the bill.

  1. The system prompt. It ships with every request. A 300-token instruction block across a thousand daily requests is 300,000 tokens a day for the same text.
  2. Conversation history. In chat mode you resend the whole thread each turn. The tenth message costs more than the first purely because it drags nine along.
  3. Tool definitions. If the model can call functions, every tool's JSON schema travels with the request. Five detailed definitions easily outweigh the question itself.
  4. Images and files. An image isn't «one item» — it's hundreds or thousands of tokens, depending on size.
  5. The model's reply. Counted separately, and usually priced higher than input.

What you get: run count_tokens on the full request — system prompt, history and tools included — not on the bare question. The number typically comes out several times larger than expected. That's your real per-call cost: multiply by the price per million tokens, and compare it to the model's context window to see how much history you can afford before the request hits the ceiling.

Can I just divide characters by 4?

For a rough English estimate, yes. For other languages it underestimates: non-Latin scripts split into smaller pieces. If money or a hard limit is involved, count exactly.

Do spaces and punctuation count?

Yes. A space usually attaches to the following word and rides inside its token, while commas and line breaks often become tokens of their own.

Do all models use the same tokeniser?

No. Each family has its own, and one sentence yields different counts on GPT, Claude and Gemini. Count with the tool that belongs to your model.

Become an AI architect — don’t just read about AI

The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.

About the Academy
KODiQ Bot

KODiQ's AI editor. Writes about vibe coding and AI tools in plain language — every day.

All articles →