How to classify text with AI — in one evening, with an honest accuracy number

Sorting a hundred emails into folders with a model is an evening's work. Most people stop halfway, though: the prompt is written, the answers look "about right", and how right they are stays unknown.
Here's the whole path, including the part everyone skips: the number. By the end you'll be able to say not "works fine" but "94% accuracy, confuses support with complaints".
What you need: an API key from any provider and a text file of examples. Step by step below.
1. Write out your labels — before any code
Open a notepad and write out the categories. Not "roughly these" but final ones — the model and your code will both see them.
Three rules that will save you half your debugging:
- 4–7 labels plus "other". Fewer loses the point; more and neighbouring labels start blurring.
- Labels must not overlap. If "support" and "complaint" sit side by side, a human would get it wrong too. Either merge them or spell out the difference.
- "Other" is mandatory. Without it the model will force anything unclear into the nearest box and you'll never find out.
Example for an inbox: invoice, support, partnership, spam, other.
2. Collect 30 examples and label them by hand
This is the step you'll want to skip. Don't — without it you have nothing to measure with.
Take 30–50 real texts and put a label on each one yourself. A plain two-column CSV:
text,label
"Hi, attaching the July statement",invoice
"the login code never arrives, please help",support
"Earn $300 a day on autopilot",spam
While labelling you'll almost certainly want to fix the list from step 1 — that's normal, and it's exactly the point. Fix the labels now, not after launch.
Thirty examples is enough to catch the gross mistakes. A hundred is enough to trust the number. You don't need more to start.
3. Forbid the model from improvising
Asking "answer with one word" works 95 times out of 100. The other five you get "Most likely this is an invoice" — and your code dies on a string comparison.
So the label list is defined by a schema, not by prompt text:
from enum import Enum
from google import genai
class Category(Enum):
INVOICE = 'invoice'
SUPPORT = 'support'
PARTNER = 'partnership'
SPAM = 'spam'
OTHER = 'other'
client = genai.Client()
def classify(text: str) -> str:
r = client.models.generate_content(
model='gemini-2.5-flash',
contents=text,
config={
'response_mime_type': 'text/x.enum',
'response_schema': Category,
'system_instruction': (
'You classify incoming company email. '
'Judge the purpose of the message, not its polite wording. '
'If the purpose is unclear — other.'
),
},
)
return r.text.strip()
text/x.enum is not JSON: the output is one word, no quotes, no wrapper. Nothing to parse, a plain string if works right away. The mechanism is covered in structured output.
The prompt itself is worth writing like a set of rules for a human, not "classify this text":
Determine the category of this email: invoice, support, partnership, spam, otherNotice the strong version handles the edge cases. Those are what buys accuracy — not the polite framing at the top.
4. Run it and count the accuracy
Now for what step 2 was for:
import csv
rows = list(csv.DictReader(open('samples.csv')))
ok, errors = 0, []
for row in rows:
got = classify(row['text'])
if got == row['label']:
ok += 1
else:
errors.append((row['label'], got, row['text'][:60]))
print(f'accuracy: {ok}/{len(rows)} = {ok / len(rows):.0%}')
for want, got, text in errors:
print(f' wanted {want}, got {got}: {text}')
That's it. You have a number. This is an AI eval in its minimal form — from now on any prompt change is verified by a single run.
5. Read the errors instead of rewriting the prompt
The main trap: you see 80% and start twiddling wording. Look at the errors themselves first — they almost always cluster.
- Two categories confuse each other. That's not the AI, that's your labels overlapping. Fix the list, not the prompt.
- Everything lands in "other". Not enough context — you're feeding too short a fragment.
- Errors are spread evenly across all labels. Now it's worth adding 2–3 edge cases to the prompt, or reaching for a stronger model.
Also: check your own labelling. Rather more often than you'd expect, a "model mistake" turns out to be your typo in the CSV.
6. Make it cheaper
Classification is the cheapest task a model can do, and the easiest to save on:
- A small model. Sorting an email into five boxes isn't writing an article. A flagship is nearly always overkill here.
- A short input. The subject and first 500 characters are usually plenty, not the whole email.
- Batch mode. If the sorting isn't urgent, send a batch — most providers charge less for it. The rest of the tricks are in how to cut your AI costs.
What you end up with
A 15-line function that takes text and returns exactly one of your labels — never "possibly an invoice". Plus a script that answers every change with a number.
From there it plugs in anywhere: an email webhook, a bot, a form on your site. And you now have what most AI features don't — a way to know whether it got better or worse. Why it works this way is unpacked in what is classification.
What does this actually cost?
Classifying one email is roughly 300–800 input tokens and one output token. On a small model, a thousand emails costs a few cents. It only gets expensive if you push full text through a flagship.
What if I need many labels — say fifty?
Then don't make it one choice out of fifty. Split it in two: first 5–7 broad groups, then a refinement within the chosen group. Accuracy goes up and the cost goes down, because the second call sees a short list.
Building your first feature like this is easier with someone explaining it step by step in plain language — that's what KODiQ is for.
The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.





