What is a false positive — and why a 99% accurate filter is wrong half the time

Let's count this one together, because the number surprises almost everyone.

Your app handles 10,000 messages a day. Spam is 1% of them — a hundred. You wire up a filter that's 99% accurate: it catches 99 spam messages out of 100 and errs on just 1% of legitimate ones.

Now count the flags. Spam caught: 99. Legitimate messages number 9,900, and 1% of those is another 99.

That's 198 flags, and exactly half of them are innocent. A 99% accurate filter accuses a real human as often as it catches a spammer.

Nothing broke. That's just the math.

What a false positive is

A false positive is when the system said "yes" and the right answer was "no". The filter marked a normal email as spam. Fraud detection blocked an honest purchase. An AI text detector accused someone who wrote every word themselves.

The error has a mirror twin: a false negative — the system said "no" when it should have said "yes". Spam lands in the inbox. The fraudster gets through. The abuse stays in the comments.

Any system that assigns labels gets it wrong both ways. The question isn't whether it errs, but which way you'd rather it erred.

Why "99% accurate" tells you nothing

Go back to our count. The problem wasn't the filter — it was how rare the thing you're catching is.

The rarer the event, the more "normal" items flow past, and the larger the absolute number of mistakes on them. Drop spam from 1% to 0.1%:

• there are now 10 spam messages, and the filter catches all 10;
• there are 9,990 legitimate ones, and 1% of those is about 100 false flags;
• out of 110 flags, more than nine in ten are wrong.

Same filter. Same "99% accuracy". The event just got rarer.

Hence the practical rule: never judge a filter by a single accuracy number. Ask instead: of everyone it flagged, how many were actually guilty? That share is called precision. The share of real offenders it caught is recall.

The two errors cost different amounts

This is the important part, and it's no longer math — it's your decision.

• A spam filter. Letting one ad through is annoying. Sending a job offer to spam is a disaster. Here the false positive costs more.
• Moderation in a kids' app. Hiding a harmless comment by mistake is trivial. Missing bullying is not. Here the false negative costs more.
• Payment fraud. Both cost money: missed fraud is a direct loss, a blocked honest payment is a customer gone for good.

Until you've answered which of the two errors is more expensive for your product, you can't tune the filter. Not because you're short on data — because it isn't a data question.

The threshold is a dial, not a truth

Almost every classifier returns a number rather than a yes/no: 0.03 · 0.44 · 0.91. You decide where flagging starts. That's the threshold.

It works like a seesaw:

• lower it (flag from 0.3) — catch more offenders, snag more innocents;
• raise it (flag from 0.9) — barely touch innocents, let half the offenders through.

There is no magic value. There's only trading one error for the other. And remember the number itself is the model's confidence, not the probability of being right: 0.9 does not mean "true 90% of the time".

A more practical move than a single threshold is three outcomes:

1. below 0.3 — let it through silently;
2. 0.3 to 0.8 — let it through, but queue it for review;
3. above 0.8 — block it.

That way the expensive mistakes stay rare, and borderline cases don't turn into automatic verdicts.

Where you'll meet this in your own app

Practically anywhere the word "check" appears:

• moderating comments and uploaded images;
• finding duplicates and "similar" records;
• captchas and bot protection on forms;
• automatic unsubscribes, bans, rate limits;
• any checks around a model — on the way in and on the way out.

A simple starting rule: first work out how rare your event is. If offenders are under one percent of traffic, expect most flags to be wrong — and decide in advance where they go, other than straight to a ban.

Which matters more, precision or recall?

It depends on who pays for the mistake. If the user pays (they got blocked for nothing), precision matters more. If you or a victim pays (missed threat, missed fraud), recall matters more. You can't push both up at once, so the call is always a product decision, not a technical one.

How do I pick a threshold with no data?

Label by hand. Take 100–200 real examples from your own app, mark them yourself, run them through the filter and see what lands in the flags at different thresholds. A hundred honestly labelled examples beat any third-party benchmark — that's the simplest form of an eval. Then re-measure after every prompt change or model swap.