What is a confidence score — and why "97% sure" doesn't mean "true"

Try this. Ask a model something real and ask it to rate its confidence — you'll get "95%". Now ask about a book that doesn't exist and ask for the same rating. You'll probably hear something around 90%.

The number barely moved. That's not a fault: a model's confidence measures something other than what you think.

What a confidence score is

A confidence score is a number from 0 to 1 (or a percentage) showing how sure the model is about what it produced. You'll meet it in three forms: as a classifier's score ("spam: 0.93"), as a log-probability attached to each word in an answer, and as the phrase "I'm about 80% sure" written straight into the text.

The trouble is that all three answer different questions while looking identical.

Where the number comes from

At every step, a language model picks the next token out of its whole vocabulary. Before it picks, it has a probability for each option: "Paris" — 0.91, "Lyon" — 0.04, "capital" — 0.01, and so on.

APIs can hand those to you — they're called logprobs (log probabilities). A logprob is always negative or zero, where zero means total certainty. Usually you can get the chosen token plus a few runners-up with their probabilities.

That sounds like an honest window into the model's insides. It is — but the window faces the wrong way.

Confidence in a token isn't confidence in the answer

This is the part worth reading twice.

A logprob says: "in this position I'm nearly certain the word is Paris". It does not say: "the claim I'm writing matches reality".

If the model invented a book that doesn't exist, it will write the title with great confidence — because after three words of a made-up heading, the fourth is easy to predict. Inside its own fiction the model is perfectly consistent. That's exactly why a hallucination sounds so convincing: high confidence in the wording, zero information about truth.

What calibration is — and how fine-tuning breaks it

A well-calibrated model is one where, across all the answers it rated "80%", about 80% turn out correct. A badly calibrated one says "95%" and is right half the time.

And here's the surprise. The GPT-4 technical report showed two versions of one model:

• the base model (the one that just predicts text) came out almost perfectly calibrated;
• the same model after fine-tuning on human feedback (RLHF) ended up roughly ten times worse calibrated.

The training that made it a polite, useful assistant broke the honesty of its numbers. Which makes sense: human raters like confident, smooth answers, so the model learned to sound confident. Confident always.

That's why you can't lift a logprob out of a chat model and drop it into your logic as a probability of truth.

Three ways to get confidence — and what to do with each

1. Logprobs. Cheap, they arrive with the answer. They work well where the answer is a single word from a closed set: spam or not, ticket category, yes or no. On multiple-choice work they're honest.

2. Just ask. Tell the model: "rate your confidence from 0 to 100". It sounds naive, but for chat models it often works better than raw probabilities: in a Stanford and Harvard study (EMNLP 2023), confidence stated in words came about twice as close to reality as the same models' internal probabilities.

3. Ask several times. Put the same question three to five times and compare. Answers agree — you can trust it. They diverge — the model doesn't know. It costs three to five times more, but it's the most reliable signal of the three, because it measures the stability of the answer rather than the wording.

How to actually use it

Don't turn confidence into a true/false verdict. Use it as a router:

• high confidence → show the answer;
• medium → show it, flagged "check this" or with a source link;
• low → don't show it; ask again or hand it to a human.

And keep in mind that any threshold on that number is a choice between two kinds of error: let nonsense through, or reject good answers for nothing. The arithmetic is in the piece on false positives.

Can I just ask the model for a confidence percentage?

Yes, and for chat models it's a surprisingly solid option — often better than raw probabilities. But ask for a number on a scale with a one-line reason, not a plain "are you sure": that way the answer slides into a reflexive "95%" less often. Even then, verify — the figure is an estimate, not a measurement.

Do all models expose logprobs?

No. Some APIs return them behind a flag, some don't offer them at all — especially reasoning models, where the internal thinking is hidden. With no logprobs you still have the other two routes: ask in words, or run the same request several times and compare the answers.