What is overfitting — why a model aces the test and flunks real work

Here's the thing that breaks your intuition: a model can score 99% on its check and still be useless in real work. Not because it was undertrained. The opposite — because it was trained too long. That's called overfitting, and in five minutes you'll be spotting it yourself.
What overfitting means in plain words
Overfitting is when a model memorises the specific examples instead of picking up the general rule.
Think of a suit. Imagine a jacket tailored to one person so tightly that it follows every crease and every slope of the shoulder. On that person it's perfect. On anyone else it doesn't go on at all. A model tailors exactly that kind of suit to its training data: it fits not just the pattern, but the random noise, the typos and the quirks of that particular sample.
The key part: overfitting is not a bug. The code works, training runs as designed, the numbers on the training set look great. The trouble is that your training data isn't the whole world.
Why "train a bit longer" starts to hurt
While a model learns, progress is measured on two different sets. One is what it trains on. The other is held back, never seen.
Here's what happens:
- Early on both numbers improve together. The model is catching real patterns — this is the useful part of learning.
- At some point the training score still climbs while the held-out score flattens.
- Then the training score keeps climbing and the held-out score starts to fall. That's where overfitting begins: the model isn't learning the rule any more, it's memorising specifics.
The point where those two lines part ways is where you should have stopped. The technique is literally called early stopping: end training not when the training number looks perfect, but when the held-out number stops improving.
That's why a dataset is almost always split three ways: one part to train on, a second to watch that gap, a third to check once at the very end. The third exists so you don't tune the model to the second one too.
Why even model benchmarks trip over this
Now the twist — the same trap works at industry scale.
A benchmark is honest only as long as the model hasn't seen its questions during training. But training corpora are scraped from the internet, and popular test sets sit there in the open — answers included. If a test leaks into training data, the model shines on it not because it can reason, but because it remembers. This is called test leakage, or contamination.
Practical takeaway: a high spot on a leaderboard is not a promise the model will handle your task. Your data and your phrasing were definitely not in its training. So your own small set of checks on 20 real examples from your work tells you more than any public ranking.
How to spot overfitting in your own work
You'll meet this the moment you try fine-tuning a model for your task. Check like this:
- Hold data back up front. Before training starts, pull out 10–20% of examples and never touch them. Without that you simply can't see the problem.
- Watch the gap. Great on training, clearly worse on held-out data — that's it. The gap matters more than the absolute numbers.
- Read a couple of answers yourself. Classic symptom: the model reproduces phrasing from your examples word for word instead of answering on the merits.
What helps: more and more varied data (a hundred examples of one kind is worse than thirty different ones), fewer training passes, a simpler model. A small dataset plus many passes over it is the most reliable recipe for overfitting there is.
How is overfitting different from underfitting?
Two ends of one axis. Underfitting means the model never caught the general rule — bad on training data and on new data alike. Overfitting means great on training, poor on new. The first is cured by training longer with a bigger model; the second by exactly the reverse.
Does more data always fix it?
Often yes — it's the most reliable cure. But what matters is variety, not volume: a thousand near-identical examples give the model the same narrow view of the world as a hundred. Add cases that differ, not copies of ones you have.
Can a finished model like ChatGPT overfit?
Large language models train on so much text that classic overfitting across the whole corpus is rare. But its relative is very much there — memorisation of individual chunks: a model can reproduce a passage that appeared often in training. And there's that benchmark leakage. So the phenomenon doesn't go away — it just looks different.
The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.





