What is a flaky test — and why it's worse than a failing one

A failing test is good news. It says loudly: "broken here." You go and fix it.
A test that fails every other time is bad news. Not because it fails — because a week later you stop believing red at all. "Oh, that's the flaky one again, just rerun it" — and one day a real bug rides in under that sentence. A flaky test doesn't break your build. It breaks your trust in tests.
What a flaky test is
A flaky test is one that gives different results on the exact same code. You changed nothing, ran it twice — green once, red once.
The key phrase is "the same code." If a test fails after your edit, it isn't flaky, it just found a bug. Flaky is when only the clock changed and the result wandered.
And here's the thing worth accepting early: a flaky test is almost never a "broken test." It's usually an honest report that your code behaves non-deterministically. The test isn't inventing a problem — it's catching a rare one that production will hit at the worst possible moment.
Three causes that make up 90% of flakiness
1. The test didn't wait. The most common one. The code is asynchronous: a click fires a request, the answer arrives in 40 ms, the assertion runs at 30. Fast machine — fine. Busy machine — not.
How to spot it: the test fails on "element not found" or "expected X, got undefined," and more often in CI than on your laptop.
2. Tests share state. Test one created user [email protected]; test two creates the same one and dies on "already exists." While they ran in order, it worked. The moment the runner went parallel — or just reshuffled — hello.
How to spot it: green alone, red in the full run. That's the diagnosis.
3. The outside world. The test calls a real API, reads the system clock, or uses random data. The classic is a test that fails exactly at midnight because "today" inside the test and "today" on the server drifted by a day.
All three are relatives of a race condition: the result depends on order, and nobody guaranteed the order.
Why a retry is the worst fix (and why AI will suggest it)
Ask an AI to "fix the failing test" and you'll likely get one of two things: a sleep(1000) before the assertion, or a flag that reruns the test three times.
It works. And it's the worst thing you can do.
sleep doesn't fix a race, it hides it: the window gets wider, but the window is still there. On a loaded CI runner your one second will eventually not be enough — and now the test is back, only every run is a second slower. Automatic retries are even more honest about their uselessness: they literally say "there's a bug, but we'll look away."
The model isn't at fault — it solved exactly the task you set: "make it green." So set a different one. A working phrasing: "don't make the test pass — explain why the result is non-deterministic and name the shared state or missing wait that causes it." From there you can fix it deliberately — the same move works in ordinary debugging with AI.
How to catch a flaky test
You don't catch flakiness by staring. You catch it by repeating:
- Run it 20 times in a row. Most runners have a repeat flag; if not, loop it in the terminal. One run proves nothing.
- Run it alone. Green alone, red in the pack means shared state, not timing.
- Shuffle the order. Many runners can randomize test order. If things collapse after that, your tests depend on each other.
- Remove randomness. Freeze the clock and any random values. A test should depend on the code only.
What you get: in fifteen minutes you know which of the three causes is yours. The fix after that is usually short and boring — await the event instead of pausing, clean data between tests, stub the external call.
And a hygiene rule: a flaky test is not a "later." Either fix it now, or honestly mark it skipped with a ticket. A red nobody trusts is worse than a missing one — it quietly devalues the whole CI run.
Is a flaky test always a bug in the code?
Not always, but more often than you'd like. Sometimes the assertion is at fault (it expects a fixed order from a list that doesn't guarantee one). But when a test wobbles because of timing or shared state, the problem is in the code — and a user will step on it eventually.
How many flaky tests are acceptable?
Zero, as a target, because the cost isn't linear: one flaky test in a hundred already trains the team to rerun instead of reading the log. It's easier to keep the known-flaky list short and burn it down than to get used to a yellow light.
What if I have no tests yet?
Start with one test on the function you're most afraid of breaking. And write it deterministic from day one: no real network, no sleep, no "today." How to do that with a model is covered in the guide on writing tests with AI.
The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.





