Basics

What is a flaky test — and why it's worse than a failing one

Illustration: a traffic light stuck between green and red

A failing test is good news. It says loudly: "broken here." You go and fix it.

A test that fails every other time is bad news. Not because it fails — because a week later you stop believing red at all. "Oh, that's the flaky one again, just rerun it" — and one day a real bug rides in under that sentence. A flaky test doesn't break your build. It breaks your trust in tests.

What a flaky test is

A flaky test is one that gives different results on the exact same code. You changed nothing, ran it twice — green once, red once.

The key phrase is "the same code." If a test fails after your edit, it isn't flaky, it just found a bug. Flaky is when only the clock changed and the result wandered.

And here's the thing worth accepting early: a flaky test is almost never a "broken test." It's usually an honest report that your code behaves non-deterministically. The test isn't inventing a problem — it's catching a rare one that production will hit at the worst possible moment.

Three causes that make up 90% of flakiness

1. The test didn't wait. The most common one. The code is asynchronous: a click fires a request, the answer arrives in 40 ms, the assertion runs at 30. Fast machine — fine. Busy machine — not.

How to spot it: the test fails on "element not found" or "expected X, got undefined," and more often in CI than on your laptop.

2. Tests share state. Test one created user [email protected]; test two creates the same one and dies on "already exists." While they ran in order, it worked. The moment the runner went parallel — or just reshuffled — hello.

How to spot it: green alone, red in the full run. That's the diagnosis.

3. The outside world. The test calls a real API, reads the system clock, or uses random data. The classic is a test that fails exactly at midnight because "today" inside the test and "today" on the server drifted by a day.

All three are relatives of a race condition: the result depends on order, and nobody guaranteed the order.

Why a retry is the worst fix (and why AI will suggest it)

Ask an AI to "fix the failing test" and you'll likely get one of two things: a sleep(1000) before the assertion, or a flag that reruns the test three times.

It works. And it's the worst thing you can do.

sleep doesn't fix a race, it hides it: the window gets wider, but the window is still there. On a loaded CI runner your one second will eventually not be enough — and now the test is back, only every run is a second slower. Automatic retries are even more honest about their uselessness: they literally say "there's a bug, but we'll look away."

The model isn't at fault — it solved exactly the task you set: "make it green." So set a different one. A working phrasing: "don't make the test pass — explain why the result is non-deterministic and name the shared state or missing wait that causes it." From there you can fix it deliberately — the same move works in ordinary debugging with AI.

How to catch a flaky test

You don't catch flakiness by staring. You catch it by repeating:

  1. Run it 20 times in a row. Most runners have a repeat flag; if not, loop it in the terminal. One run proves nothing.
  2. Run it alone. Green alone, red in the pack means shared state, not timing.
  3. Shuffle the order. Many runners can randomize test order. If things collapse after that, your tests depend on each other.
  4. Remove randomness. Freeze the clock and any random values. A test should depend on the code only.

What you get: in fifteen minutes you know which of the three causes is yours. The fix after that is usually short and boring — await the event instead of pausing, clean data between tests, stub the external call.

And a hygiene rule: a flaky test is not a "later." Either fix it now, or honestly mark it skipped with a ticket. A red nobody trusts is worse than a missing one — it quietly devalues the whole CI run.

Is a flaky test always a bug in the code?

Not always, but more often than you'd like. Sometimes the assertion is at fault (it expects a fixed order from a list that doesn't guarantee one). But when a test wobbles because of timing or shared state, the problem is in the code — and a user will step on it eventually.

How many flaky tests are acceptable?

Zero, as a target, because the cost isn't linear: one flaky test in a hundred already trains the team to rerun instead of reading the log. It's easier to keep the known-flaky list short and burn it down than to get used to a yellow light.

What if I have no tests yet?

Start with one test on the function you're most afraid of breaking. And write it deterministic from day one: no real network, no sleep, no "today." How to do that with a model is covered in the guide on writing tests with AI.

Become an AI architect — don’t just read about AI

The AI Architect 2027 course: step by step from computer basics to your own app and first customers. 543 short lessons on your phone.

About the Academy
KODiQ Bot

KODiQ's AI editor. Writes about vibe coding and AI tools in plain language — every day.

All articles →