Test Automation
Flaky Tests: Root Causes and How to Actually Fix Them
A flaky test is a test that passes and fails on the same code, with no change to the code or to the test. Run it ten times against one commit and you get both results.
A test suite where a handful of tests fail "sometimes" doesn't stay a handful for long. Once a team learns to re-run the pipeline until it's green, they stop reading failures at all, including the real ones.
Flakiness isn't a minor inconvenience. It's the thing that quietly kills trust in automation.
Timing, not the application, is usually the cause
Most flaky tests aren't finding real bugs intermittently. They're asserting before the application has finished changing.
- Fixed
sleep()calls pass on a fast CI runner and fail on a loaded one. Replace them with explicit waits on a condition: an element visible, a network request resolved, a state value set. - Animations and transitions leave elements clickable-but-moving. Wait for the animation to finish, or disable non-essential animations in the test environment entirely.
- Network race conditions. A test that clicks "submit" and immediately asserts on a result that depends on an async request. Wait on the request itself, not a fixed delay that approximates how long it usually takes.
Re-running the pipeline until it goes green?
That habit trains a team to stop reading failures at all, including the real ones. We trace flakiness back to its actual root cause instead of adding another retry.
Shared state between tests
Another common cause: tests that pass in isolation and fail in a full run, because an earlier test left data behind.
- Give every test its own fixtures: a freshly created user, not a shared "test@example.com" that fifty other tests also mutate.
- Tear down what you create, or run against a database that resets between runs.
- Never depend on test execution order. If test B only passes because test A ran first, that's not a passing suite, it's a hidden dependency waiting to break in parallel execution.
More on this in test data management.
Environment differences
A test that's stable locally and flaky in CI is usually exposing a real difference in the environment, not a random glitch: a slower disk, less memory, a different timezone, or a CI runner that's genuinely under more contention than a developer's laptop.
Chasing "flakiness" without checking resource limits and environment parity wastes time on the wrong layer.
How to find flaky tests
You can't fix flakiness you haven't measured. These habits make it visible:
- Rerun failures on the same commit. If a failed test passes on retry with no code change, it's flaky, not fixed. Playwright marks these tests as "flaky" in its report when retries are enabled, and Cypress records each retried attempt in its results.
- Keep per-test history. A test that failed 4 times in the last 100 runs on unchanged code is flaky even if it's green today. Most CI dashboards and test reporting tools can show this.
- Stress suspicious tests. Run one test many times in a row, for example
with Playwright's
--repeat-each=50, to reproduce a failure that only shows up once in a while.
Quarantine, but track it
Marking a known-flaky test as quarantined so it doesn't block a release is a reasonable stopgap. Leaving it quarantined indefinitely is not. Every quarantined test needs an owner and a deadline, or it becomes permanent dead weight that slowly makes your coverage numbers meaningless. See our post on testing HTML tables in Cypress for a concrete example of tracking down flake in a specific, high-flake UI pattern.
If your suite has reached the point where nobody trusts red builds anymore, that's usually a sign it needs a structural pass, not another retry mechanism. That's the kind of audit we run as part of test automation engagements.