A flaky test - one that now passes, now fails with no code change - is more dangerous than an honestly failed one. A failed one points at a problem; a flaky one undermines trust in the whole suite: the team gets used to rerunning until green and, along with the noise, misses real regressions. A retry in Playwright here is not a cure but a sensor: it reports the test is unstable but does not remove the cause. What is treated is not the retry but what stands behind it.
Playwright itself classifies such tests. If the first attempt failed and the retry passed, the test is marked as flaky - and this is a valuable signal, not a reason to relax. A flaky test must be counted a defect on par with a failed one: surfaced as a separate metric in the report, assigned an owner and a fix deadline. Silently relying on the retry to "pull it through" means accumulating instability that will one day break out at the most inconvenient moment.
Almost every flake has a concrete, identifiable cause. A wrong wait gives a timeout on a locator - the test did not wait for the business state. Shared data fails the scenario only under a parallel run. An animation or overlay intercepts a click. An unstable external network answers 429 or 5xx. A dependence on time or time zone fails the test at night or at the end of the month. The diagnosis almost always comes down to one of these categories.
It helps to gather the causes, symptoms and solutions into one table once, so that on a flake you go by diagnosis rather than guess. Below is a map: from the symptom you find the likely cause and a targeted solution. You return to it each time a test behaves unstably: first classify the symptom, then apply the corresponding treatment, rather than hanging a retry over an un-understood failure.
| Cause | Symptom | Solution |
|---|---|---|
| Wrong wait | Timeout on a locator | Wait for the business state (web-first) |
| Shared data | Fails only in parallel | Namespace by worker |
| Animation / overlay | Click intercepted | Wait for actionability, fix the UI |
| External network | 429 / 5xx |
| A controlled boundary, sandbox |
| Time / TZ | Fails at night or in CI | page.clock, a fixed date |
|---|
Each cause has its own treatment, and it is almost always removing non-determinism rather than masking it. A wrong wait is fixed by switching to a web-first check of the business state. Shared data - a namespace by worker. An intercepted click - waiting for actionability or fixing the overlay. An unstable network - moving to a controlled boundary. A time dependence - fixing the clock via page.clock. The symptom points directly at the tool from previous chapters.
The method of fighting flakiness is the same as in debugging: reproduce, then fix. Non-determinism is reproduced deterministically - by fixing data, order, the clock - and only after catching a stable failure do you remove its source. A flake defeated by reproduction does not return; a flake smeared over with a retry has merely hidden. So a retry is kept as insurance against rare infrastructure hiccups, not as a way to avoid understanding the cause.
A retry has a narrow legitimate role. On CI a couple of retries are justified against truly external, unremovable instability - a network blip between data centers - so one fluke does not fail the whole pipeline. But this is quarantine, not treatment: a test that regularly passes only on the second attempt remains a defect queued for a fix, not "working". A retry buys time but does not close the problem.
The typical failures in fighting flakiness come down to putting up with it. Rerunning the run until green instead of investigating the cause. Hanging a retry on an unstable test, hiding a data race or a wrong wait. Leaving a dependence on real time and being surprised by failures in CI with a different time zone. Every flake has a cause from a clear list - find it by the symptom and remove it with a tool from the book, rather than muffling it with a rerun.