Why does my test pass sometimes and fail other times?

Question

One of my tests passes locally but intermittently fails in CI without code changes. How do I identify the source of the flakiness instead of adding retries?

IT-QA Assistant · Accepted Answer

*AI-drafted answer — reviewed for correctness. Please verify against your own environment and versions; corrections and better answers are welcome below.*

A flaky test usually depends on uncontrolled state, timing, or environment differences. First capture the failure, random seed if applicable, test order, runtime version, timezone, and relevant logs. Compare repeated isolated runs with full-suite runs; failures only in the suite often indicate leaked state.

Check these common causes:

- **Unfinished asynchronous work:** await promises and explicit readiness conditions instead of sleeping for a guessed duration.
- **Shared state:** reset mocks and globals, clean database fixtures, and give parallel tests separate files, ports, and records.
- **Real time or randomness:** inject a clock and random source, or use supported fake timers and a recorded seed.
- **External dependencies:** replace network calls in unit tests and control services used by integration tests.
- **Unspecified ordering:** add SQL `ORDER BY` when ordering matters; otherwise compare results without assuming an order.

For a pytest test, repeated runs can help reproduce the failure:

```bash
for i in {1..30}; do
  python -m pytest tests/test_orders.py::test_total -q || break
done
```

This loop uses Bash or Zsh brace expansion. Reproduce CI's parallelism too. Retries can temporarily reduce disruption, but they should not replace investigating a race that may also exist in production.

Why does my test pass sometimes and fail other times?

1 Answer

Your answer