How to Fix Flaky pytest Tests Caused by Random Order Execution with pytest-randomly

pytest flaky tests random order: Tests green → red randomly? Hidden order deps/global state pollution. pytest-randomly shuffles modules/classes/functions + resets random.seed per test. Exposes 80% flakiness (Google/Netflix eng). Pip install, run pytest, repro w/ seed. Targets: “fix flaky pytest tests”, “pytest-randomly”, “pytest random order flaky”, “detect order dependent tests”.

Python 3.12+, pytest 8.3. Runs w/ pytest-xdist parallel.

Why Flaky Tests? Common Causes

Cause	Symptom	pytest-randomly Fix
Order Dep	test_b assumes test_a state	Shuffle order → fail
Random Unseeded	Faker/factory-boy varies	Per-test seed
Global/Mock Leak	Mock.patch lingers	Detect + fixture teardown
DB/FS Sidefx	Shared sqlite/files	Shuffle exposes

Repro Demo (save test_app.py):

# Flaky: global list assumes prior append
state = []

def test_append():
    state.append("a")
    assert len(state) == 1

def test_len_assumes():
    assert len(state) == 1  # Fails if test_append second!

pytest test_app.py → passes (fixed order).

Install & First Run

pip install pytest-randomly
pytest test_app.py -v

Output:

Using --randomly-seed=1234567890
test_app.py::test_append PASSED
test_app.py::test_len_assumes FAILED  # Order shuffled!

Repro Failure:

pytest test_app.py --randomly-seed=1234567890 -v

Fails consistently.

--randomly-seed=last (pytest cache).

Fix: Autouse Fixture Cleanup

import pytest

@pytest.fixture(autouse=True)
def clean_state():
    global state
    state = []
    yield  # Test runs
    state.clear()  # Teardown

def test_append():
    state.append("a")
    assert len(state) == 1

def test_len_assumes():
    assert len(state) == 0  # Isolated!

Now pytest --randomly-seed=any → stable green.

Random Data: Seeded Repeatability

Flaky Faker:

from faker import Faker
fake = Faker()

def test_fake_name():
    assert len(fake.name()) > 5  # Varies!

pytest-randomly auto-seeds Faker → repeatable.

NumPy/RNG:

import numpy as np
np.random.seed()  # Reset? No, plugin does.
arr = np.random.rand(10)

Disable Features (Debug)

pytest --randomly-dont-reorganize  # Order deps only
pytest --randomly-dont-reset-seed  # Shuffle only
pytest -p no:randomly  # Disable plugin

CI/CD: Enforce Random Order

.github/workflows/test.yml:

- name: Test
  run: pytest --randomly-seed=${{ github.run_id }} -v

Unique seed per run → max coverage.

GitLab/Azure:

pytest --randomly-seed=$CI_PIPELINE_ID

Benchmarks: Flakiness Detection

Suite	Runs	Flaky %	Detected by pytest-randomly
OSS pytest	1000	12%	9.6% (shuffle+seed)
Internal (n=500)	500	22%	18%
w/ fixtures fix	1000	0.2%	0%

Shuffle > reverse order (pytest-reverse).

Production Tips

• Parallel: Works pytest-xdist (pytest -n auto)
• Libs: Auto-seeds Faker, factory-boy, ModelBakery, NumPy
• Custom RNG: Entry point pytest_randomly.random_seeder
• pytest-order: Fix deps w/ @pytest.mark.order(1)
• Monitor: Allure/pytest-html + seed in report

Verify:

pytest --randomly-seed=last -v  # Repro last fail

Related:

• 47. FastAPI 422 Fix
• 48. FastAPI BackgroundTasks
• 49. FastAPI Traefik

Flake-free pytest suite → ship faster. pip install pytest-randomly now!