π‘οΈ Taming Flakiness & Protecting CI/CD Pipeline Gates
In large-scale continuous deployment pipelines, transient environmental friction (such as momentary network latency spikes, third-party gateway lag, or database deadlocks) can trigger intermittent test failures known as flaky tests. Left unmanaged, false-positive test failures block releases, erode developer trust, and prompt teams to ignore automated test results. Playwright paired with Pytest provides an enterprise resilience model: combining automated failure reruns via pytest-rerunfailures, isolating unstable features into dedicated Quarantine test suites that run asynchronously without blocking production deployments, and configuring Playwright's trace recorder to capture forensics on-first-retry to debug root causes without disk bloat.
MAIN BLOCKING CI PIPELINE (Blocks Pull Request / Deployment Merge):
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Executes Production-Critical Suites: pytest -m "not quarantine" β
β βββ Test Run 1 βββΊ PASS β
(Continues immediately) β
β βββ Test Run 1 βββΊ FAIL β β
β β β
β βΌ (Automatic Re-execution via pytest-rerunfailures) β
β Retry Run 2 (Tracer starts recording DOM snapshots & network calls) β
β βββ PASS β
βββΊ Marked as "FLAKY PASS" (Alert logged, build green) β
β βββ FAIL β βββΊ True Bug Confirmed (Pipeline Blocks, trace.zip saved)β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
NON-BLOCKING QUARANTINE SUITE (Runs in Parallel / Nightly Build):
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Executes Unstable / Bug-Tracked Tests: pytest -m "quarantine" β
β * Tagged with issue tracking metadata (@pytest.mark.quarantine(issue=...)) β
β * Failures log warning metrics to Jira / Slack; NEVER block release gates β
β * Graduated back to core suite once 100 consecutive green runs pass β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
pytest-rerunfailures and run suites using CLI parameters: specifying retry attempt counts and adding delay intervals between retry attempts to allow services to recover.
# 1. Install pytest-rerunfailures plugin
pip install pytest-rerunfailures
# 2. Rerun failing tests up to 2 times automatically before marking as failed
pytest --reruns 2
# 3. Rerun up to 2 times with a 1-second delay between attempts (allows network to settle)
pytest --reruns 2 --reruns-delay 1
# 4. Filter reruns only on specific network timeout exceptions (avoids retrying logic bugs)
pytest --reruns 2 --only-rerun TimeoutError
# 5. Combine reruns with parallel workers and video capture on retry
pytest -n auto --reruns 2 --video on-first-retry --tracing retain-on-failure
@pytest.mark.flaky decorator to target specific known volatile endpoints (such as third-party payment gateways) without enabling retries globally across the entire test suite.
import pytest
from playwright.sync_api import Page, expect
# Apply retries ONLY to this specific volatile third-party interaction
@pytest.mark.flaky(reruns=2, reruns_delay=1)
def test_third_party_payment_authorization(page: Page):
"""Simulates a payment flow that occasionally suffers from external gateway latency."""
page.goto("https://httpbin.org/status/200")
# Simulate flaky network condition
page.set_content("""
Idle
""")
page.locator("#pay-btn").click()
status_locator = page.locator("#status")
# First run might see 'GATEWAY_TIMEOUT' and fail; retry re-evaluates and passes!
expect(status_locator).to_have_text("GATEWAY_APPROVED")
print(f"Payment Authorization Succeeded: '{status_locator.inner_text()}'")
conftest.py. Exclude them from blocking CI builds while running them in non-blocking scheduled verification pipelines.
# content of pytest.ini
[pytest]
markers =
quarantine: Unstable or bug-blocked tests isolated from blocking merge pipelines.
flaky: Tests with targeted automatic retry parameters.
# content of test_inventory_ledger.py
import pytest
from playwright.sync_api import Page, expect
# Stable test running in primary CI
def test_active_inventory_catalog_view(page: Page):
page.goto("https://example.com")
expect(page.locator("h1")).to_have_text("Example Domain")
# Quarantined test tied to an active Jira bug ticket
@pytest.mark.quarantine(reason="Blocked by JIRA-8841: Redis cache sync delay", defect="JIRA-8841")
def test_realtime_stock_count_refresh(page: Page):
page.goto("https://example.com")
# This known failure runs in quarantine without failing the primary pipeline
expect(page.locator("#stock-counter")).to_have_text("100 Units Available")
# 1. PRIMARY BLOCKING BUILD (Runs on every PR; strictly excludes quarantined tests)
pytest -m "not quarantine" --reruns 1
# 2. NON-BLOCKING QUARANTINE JOB (Runs scheduled nightly to track defect fixes)
pytest -m "quarantine" --continue-on-collection-errors || true
# content of conftest.py
import os
import pytest
from playwright.sync_api import Browser, BrowserContext, Page
@pytest.fixture
def resilient_tracing_page(request, browser: Browser) -> Page:
"""Activates Playwright Tracing dynamically when a test is undergoing a retry."""
# pytest-rerunfailures sets execution_count on the test node
execution_count = getattr(request.node, "execution_count", 1)
is_retry = execution_count > 1
context: BrowserContext = browser.new_context()
if is_retry:
print(f"\n[Trace Trigger] Test '{request.node.name}' is on retry #{execution_count}. Starting full trace capture...")
context.tracing.start(screenshots=True, snapshots=True, sources=True)
page: Page = context.new_page()
yield page
if is_retry:
artifacts_dir = "test-results/retry_traces"
os.makedirs(artifacts_dir, exist_ok=True)
trace_file = os.path.join(artifacts_dir, f"{request.node.name}_retry_{execution_count}.zip")
context.tracing.stop(path=trace_file)
print(f"[Trace Saved] Saved forensic retry trace to: {trace_file}")
context.close()
# content of conftest.py
import pytest
def pytest_collection_modifyitems(items):
"""Automatically marks quarantined tests as xfail so failures do not break builds."""
for item in items:
quarantine_marker = item.get_closest_marker("quarantine")
if quarantine_marker:
reason = quarantine_marker.kwargs.get("reason", "Quarantined test")
defect = quarantine_marker.kwargs.get("defect", "UNTRACKED")
# Convert to Pytest expected failure
item.add_marker(pytest.mark.xfail(
reason=f"[QUARANTINE - {defect}] {reason}",
strict=False # If it unexpectedly passes (XPASS), still keep build green!
))
# content of test_demo_quarantine.py
from playwright.sync_api import Page, expect
@pytest.mark.quarantine(reason="Known backend schema bug", defect="PROD-404")
def test_known_failing_checkout_widget(page: Page):
page.goto("https://example.com")
expect(page.locator("#missing-checkout")).to_be_visible()
AssertionError: 'Cart' != 'Checkout') should fail immediately without wasting time on retries.
# 1. Rerun ONLY when Playwright raises a TimeoutError (Transient network/DOM delay)
pytest --reruns 2 --only-rerun TimeoutError
# 2. Exclude AssertionError from reruns (Deterministic assertion bugs fail instantly!)
pytest --reruns 2 --exclude-rerun AssertionError
# 3. Specify multiple transient exceptions using regular expressions
pytest --reruns 2 --only-rerun "(TimeoutError|ConnectionResetError|ServerDisconnectedError)"
# content of conftest.py
import pytest
flaky_tests_detected = []
@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
outcome = yield
report = outcome.get_result()
# Track tests that had a rerun and eventually passed
if report.when == "call":
execution_count = getattr(item, "execution_count", 1)
if execution_count > 1 and report.passed:
flaky_tests_detected.append({
"test_name": item.nodeid,
"attempts": execution_count
})
def pytest_sessionfinish(session, exitstatus):
"""Executes after all tests complete to broadcast flaky test metrics."""
if flaky_tests_detected:
print("\n" + "=" * 60)
print("π¨ FLAKY TEST TELEMETRY REPORT (Passed on Retry)")
print("=" * 60)
for entry in flaky_tests_detected:
print(f" β’ {entry['test_name']} (Required {entry['attempts']} attempts)")
print("=" * 60)
print(f"Total Flaky Tests Detected: {len(flaky_tests_detected)}")
# In real CI: send payload to Datadog, Slack, or Elastic webhook!
name: Playwright Resilient Test Suite
on: [push, pull_request]
jobs:
core-test-suite:
name: Core Regression Suite (Blocking)
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install dependencies
run: pip install -r requirements.txt
- name: Install Playwright Browsers
run: playwright install --with-deps chromium
- name: Run Core Suite with Retries
run: |
pytest -n auto -m "not quarantine" \
--reruns 2 \
--only-rerun TimeoutError \
--tracing retain-on-failure \
--output test-results/
- name: Upload Test Traces on Failure
if: failure()
uses: actions/upload-artifact@v4
with:
name: core-failure-traces
path: test-results/
quarantine-suite:
name: Quarantined Tests (Non-Blocking)
runs-on: ubuntu-latest
continue-on-error: true # Guarantees quarantine failures do NOT fail PR checks!
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- run: pip install -r requirements.txt && playwright install --with-deps chromium
- name: Execute Quarantined Tests
run: pytest -m "quarantine" --tracing on
FAILURE RERUNS & TEST QUARANTINE MATRIX:
ββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Strategy / Parameter β Example Usage β Architectural Purpose / Primary Benefit β
ββββββββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β --reruns <count> β pytest --reruns 2 β Re-executes failed tests up to N times before failing. β
β --reruns-delay <sec> β pytest --reruns-delay 1 β Injects delay between retry attempts for system recoveryβ
β --only-rerun <Exception> β --only-rerun TimeoutError β Restricts retries strictly to transient network issues. β
β --exclude-rerun <Exception> β --exclude-rerun AssertionError Direct logic bugs fail immediately on first attempt. β
β @pytest.mark.flaky(reruns=N) β Function-level decorator β Isolates retries to specific volatile endpoints. β
β @pytest.mark.quarantine β Custom Pytest marker β Tags tests tied to open bug tickets. β
β pytest -m "not quarantine" β Primary CI execution flag β Excludes unstable tests from blocking merge gates. β
β continue-on-error: true β CI Step configuration β Runs quarantine tests asynchronously without red builds.β
β --tracing retain-on-failure β Playwright CLI flag β Captures trace ZIP forensics only when test fails. β
β --video on-first-retry β Playwright CLI flag β Records video evidence strictly during retry runs. β
ββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β ENTERPRISE QUARANTINE RULES: β
β 1. Never leave a test in quarantine indefinitely: every quarantined test must be linked to an active issue ticket. β
β 2. Graduate tests out of quarantine only after 100 consecutive successful runs on scheduled nightly builds. β
β 3. Retries should never mask true bugs: always alert and monitor tests that require retries to pass. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ