FAILURE RERUNS, FLAKY MANAGEMENT & TEST QUARANTINE

Master enterprise resilience and pipeline stability in Playwright with Python: automated test retries with pytest-rerunfailures, targeted flaky markers, non-blocking quarantine test buckets, dynamic artifact retention across retry cycles, and defect tracking integrations.

πŸ›‘οΈ Taming Flakiness & Protecting CI/CD Pipeline Gates

In large-scale continuous deployment pipelines, transient environmental friction (such as momentary network latency spikes, third-party gateway lag, or database deadlocks) can trigger intermittent test failures known as flaky tests. Left unmanaged, false-positive test failures block releases, erode developer trust, and prompt teams to ignore automated test results. Playwright paired with Pytest provides an enterprise resilience model: combining automated failure reruns via pytest-rerunfailures, isolating unstable features into dedicated Quarantine test suites that run asynchronously without blocking production deployments, and configuring Playwright's trace recorder to capture forensics on-first-retry to debug root causes without disk bloat.

1. Flaky Triage Architecture: Retries vs. Quarantine Isolation
A production-ready test automation strategy balances automated retries for transient environmental hiccups with strict quarantine buckets for known product defects.
Enterprise Resilience & Quarantine Pipeline
MAIN BLOCKING CI PIPELINE (Blocks Pull Request / Deployment Merge):
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Executes Production-Critical Suites: pytest -m "not quarantine"               β”‚
β”‚  β”œβ”€β”€ Test Run 1 ──► PASS βœ… (Continues immediately)                          β”‚    
β”‚  └── Test Run 1 ──► FAIL ❌                                                  β”‚       
β”‚         β”‚                                                                     β”‚
β”‚         β–Ό (Automatic Re-execution via pytest-rerunfailures)                   β”‚ 
β”‚      Retry Run 2 (Tracer starts recording DOM snapshots & network calls)      β”‚
β”‚         β”œβ”€β”€ PASS βœ… ──► Marked as "FLAKY PASS" (Alert logged, build green)   β”‚  
β”‚         └── FAIL ❌ ──► True Bug Confirmed (Pipeline Blocks, trace.zip saved)β”‚  
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

NON-BLOCKING QUARANTINE SUITE (Runs in Parallel / Nightly Build):
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Executes Unstable / Bug-Tracked Tests: pytest -m "quarantine"               β”‚
β”‚  * Tagged with issue tracking metadata (@pytest.mark.quarantine(issue=...)) β”‚
β”‚  * Failures log warning metrics to Jira / Slack; NEVER block release gates  β”‚
β”‚  * Graduated back to core suite once 100 consecutive green runs pass        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Strategic Rule: Automated retries absorb transient network blips; Quarantine isolate known product or third-party bugs to prevent false-alarm release stoppages.
Real-Time Example 1: Installing & Configuring pytest-rerunfailures
Install pytest-rerunfailures and run suites using CLI parameters: specifying retry attempt counts and adding delay intervals between retry attempts to allow services to recover.
Terminal / Shell CLI Commands
# 1. Install pytest-rerunfailures plugin
pip install pytest-rerunfailures

# 2. Rerun failing tests up to 2 times automatically before marking as failed
pytest --reruns 2

# 3. Rerun up to 2 times with a 1-second delay between attempts (allows network to settle)
pytest --reruns 2 --reruns-delay 1

# 4. Filter reruns only on specific network timeout exceptions (avoids retrying logic bugs)
pytest --reruns 2 --only-rerun TimeoutError

# 5. Combine reruns with parallel workers and video capture on retry
pytest -n auto --reruns 2 --video on-first-retry --tracing retain-on-failure
Typical Terminal Execution Output: test_catalog.py::test_dynamic_pricing RERUN [gw0] test_catalog.py::test_dynamic_pricing PASSED [gw0] 1 passed, 1 rerun in 2.34s (Pipeline remains green!)
Real-Time Example 2: Function-Level @pytest.mark.flaky Decorators
Apply the @pytest.mark.flaky decorator to target specific known volatile endpoints (such as third-party payment gateways) without enabling retries globally across the entire test suite.
Python (Targeted Function Flaky Marker)
import pytest
from playwright.sync_api import Page, expect

# Apply retries ONLY to this specific volatile third-party interaction
@pytest.mark.flaky(reruns=2, reruns_delay=1)
def test_third_party_payment_authorization(page: Page):
    """Simulates a payment flow that occasionally suffers from external gateway latency."""
    page.goto("https://httpbin.org/status/200")
    
    # Simulate flaky network condition
    page.set_content("""
        
        
Idle
""") page.locator("#pay-btn").click() status_locator = page.locator("#status") # First run might see 'GATEWAY_TIMEOUT' and fail; retry re-evaluates and passes! expect(status_locator).to_have_text("GATEWAY_APPROVED") print(f"Payment Authorization Succeeded: '{status_locator.inner_text()}'")
Output: test_payment.py::test_third_party_payment_authorization RERUN Payment Authorization Succeeded: 'GATEWAY_APPROVED' test_payment.py::test_third_party_payment_authorization PASSED
Real-Time Example 3: Defining and Managing the Quarantine Suite
Quarantine tests associated with open bug tickets using a custom marker in conftest.py. Exclude them from blocking CI builds while running them in non-blocking scheduled verification pipelines.
Configuration & Python (Quarantine Marker Setup)
# content of pytest.ini
[pytest]
markers =
    quarantine: Unstable or bug-blocked tests isolated from blocking merge pipelines.
    flaky: Tests with targeted automatic retry parameters.

# content of test_inventory_ledger.py
import pytest
from playwright.sync_api import Page, expect

# Stable test running in primary CI
def test_active_inventory_catalog_view(page: Page):
    page.goto("https://example.com")
    expect(page.locator("h1")).to_have_text("Example Domain")

# Quarantined test tied to an active Jira bug ticket
@pytest.mark.quarantine(reason="Blocked by JIRA-8841: Redis cache sync delay", defect="JIRA-8841")
def test_realtime_stock_count_refresh(page: Page):
    page.goto("https://example.com")
    # This known failure runs in quarantine without failing the primary pipeline
    expect(page.locator("#stock-counter")).to_have_text("100 Units Available")
CI Execution Commands
# 1. PRIMARY BLOCKING BUILD (Runs on every PR; strictly excludes quarantined tests)
pytest -m "not quarantine" --reruns 1

# 2. NON-BLOCKING QUARANTINE JOB (Runs scheduled nightly to track defect fixes)
pytest -m "quarantine" --continue-on-collection-errors || true
Pipeline Result: 150 tests passed in primary build (0 PR blocks). Quarantined defect JIRA-8841 runs asynchronously without stopping development velocity.
Real-Time Example 4: Playwright on-first-retry Trace & Video Retention
Recording video and full traces on every passing test causes excessive disk usage and slows execution. Configure Playwright to record artifacts strictly on-first-retry to capture forensics only when needed.
Python (conftest.py Dynamic on-first-retry Harness)
# content of conftest.py
import os
import pytest
from playwright.sync_api import Browser, BrowserContext, Page

@pytest.fixture
def resilient_tracing_page(request, browser: Browser) -> Page:
    """Activates Playwright Tracing dynamically when a test is undergoing a retry."""
    # pytest-rerunfailures sets execution_count on the test node
    execution_count = getattr(request.node, "execution_count", 1)
    is_retry = execution_count > 1

    context: BrowserContext = browser.new_context()

    if is_retry:
        print(f"\n[Trace Trigger] Test '{request.node.name}' is on retry #{execution_count}. Starting full trace capture...")
        context.tracing.start(screenshots=True, snapshots=True, sources=True)

    page: Page = context.new_page()
    yield page

    if is_retry:
        artifacts_dir = "test-results/retry_traces"
        os.makedirs(artifacts_dir, exist_ok=True)
        trace_file = os.path.join(artifacts_dir, f"{request.node.name}_retry_{execution_count}.zip")
        context.tracing.stop(path=trace_file)
        print(f"[Trace Saved] Saved forensic retry trace to: {trace_file}")

    context.close()
Output: [Trace Trigger] Test 'test_order_submission' is on retry #2. Starting full trace capture... [Trace Saved] Saved forensic retry trace to: test-results/retry_traces/test_order_submission_retry_2.zip
Real-Time Example 5: Defect Tracking & Auto-Quarantine Hook in conftest.py
Build a custom Pytest hook that inspects quarantine markers, intercepts failures, and reclassifies them as XFAIL (Expected Failure) so they do not exit with a non-zero exit code.
Python (conftest.py Auto-XFAIL Quarantine Hook)
# content of conftest.py
import pytest

def pytest_collection_modifyitems(items):
    """Automatically marks quarantined tests as xfail so failures do not break builds."""
    for item in items:
        quarantine_marker = item.get_closest_marker("quarantine")
        if quarantine_marker:
            reason = quarantine_marker.kwargs.get("reason", "Quarantined test")
            defect = quarantine_marker.kwargs.get("defect", "UNTRACKED")
            # Convert to Pytest expected failure
            item.add_marker(pytest.mark.xfail(
                reason=f"[QUARANTINE - {defect}] {reason}",
                strict=False  # If it unexpectedly passes (XPASS), still keep build green!
            ))

# content of test_demo_quarantine.py
from playwright.sync_api import Page, expect

@pytest.mark.quarantine(reason="Known backend schema bug", defect="PROD-404")
def test_known_failing_checkout_widget(page: Page):
    page.goto("https://example.com")
    expect(page.locator("#missing-checkout")).to_be_visible()
Output: test_demo_quarantine.py::test_known_failing_checkout_widget XFAIL [QUARANTINE - PROD-404] === 1 xfailed in 0.42s (Build passes with exit code 0!) ===
Real-Time Example 6: Identifying Transient vs. Permanent Errors via Exception Filtering
Only transient exceptions (timeouts, socket resets, 502/503 HTTP codes) should be retried. Hard assertion mismatches (e.g. AssertionError: 'Cart' != 'Checkout') should fail immediately without wasting time on retries.
Terminal / Shell CLI Commands
# 1. Rerun ONLY when Playwright raises a TimeoutError (Transient network/DOM delay)
pytest --reruns 2 --only-rerun TimeoutError

# 2. Exclude AssertionError from reruns (Deterministic assertion bugs fail instantly!)
pytest --reruns 2 --exclude-rerun AssertionError

# 3. Specify multiple transient exceptions using regular expressions
pytest --reruns 2 --only-rerun "(TimeoutError|ConnectionResetError|ServerDisconnectedError)"
Efficiency Gain: Pure assertion bugs fail instantly on Run 1, while genuine network timeouts retry and pass, saving minutes of needless retry execution across the suite.
Real-Time Example 7: Flaky Test Telemetry & Reporting to Slack / Webhooks
Capture flaky tests that passed on a retry attempt and send a notification payload to an engineering webhook to prioritize stabilization before tests degrade further.
Python (conftest.py Flaky Telemetry Webhook)
# content of conftest.py
import pytest

flaky_tests_detected = []

@pytest.hookimpl(tryfirst=True, hookwrapper=True)
def pytest_runtest_makereport(item, call):
    outcome = yield
    report = outcome.get_result()

    # Track tests that had a rerun and eventually passed
    if report.when == "call":
        execution_count = getattr(item, "execution_count", 1)
        if execution_count > 1 and report.passed:
            flaky_tests_detected.append({
                "test_name": item.nodeid,
                "attempts": execution_count
            })

def pytest_sessionfinish(session, exitstatus):
    """Executes after all tests complete to broadcast flaky test metrics."""
    if flaky_tests_detected:
        print("\n" + "=" * 60)
        print("🚨 FLAKY TEST TELEMETRY REPORT (Passed on Retry)")
        print("=" * 60)
        for entry in flaky_tests_detected:
            print(f"  β€’ {entry['test_name']} (Required {entry['attempts']} attempts)")
        print("=" * 60)
        print(f"Total Flaky Tests Detected: {len(flaky_tests_detected)}")
        # In real CI: send payload to Datadog, Slack, or Elastic webhook!
Output: ============================================================ 🚨 FLAKY TEST TELEMETRY REPORT (Passed on Retry) ============================================================ β€’ tests/test_orders.py::test_dynamic_pricing (Required 2 attempts) β€’ tests/test_auth.py::test_token_refresh (Required 2 attempts) ============================================================ Total Flaky Tests Detected: 2
Real-Time Example 8: CI Build Matrix Configuration with Failure Reruns
A production GitHub Actions workflow configuration combining parallel test execution, failure reruns, trace retention on failure, and separate non-blocking quarantine steps.
YAML (.github/workflows/playwright_resilience.yml)
name: Playwright Resilient Test Suite

on: [push, pull_request]

jobs:
  core-test-suite:
    name: Core Regression Suite (Blocking)
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - name: Install dependencies
        run: pip install -r requirements.txt
      - name: Install Playwright Browsers
        run: playwright install --with-deps chromium
      - name: Run Core Suite with Retries
        run: |
          pytest -n auto -m "not quarantine" \
            --reruns 2 \
            --only-rerun TimeoutError \
            --tracing retain-on-failure \
            --output test-results/
      - name: Upload Test Traces on Failure
        if: failure()
        uses: actions/upload-artifact@v4
        with:
          name: core-failure-traces
          path: test-results/

  quarantine-suite:
    name: Quarantined Tests (Non-Blocking)
    runs-on: ubuntu-latest
    continue-on-error: true # Guarantees quarantine failures do NOT fail PR checks!
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: pip install -r requirements.txt && playwright install --with-deps chromium
      - name: Execute Quarantined Tests
        run: pytest -m "quarantine" --tracing on
CI Pipeline Architecture: Core regression runs with retries to block regressions. Quarantine suite runs in parallel with continue-on-error: true to gather failure trends without halting releases.
Real-Time Example 9: Failure Reruns & Quarantine Reference Matrix
A comprehensive reference summary of CLI flags, configuration parameters, markers, and best practice rules for managing test failures and quarantine pipelines.
Resilience & Quarantine Reference Matrix
FAILURE RERUNS & TEST QUARANTINE MATRIX:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Strategy / Parameter               β”‚ Example Usage                β”‚ Architectural Purpose / Primary Benefit                 β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ --reruns <count>                   β”‚ pytest --reruns 2            β”‚ Re-executes failed tests up to N times before failing.  β”‚
β”‚ --reruns-delay <sec>               β”‚ pytest --reruns-delay 1      β”‚ Injects delay between retry attempts for system recoveryβ”‚
β”‚ --only-rerun <Exception>           β”‚ --only-rerun TimeoutError    β”‚ Restricts retries strictly to transient network issues. β”‚
β”‚ --exclude-rerun <Exception>        β”‚ --exclude-rerun AssertionError Direct logic bugs fail immediately on first attempt.    β”‚
β”‚ @pytest.mark.flaky(reruns=N)       β”‚ Function-level decorator     β”‚ Isolates retries to specific volatile endpoints.        β”‚
β”‚ @pytest.mark.quarantine            β”‚ Custom Pytest marker         β”‚ Tags tests tied to open bug tickets.                    β”‚
β”‚ pytest -m "not quarantine"         β”‚ Primary CI execution flag    β”‚ Excludes unstable tests from blocking merge gates.      β”‚
β”‚ continue-on-error: true            β”‚ CI Step configuration        β”‚ Runs quarantine tests asynchronously without red builds.β”‚
β”‚ --tracing retain-on-failure        β”‚ Playwright CLI flag          β”‚ Captures trace ZIP forensics only when test fails.      β”‚
β”‚ --video on-first-retry             β”‚ Playwright CLI flag          β”‚ Records video evidence strictly during retry runs.      β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ ENTERPRISE QUARANTINE RULES:                                                                                                β”‚
β”‚ 1. Never leave a test in quarantine indefinitely: every quarantined test must be linked to an active issue ticket.          β”‚
β”‚ 2. Graduate tests out of quarantine only after 100 consecutive successful runs on scheduled nightly builds.                 β”‚
β”‚ 3. Retries should never mask true bugs: always alert and monitor tests that require retries to pass.                        β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Engineering Best Practice: Use --reruns 1 or 2 with --only-rerun TimeoutError in CI to absorb transient network blips. Tag known persistent defect tests with @pytest.mark.quarantine and exclude them from blocking pull request checks until resolved.