HYPOTHESIS TESTING: P-VALUES, T-TESTS, ANOVA & CHI-SQUARE

Master rigorous statistical validation for machine learning and experimentation: Null Hypothesis Significance Testing (NHST), Type I and Type II error bounds, statistical power analysis, Student's t-tests, Welch's t-test, Paired t-tests, One-Way ANOVA, Tukey HSD post-hoc tests, Chi-Square contingency tests, and Bonferroni/FDR multiple testing corrections in Python.

πŸ§ͺ Statistical Significance: Distinguishing True Signal from Random Flukes

In production machine learning and business operations, decisions cannot be made on raw metric deltas alone. A new conversion model achieving a 0.8% lift or an algorithmic trading strategy reporting an increased Sharpe ratio might simply be the artifact of random sample variance. Hypothesis Testing provides the mathematical machinery to determine whether observed differences are statistically genuine or attributable to random sampling noise. Through Null Hypothesis Significance Testing (NHST), data scientists quantify risk via p-values, balance Type I false positives ($\alpha$) against Type II false negatives ($\beta$), size experiments through statistical power ($1 - \beta$), compare continuous targets across cohorts using Independent, Welch's, and Paired t-tests, analyze multi-group variance with One-Way ANOVA and Tukey's HSD, validate categorical dependencies using Chi-Square contingency matrices, and correct for false discovery inflation using Bonferroni and Benjamini-Hochberg (FDR) algorithms.

1. Hypothesis Testing Workflow & Decision Matrix Architecture
The canonical lifecycle of statistical hypothesis testing translates commercial business assumptions into null ($H_0$) and alternative ($H_1$) hypotheses, calculates test statistics, evaluates p-values against significance thresholds, and controls error rates.
Hypothesis Testing Decision Pipeline
HYPOTHESIS TESTING DECISION PIPELINE & ERROR MATRIX:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ 1. FORMULATE HYPOTHESES & SET SIGNIFICANCE LEVEL (Ξ± = 0.05)                 β”‚
β”‚    H0 (Null Hypothesis):        ΞΌ_treatment == ΞΌ_control (No Effect)        β”‚
β”‚    H1 (Alternative Hypothesis): ΞΌ_treatment != ΞΌ_control (Significant Shift)β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                       β”‚
                                       β–Ό 
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ 2. SELECT TEST STATISTIC BASED ON DATA TOPOLOGY                             β”‚
β”‚    β”œβ”€β”€ 2 Continuous Independent Samples (Equal Var)   ──► Student's t-Test  β”‚
β”‚    β”œβ”€β”€ 2 Continuous Independent Samples (Unequal Var) ──► Welch's t-Test    β”‚
β”‚    β”œβ”€β”€ 2 Repeated / Paired Measurements (Before/After) ──► Paired t-Test    β”‚
β”‚    β”œβ”€β”€ 3+ Continuous Group Means                      ──► One-Way ANOVA     β”‚
β”‚    └── Categorical Frequencies & Proportions          ──► Chi-Square (χ²)   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                       β”‚
                                       β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ 3. P-VALUE EVALUATION & DECISION MATRIX                                     β”‚
β”‚    p-value <= Ξ± ──► Reject H0 (Statistically Significant Effect)            β”‚
β”‚    p-value >  Ξ± ──► Fail to Reject H0 (Insufficient Evidence)               β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ DECISION ERROR MATRIX:                                                      β”‚
β”‚                     β”‚ H0 is Actually TRUE          β”‚ H0 is Actually FALSE   β”‚
β”‚ ────────────────────┼──────────────────────────────┼────────────────────────│
β”‚ Reject H0           β”‚ Type I Error (False Alarm: Ξ±)β”‚ Correct (Power: 1 - Ξ²) β”‚
β”‚ Fail to Reject H0   β”‚ Correct (Confidence: 1 - Ξ±)  β”‚ Type II Error (Miss: Ξ²)β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Architectural Standard: A p-value is the probability of observing a test statistic at least as extreme as the empirical sample, assuming the null hypothesis is true. It is NOT the probability that the null hypothesis itself is true.
Real-Time Example 1: Two-Sample Independent Student's t-Test vs. Welch's t-Test
Evaluate conversion revenue between two independent cohorts in an A/B test. Standard Student's t-test assumes equal variances; when sample variances diverge (heteroscedasticity), Welch's t-test adjusts degrees of freedom to avoid severe Type I error inflation.
Python (independent_welch_ttest.py)
import numpy as np
from scipy import stats

np.random.seed(42)

# E-commerce A/B Test: Revenue per checkout (INR)
# Control Group (A): n=120, ΞΌ=1450, Οƒ=320
control_rev = np.random.normal(loc=1450.0, scale=320.0, size=120)

# Treatment Group (B): n=100, ΞΌ=1560, Οƒ=540 (Higher mean and significantly higher variance!)
treatment_rev = np.random.normal(loc=1560.0, scale=540.0, size=100)

# 1. Test for Homogeneity of Variances (Levene's Test)
levene_stat, levene_p = stats.levene(control_rev, treatment_rev)

# 2. Standard Student's t-Test (Assumes equal_var=True)
t_student, p_student = stats.ttest_ind(treatment_rev, control_rev, equal_var=True)

# 3. Welch's t-Test (Robust against unequal variance: equal_var=False)
t_welch, p_welch = stats.ttest_ind(treatment_rev, control_rev, equal_var=False)

# 4. Effect Size: Cohen's d
pooled_std = np.sqrt(
    ((len(control_rev) - 1) * np.var(control_rev, ddof=1) + (len(treatment_rev) - 1) * np.var(treatment_rev, ddof=1))
    / (len(control_rev) + len(treatment_rev) - 2)
)
cohens_d = (np.mean(treatment_rev) - np.mean(control_rev)) / pooled_std

print("=" * 65)
print("INDEPENDENT A/B TEST: STUDENT'S VS. WELCH'S T-TEST")
print("=" * 65)
print(f"Control (A)   : Mean = β‚Ή {np.mean(control_rev):6.2f} | Std = β‚Ή {np.std(control_rev, ddof=1):6.2f} | N = {len(control_rev)}")
print(f"Treatment (B) : Mean = β‚Ή {np.mean(treatment_rev):6.2f} | Std = β‚Ή {np.std(treatment_rev, ddof=1):6.2f} | N = {len(treatment_rev)}")
print("-" * 65)
print(f"Levene's Variance Test p-value : {levene_p:.4e} {'[UNEQUAL VARIANCES]' if levene_p < 0.05 else '[EQUAL VARIANCES]'}")
print(f"Student's t-Test (equal_var=T) : t = {t_student:+.4f} | p-value = {p_student:.4e}")
print(f"Welch's t-Test   (equal_var=F) : t = {t_welch:+.4f} | p-value = {p_welch:.4e} (Robust)")
print(f"Standardized Effect (Cohen's d): {cohens_d:.4f} (Medium Effect Size)")
Output: ================================================================= INDEPENDENT A/B TEST: STUDENT'S VS. WELCH'S T-TEST ================================================================= Control (A) : Mean = β‚Ή 1441.97 | Std = β‚Ή 314.95 | N = 120 Treatment (B) : Mean = β‚Ή 1555.23 | Std = β‚Ή 522.61 | N = 100 ----------------------------------------------------------------- Levene's Variance Test p-value : 1.3418e-05 [UNEQUAL VARIANCES] Student's t-Test (equal_var=T) : t = +1.9868 | p-value = 4.8193e-02 Welch's t-Test (equal_var=F) : t = +1.9189 | p-value = 5.6729e-02 (Robust) Standardized Effect (Cohen's d): 0.2694 (Medium Effect Size)
Real-Time Example 2: Paired / Dependent t-Test for Model Latency Optimization
When the same entity is measured under two conditions (e.g. inference latency before and after neural network quantization on identical batches), the Paired t-test isolates the treatment effect from individual sample baseline variance.
Python (paired_ttest_latency.py)
import numpy as np
from scipy import stats

np.random.seed(42)

# Benchmark 50 benchmark payloads through an NLP Transformer pipeline
# Baseline unquantized model latency (ms) per payload
baseline_latency = np.random.normal(loc=125.0, scale=20.0, size=50)

# Quantized ONNX FP16 runtime on the EXACT same 50 payloads (Reduction with batch correlation)
noise = np.random.normal(loc=0.0, scale=3.0, size=50)
quantized_latency = baseline_latency * 0.82 + noise

# 1. Compute Pairwise Latency Differences: D_i = Baseline_i - Quantized_i
differences = baseline_latency - quantized_latency

# 2. Paired Student's t-Test
t_stat, p_val = stats.ttest_rel(baseline_latency, quantized_latency)

# 3. 95% Confidence Interval for the True Average Speedup
df = len(differences) - 1
se_diff = stats.sem(differences)
t_crit = stats.t.ppf(0.975, df=df)
ci_lower = np.mean(differences) - (t_crit * se_diff)
ci_upper = np.mean(differences) + (t_crit * se_diff)

print("=" * 65)
print("PAIRED T-TEST: MODEL QUANTIZATION LATENCY AUDIT")
print("=" * 65)
print(f"Payload Batches Evaluated   : {len(baseline_latency)}")
print(f"Mean Baseline Latency (FP32): {np.mean(baseline_latency):6.2f} ms")
print(f"Mean Quantized Latency (FP16): {np.mean(quantized_latency):6.2f} ms")
print(f"Mean Pairwise Latency Saving : {np.mean(differences):6.2f} ms (p < 0.001)")
print("-" * 65)
print(f"Paired t-Statistic          : t = {t_stat:+.4f}")
print(f"Two-Tailed p-value          : {p_val:.4e} (Highly Significant)")
print(f"95% CI of Latency Reduction : [{ci_lower:.2f}, {ci_upper:.2f}] ms")
Output: ================================================================= PAIRED T-TEST: MODEL QUANTIZATION LATENCY AUDIT ================================================================= Payload Batches Evaluated : 50 Mean Baseline Latency (FP32): 123.63 ms Mean Quantized Latency (FP16): 101.40 ms Mean Pairwise Latency Saving : 22.23 ms (p < 0.001) ----------------------------------------------------------------- Paired t-Statistic : t = +31.8492 Two-Tailed p-value : 4.1924e-34 (Highly Significant) 95% CI of Latency Reduction : [20.83, 23.63] ms
Real-Time Example 3: One-Way ANOVA (F-Test) for Multi-Cohort Experimentation
Running multiple pairwise t-tests across 3 or more groups severely inflates the Family-Wise Error Rate ($\alpha_{total} = 1 - (1-\alpha)^k$). One-Way Analysis of Variance (ANOVA) evaluates the ratio of between-group variance to within-group variance ($F = \text{MS}_{between} / \text{MS}_{within}$) in a single omnibus test.
Python (anova_multigroup_experiment.py)
import numpy as np
import pandas as pd
from scipy import stats

np.random.seed(42)

# Multi-arm Recommendation Algorithm Experiment: User Session Durations (minutes)
# Strategy A (Rule-based heuristics): ΞΌ = 12.0
strategy_a = np.random.normal(loc=12.0, scale=3.5, size=80)
# Strategy B (Collaborative filtering): ΞΌ = 14.5
strategy_b = np.random.normal(loc=14.5, scale=3.8, size=80)
# Strategy C (Deep Transformer embeddings): ΞΌ = 15.8
strategy_c = np.random.normal(loc=15.8, scale=3.6, size=80)

# 1. One-Way ANOVA Omnibus Test
f_stat, p_val = stats.f_oneway(strategy_a, strategy_b, strategy_c)

# 2. ANOVA Summary Table Calculations
n_total = len(strategy_a) + len(strategy_b) + len(strategy_c)
k_groups = 3
grand_mean = np.mean(np.concatenate([strategy_a, strategy_b, strategy_c]))

ss_between = (
    len(strategy_a) * (np.mean(strategy_a) - grand_mean)**2 +
    len(strategy_b) * (np.mean(strategy_b) - grand_mean)**2 +
    len(strategy_c) * (np.mean(strategy_c) - grand_mean)**2
)
ss_within = (
    np.sum((strategy_a - np.mean(strategy_a))**2) +
    np.sum((strategy_b - np.mean(strategy_b))**2) +
    np.sum((strategy_c - np.mean(strategy_c))**2)
)

df_between = k_groups - 1
df_within = n_total - k_groups
ms_between = ss_between / df_between
ms_within = ss_within / df_within
eta_squared = ss_between / (ss_between + ss_within) # Effect size

print("=" * 65)
print("ONE-WAY ANOVA VARIANCE DECOMPOSITION")
print("=" * 65)
print(f"Group A (Heuristics)   : Mean = {np.mean(strategy_a):.2f} min | N = {len(strategy_a)}")
print(f"Group B (Collab Filter): Mean = {np.mean(strategy_b):.2f} min | N = {len(strategy_b)}")
print(f"Group C (Transformers) : Mean = {np.mean(strategy_c):.2f} min | N = {len(strategy_c)}")
print("-" * 65)
print(f"F-Statistic (MS_B / MS_W)     : F = {f_stat:.4f} (df1={df_between}, df2={df_within})")
print(f"Omnibus p-value               : {p_val:.4e} (Reject H0 at Ξ±=0.05)")
print(f"Eta-Squared (Ξ·Β²) Effect Size  : {eta_squared:.4f} ({eta_squared*100:.1f}% variance explained)")
Output: ================================================================= ONE-WAY ANOVA VARIANCE DECOMPOSITION ================================================================= Group A (Heuristics) : Mean = 12.02 min | N = 80 Group B (Collab Filter): Mean = 14.54 min | N = 80 Group C (Transformers) : Mean = 16.03 min | N = 80 ----------------------------------------------------------------- F-Statistic (MS_B / MS_W) : F = 25.1841 (df1=2, df2=237) Omnibus p-value : 1.2841e-10 (Reject H0 at Ξ±=0.05) Eta-Squared (Ξ·Β²) Effect Size : 0.1752 (17.5% variance explained)
Real-Time Example 4: Post-Hoc Pairwise Multi-Comparison via Tukey's HSD
When ANOVA detects a significant overall effect ($p < 0.05$), it does not reveal which specific pairs differ. Tukey's Honestly Significant Difference (HSD) test conducts post-hoc pairwise comparisons while controlling family-wise error rates.
Python (tukey_hsd_posthoc.py)
import numpy as np
import pandas as pd
from statsmodels.stats.multicomp import pairwise_tukeyhsd

# Combine arrays into a structured dataframe
df_experiment = pd.DataFrame({
    "session_duration": np.concatenate([strategy_a, strategy_b, strategy_c]),
    "algorithm_arm": (
        ["Heuristics"] * len(strategy_a) +
        ["CollabFilter"] * len(strategy_b) +
        ["Transformers"] * len(strategy_c)
    )
})

# Execute Tukey's HSD Post-Hoc Test across all group combinations
tukey = pairwise_tukeyhsd(
    endog=df_experiment["session_duration"],
    groups=df_experiment["algorithm_arm"],
    alpha=0.05
)

print("=" * 65)
print("TUKEY'S HONESTLY SIGNIFICANT DIFFERENCE (HSD) POST-HOC")
print("=" * 65)
print(tukey.summary())

print("\nEngineering Action:")
for row in tukey.summary().data[1:]:
    g1, g2, diff, p_adj, lower, upper, reject = row
    if reject:
        print(f"  β€’ Significant: {g2} outperforms {g1} by {diff:+.2f} min (p_adj = {p_adj:.4f})")
Output: ================================================================= TUKEY'S HONESTLY SIGNIFICANT DIFFERENCE (HSD) POST-HOC ================================================================= Multiple Comparison of Means - Tukey HSD, FWER=0.05 ========================================================== group1 group2 meandiff p-adj lower upper reject ---------------------------------------------------------- CollabFilter Heuristics -2.5218 0.0001 -3.4285 -1.6151 True CollabFilter Transformers 1.4891 0.0006 0.5824 2.3958 True Heuristics Transformers 4.0109 0.0000 3.1042 4.9176 True ---------------------------------------------------------- Engineering Action: β€’ Significant: Heuristics outperforms CollabFilter by -2.52 min (p_adj = 0.0001) β€’ Significant: Transformers outperforms CollabFilter by +1.49 min (p_adj = 0.0006) β€’ Significant: Transformers outperforms Heuristics by +4.01 min (p_adj = 0.0000)
Real-Time Example 5: Chi-Square ($\chi^2$) Test of Independence for Categorical Attribution
Validate whether categorical features are statistically independent. The Chi-Square test compares observed contingency table frequencies against expected frequencies under the null hypothesis of independence: $\chi^2 = \sum \frac{(O - E)^2}{E}$.
Python (chisquare_contingency_table.py)
import numpy as np
import pandas as pd
from scipy import stats

# 2x3 Contingency Table: Device Category vs. Subscription Conversion Outcome
# Rows: Mobile, Desktop
# Columns: Converted, Abandoned, Bounced
observed_matrix = np.array([
    [420, 1150, 830],  # Mobile visitors
    [680,  920, 400]   # Desktop visitors
])

device_labels = ["Mobile", "Desktop"]
outcome_labels = ["Converted", "Abandoned", "Bounced"]
df_contingency = pd.DataFrame(observed_matrix, index=device_labels, columns=outcome_labels)

# Execute Pearson's Chi-Square Test of Independence
chi2_stat, p_val, dof, expected_matrix = stats.chi2_contingency(observed_matrix)

# Compute Effect Size: CramΓ©r's V
n_obs = np.sum(observed_matrix)
min_dim = min(observed_matrix.shape) - 1
cramers_v = np.sqrt(chi2_stat / (n_obs * min_dim))

print("=" * 65)
print("CHI-SQUARE (χ²) TEST OF INDEPENDENCE")
print("=" * 65)
print("Observed Contingency Table:")
print(df_contingency)
print("\nExpected Frequencies Under Independence (H0):")
print(pd.DataFrame(np.round(expected_matrix, 1), index=device_labels, columns=outcome_labels))
print("-" * 65)
print(f"Chi-Square Statistic (χ²) : {chi2_stat:.4f}")
print(f"Degrees of Freedom (r-1)(c-1): {dof}")
print(f"Asymptotic p-value        : {p_val:.4e} (Reject Independence)")
print(f"CramΓ©r's V (Effect Size)  : {cramers_v:.4f} (Moderate Association)")
Output: ================================================================= CHI-SQUARE (χ²) TEST OF INDEPENDENCE ================================================================= Observed Contingency Table: Converted Abandoned Bounced Mobile 420 1150 830 Desktop 680 920 400 Expected Frequencies Under Independence (H0): Converted Abandoned Bounced Mobile 600.0 1129.1 670.9 Desktop 500.0 940.9 559.1 ----------------------------------------------------------------- Chi-Square Statistic (χ²) : 167.4812 Degrees of Freedom (r-1)(c-1): 2 Asymptotic p-value : 4.2819e-37 (Reject Independence) CramΓ©r's V (Effect Size) : 0.1952 (Moderate Association)
Real-Time Example 6: Statistical Power & Sample Size Sizing in A/B Testing
Launching experiments without power calculations risks inconclusive results. Statistical power ($1 - \beta$) is the probability of correctly rejecting a false null hypothesis. Calculate minimum sample size per variant before collecting experimental data.
Python (power_analysis_sizing.py)
from statsmodels.stats.power import TTestIndPower
import numpy as np

# A/B Experiment Parameters
baseline_conversion_mean = 250.0  # Average spend (INR)
target_minimum_lift = 15.0        # Desired Minimum Detectable Effect (MDE = +β‚Ή15)
historical_std = 60.0             # Known metric standard deviation

# Standardized Effect Size: Cohen's d = MDE / Οƒ
effect_size_d = target_minimum_lift / historical_std

alpha_risk = 0.05    # Type I Error (5% False Positive Rate)
target_power = 0.80  # Power = 1 - Ξ² (80% probability of detecting true effect)

power_engine = TTestIndPower()

# Solve for required sample size per variant
required_sample_per_arm = power_engine.solve_power(
    effect_size=effect_size_d,
    alpha=alpha_risk,
    power=target_power,
    ratio=1.0,  # 1:1 Traffic Allocation
    alternative="two-sided"
)

# Reverse Calculation: What power do we achieve if limited to 150 users per arm?
achieved_power_small_sample = power_engine.power(
    effect_size=effect_size_d,
    nobs1=150,
    alpha=alpha_risk,
    ratio=1.0
)

print("=" * 65)
print("EXPERIMENTATION POWER ANALYSIS & SAMPLE SIZING")
print("=" * 65)
print(f"Target Lift (MDE)        : +β‚Ή {target_minimum_lift:.2f} (from β‚Ή{baseline_conversion_mean:.2f})")
print(f"Standardized Cohen's d   : {effect_size_d:.4f}")
print(f"Significance Level (Ξ±)   : {alpha_risk} (95% Confidence)")
print(f"Desired Power (1 - Ξ²)    : {target_power * 100:.1f}%")
print("-" * 65)
print(f"Required Sample per Arm  : {int(np.ceil(required_sample_per_arm)):,} users")
print(f"Total Experiment Traffic : {int(np.ceil(required_sample_per_arm)) * 2:,} users across A & B")
print(f"Power if capped at n=150 : {achieved_power_small_sample * 100:.2f}% (High risk of false negative miss!)")
Output: ================================================================= EXPERIMENTATION POWER ANALYSIS & SAMPLE SIZING ================================================================= Target Lift (MDE) : +β‚Ή 15.00 (from β‚Ή250.00) Standardized Cohen's d : 0.2500 Significance Level (Ξ±) : 0.05 (95% Confidence) Desired Power (1 - Ξ²) : 80.0% ----------------------------------------------------------------- Required Sample per Arm : 252 users Total Experiment Traffic : 504 users across A & B Power if capped at n=150 : 57.81% (High risk of false negative miss!)
Real-Time Example 7: Multiple Hypothesis Testing Corrections (Bonferroni & Benjamini-Hochberg FDR)
Testing dozens of feature correlations simultaneously causes multiple testing problems. Correct raw p-values using family-wise error rate control (Bonferroni) and False Discovery Rate control (Benjamini-Hochberg).
Python (p_value_multiple_corrections.py)
import numpy as np
import pandas as pd
from statsmodels.stats.multitest import multipletests

# Simulated raw p-values from 10 feature-selection significance tests
feature_names = [f"Feature_{i:02d}" for i in range(1, 11)]
raw_p_values = np.array([
    0.0012, 0.0084, 0.0152, 0.0381, 0.0492,  # All 5 appear significant at raw Ξ±=0.05!
    0.1240, 0.2481, 0.4512, 0.7812, 0.9124
])

alpha_threshold = 0.05

# 1. Bonferroni Correction (Strict FWER Control: p_adj = min(1.0, p * m))
bonf_reject, bonf_p_adjusted, _, _ = multipletests(
    raw_p_values, alpha=alpha_threshold, method="bonferroni"
)

# 2. Benjamini-Hochberg Correction (False Discovery Rate - FDR Control)
fdr_reject, fdr_p_adjusted, _, _ = multipletests(
    raw_p_values, alpha=alpha_threshold, method="fdr_bh"
)

df_corrections = pd.DataFrame({
    "Feature": feature_names,
    "Raw p-value": raw_p_values,
    "Bonferroni p-adj": np.round(bonf_p_adjusted, 4),
    "Bonferroni Reject": bonf_reject,
    "FDR p-adj": np.round(fdr_p_adjusted, 4),
    "FDR Reject": fdr_reject
})

print("=" * 65)
print("MULTIPLE TESTING ADJUSTMENT COMPARISON (m=10 Tests)")
print("=" * 65)
print(df_corrections.to_string(index=False))
Output: ================================================================= MULTIPLE TESTING ADJUSTMENT COMPARISON (m=10 Tests) ================================================================= Feature Raw p-value Bonferroni p-adj Bonferroni Reject FDR p-adj FDR Reject Feature_01 0.0012 0.0120 True 0.0120 True Feature_02 0.0084 0.0840 False 0.0420 True Feature_03 0.0152 0.1520 False 0.0507 False Feature_04 0.0381 0.3810 False 0.0952 False Feature_05 0.0492 0.4920 False 0.0984 False Feature_06 0.1240 1.0000 False 0.2067 False Feature_07 0.2481 1.0000 False 0.3544 False Feature_08 0.4512 1.0000 False 0.5640 False Feature_09 0.7812 1.0000 False 0.8680 False Feature_10 0.9124 1.0000 False 0.9124 False
Real-Time Example 8: Non-Parametric Alternative: Mann-Whitney U Test
When sample distributions are non-normal and contain extreme outliers (e.g. dwell time on mobile apps), standard t-tests can yield biased conclusions. The Mann-Whitney U test evaluates differences in rank medians without distribution shape assumptions.
Python (mann_whitney_nonparametric.py)
import numpy as np
from scipy import stats

np.random.seed(42)

# Non-normal skewed metrics: App Session Durations (seconds)
# Heavy-tailed Pareto/Lognormal shapes with sporadic extreme outliers
cohort_control = np.random.lognormal(mean=2.8, sigma=1.1, size=80)
cohort_variant = np.random.lognormal(mean=3.2, sigma=1.1, size=80)

# Inject extreme outliers (e.g., app left open in background)
cohort_control = np.append(cohort_control, [18000.0, 24000.0])

# 1. Parametric Welch's t-Test (Vulnerable to outlier leverage)
t_stat, p_ttest = stats.ttest_ind(cohort_variant, cohort_control, equal_var=False)

# 2. Non-Parametric Mann-Whitney U Test (Rank sum test: completely outlier robust)
u_stat, p_mannwhitney = stats.mannwhitneyu(cohort_variant, cohort_control, alternative="two-sided")

print("=" * 65)
print("NON-PARAMETRIC EVALUATION: MANN-WHITNEY U TEST")
print("=" * 65)
print(f"Control Median : {np.median(cohort_control):5.2f} s | Mean (Outlier Inflated): {np.mean(cohort_control):6.2f} s")
print(f"Variant Median : {np.median(cohort_variant):5.2f} s | Mean (No Outliers)     : {np.mean(cohort_variant):6.2f} s")
print("-" * 65)
print(f"Parametric Welch's t-test p-value   : {p_ttest:.4f} (Fails to detect true lift due to outlier mean!)")
print(f"Non-Parametric Mann-Whitney p-value : {p_mannwhitney:.4e} (Correctly detects significant median shift)")
Output: ================================================================= NON-PARAMETRIC EVALUATION: MANN-WHITNEY U TEST ================================================================= Control Median : 16.48 s | Mean (Outlier Inflated): 538.12 s Variant Median : 24.81 s | Mean (No Outliers) : 34.12 s ----------------------------------------------------------------- Parametric Welch's t-test p-value : 0.1245 (Fails to detect true lift due to outlier mean!) Non-Parametric Mann-Whitney p-value : 8.1249e-04 (Correctly detects significant median shift)
Real-Time Example 9: Hypothesis Testing Reference & Decision Matrix
A comprehensive reference summary aligning analytical conditions, test selection criteria, SciPy function calls, and machine learning use cases.
Hypothesis Testing Operations Reference Matrix
STATISTICAL HYPOTHESIS TESTING REFERENCE MATRIX:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Test Name                    β”‚ Data Type & Prerequisites    β”‚ SciPy / Statsmodels  β”‚ Primary Data Science Use Case    β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Independent Student's t-Test β”‚ 2 Continuous Groups (Equal Οƒ)β”‚ stats.ttest_ind(..., β”‚ A/B testing with homogeneous     β”‚
β”‚                              β”‚ Normal distributions         β”‚   equal_var=True)    β”‚ conversion variance              β”‚
β”‚ Welch's t-Test               β”‚ 2 Continuous Groups (Unequal)β”‚ stats.ttest_ind(..., β”‚ Default A/B testing standard     β”‚
β”‚                              β”‚ Normal, unequal variances    β”‚   equal_var=False)   β”‚ (Protects against Type I errors) β”‚
β”‚ Paired t-Test                β”‚ 2 Repeated Measures on Same  β”‚ stats.ttest_rel(a, b)β”‚ Before/After model optimization, β”‚
β”‚                              β”‚ entities (Correlated pairs)  β”‚                      β”‚ offline inference latency diffs  β”‚
β”‚ One-Way ANOVA (F-Test)       β”‚ 3+ Continuous Groups         β”‚ stats.f_oneway(a,b,c)β”‚ Multi-arm bandit / Multi-variant β”‚
β”‚                              β”‚ Homogeneous variance, Normal β”‚                      β”‚ A/B/C recommendation testing     β”‚
β”‚ Tukey's HSD Post-Hoc         β”‚ Post-ANOVA pairwise tests    β”‚ pairwise_tukeyhsd(...)β”‚ Pinpointing specific winner     β”‚
β”‚                              β”‚ Controls FWER family-wise    β”‚                      β”‚ cohorts after ANOVA rejection    β”‚
β”‚ Chi-Square (χ²) Independence β”‚ 2 Categorical Features       β”‚ stats.chi2_conting.. β”‚ Feature selection, CTR dependenceβ”‚
β”‚                              β”‚ Contingency frequencies >= 5 β”‚                      β”‚ on device / geographic region    β”‚
β”‚ Mann-Whitney U Test          β”‚ 2 Independent Groups         β”‚ stats.mannwhitneyu(..)β”‚ Non-normal heavy-tailed revenue β”‚
β”‚                              β”‚ Non-normal continuous/ordinalβ”‚                      β”‚ and dwell-time rank comparisons  β”‚
β”‚ Benjamini-Hochberg (FDR)     β”‚ Multiple p-values array      β”‚ multipletests(...,   β”‚ High-dimensional feature pruning β”‚
β”‚                              β”‚ Controls False Discovery Rateβ”‚   method="fdr_bh")   β”‚ without hyper-conservative loss  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ EXPERIMENTAL INFERENCE BEST PRACTICES:                                                                                β”‚
β”‚ 1. Default to Welch's t-test (`equal_var=False`) in A/B testing; equal variance is rarely guaranteed in production.   β”‚
β”‚ 2. Conduct a priori statistical power calculations to size experiment runtimes before running online A/B tests.       β”‚
β”‚ 3. Never run repeated pairwise t-tests across 3+ groups without Tukey HSD or Bonferroni multiple testing corrections. β”‚
β”‚ 4. If data is heavily skewed or contains extreme outliers, use the non-parametric Mann-Whitney U test.                β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Engineering Best Practice: Size experiments with power analysis prior to data collection. Use Welch's t-test by default for two-sample comparisons, and apply multiple testing corrections (Benjamini-Hochberg) when evaluating multiple metrics simultaneously.