π§ͺ Statistical Significance: Distinguishing True Signal from Random Flukes
In production machine learning and business operations, decisions cannot be made on raw metric deltas alone. A new conversion model achieving a 0.8% lift or an algorithmic trading strategy reporting an increased Sharpe ratio might simply be the artifact of random sample variance. Hypothesis Testing provides the mathematical machinery to determine whether observed differences are statistically genuine or attributable to random sampling noise. Through Null Hypothesis Significance Testing (NHST), data scientists quantify risk via p-values, balance Type I false positives ($\alpha$) against Type II false negatives ($\beta$), size experiments through statistical power ($1 - \beta$), compare continuous targets across cohorts using Independent, Welch's, and Paired t-tests, analyze multi-group variance with One-Way ANOVA and Tukey's HSD, validate categorical dependencies using Chi-Square contingency matrices, and correct for false discovery inflation using Bonferroni and Benjamini-Hochberg (FDR) algorithms.
HYPOTHESIS TESTING DECISION PIPELINE & ERROR MATRIX:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 1. FORMULATE HYPOTHESES & SET SIGNIFICANCE LEVEL (Ξ± = 0.05) β
β H0 (Null Hypothesis): ΞΌ_treatment == ΞΌ_control (No Effect) β
β H1 (Alternative Hypothesis): ΞΌ_treatment != ΞΌ_control (Significant Shift)β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 2. SELECT TEST STATISTIC BASED ON DATA TOPOLOGY β
β βββ 2 Continuous Independent Samples (Equal Var) βββΊ Student's t-Test β
β βββ 2 Continuous Independent Samples (Unequal Var) βββΊ Welch's t-Test β
β βββ 2 Repeated / Paired Measurements (Before/After) βββΊ Paired t-Test β
β βββ 3+ Continuous Group Means βββΊ One-Way ANOVA β
β βββ Categorical Frequencies & Proportions βββΊ Chi-Square (ΟΒ²) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 3. P-VALUE EVALUATION & DECISION MATRIX β
β p-value <= Ξ± βββΊ Reject H0 (Statistically Significant Effect) β
β p-value > Ξ± βββΊ Fail to Reject H0 (Insufficient Evidence) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β DECISION ERROR MATRIX: β
β β H0 is Actually TRUE β H0 is Actually FALSE β
β βββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββ
β Reject H0 β Type I Error (False Alarm: Ξ±)β Correct (Power: 1 - Ξ²) β
β Fail to Reject H0 β Correct (Confidence: 1 - Ξ±) β Type II Error (Miss: Ξ²)β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
import numpy as np
from scipy import stats
np.random.seed(42)
# E-commerce A/B Test: Revenue per checkout (INR)
# Control Group (A): n=120, ΞΌ=1450, Ο=320
control_rev = np.random.normal(loc=1450.0, scale=320.0, size=120)
# Treatment Group (B): n=100, ΞΌ=1560, Ο=540 (Higher mean and significantly higher variance!)
treatment_rev = np.random.normal(loc=1560.0, scale=540.0, size=100)
# 1. Test for Homogeneity of Variances (Levene's Test)
levene_stat, levene_p = stats.levene(control_rev, treatment_rev)
# 2. Standard Student's t-Test (Assumes equal_var=True)
t_student, p_student = stats.ttest_ind(treatment_rev, control_rev, equal_var=True)
# 3. Welch's t-Test (Robust against unequal variance: equal_var=False)
t_welch, p_welch = stats.ttest_ind(treatment_rev, control_rev, equal_var=False)
# 4. Effect Size: Cohen's d
pooled_std = np.sqrt(
((len(control_rev) - 1) * np.var(control_rev, ddof=1) + (len(treatment_rev) - 1) * np.var(treatment_rev, ddof=1))
/ (len(control_rev) + len(treatment_rev) - 2)
)
cohens_d = (np.mean(treatment_rev) - np.mean(control_rev)) / pooled_std
print("=" * 65)
print("INDEPENDENT A/B TEST: STUDENT'S VS. WELCH'S T-TEST")
print("=" * 65)
print(f"Control (A) : Mean = βΉ {np.mean(control_rev):6.2f} | Std = βΉ {np.std(control_rev, ddof=1):6.2f} | N = {len(control_rev)}")
print(f"Treatment (B) : Mean = βΉ {np.mean(treatment_rev):6.2f} | Std = βΉ {np.std(treatment_rev, ddof=1):6.2f} | N = {len(treatment_rev)}")
print("-" * 65)
print(f"Levene's Variance Test p-value : {levene_p:.4e} {'[UNEQUAL VARIANCES]' if levene_p < 0.05 else '[EQUAL VARIANCES]'}")
print(f"Student's t-Test (equal_var=T) : t = {t_student:+.4f} | p-value = {p_student:.4e}")
print(f"Welch's t-Test (equal_var=F) : t = {t_welch:+.4f} | p-value = {p_welch:.4e} (Robust)")
print(f"Standardized Effect (Cohen's d): {cohens_d:.4f} (Medium Effect Size)")
import numpy as np
from scipy import stats
np.random.seed(42)
# Benchmark 50 benchmark payloads through an NLP Transformer pipeline
# Baseline unquantized model latency (ms) per payload
baseline_latency = np.random.normal(loc=125.0, scale=20.0, size=50)
# Quantized ONNX FP16 runtime on the EXACT same 50 payloads (Reduction with batch correlation)
noise = np.random.normal(loc=0.0, scale=3.0, size=50)
quantized_latency = baseline_latency * 0.82 + noise
# 1. Compute Pairwise Latency Differences: D_i = Baseline_i - Quantized_i
differences = baseline_latency - quantized_latency
# 2. Paired Student's t-Test
t_stat, p_val = stats.ttest_rel(baseline_latency, quantized_latency)
# 3. 95% Confidence Interval for the True Average Speedup
df = len(differences) - 1
se_diff = stats.sem(differences)
t_crit = stats.t.ppf(0.975, df=df)
ci_lower = np.mean(differences) - (t_crit * se_diff)
ci_upper = np.mean(differences) + (t_crit * se_diff)
print("=" * 65)
print("PAIRED T-TEST: MODEL QUANTIZATION LATENCY AUDIT")
print("=" * 65)
print(f"Payload Batches Evaluated : {len(baseline_latency)}")
print(f"Mean Baseline Latency (FP32): {np.mean(baseline_latency):6.2f} ms")
print(f"Mean Quantized Latency (FP16): {np.mean(quantized_latency):6.2f} ms")
print(f"Mean Pairwise Latency Saving : {np.mean(differences):6.2f} ms (p < 0.001)")
print("-" * 65)
print(f"Paired t-Statistic : t = {t_stat:+.4f}")
print(f"Two-Tailed p-value : {p_val:.4e} (Highly Significant)")
print(f"95% CI of Latency Reduction : [{ci_lower:.2f}, {ci_upper:.2f}] ms")
import numpy as np
import pandas as pd
from scipy import stats
np.random.seed(42)
# Multi-arm Recommendation Algorithm Experiment: User Session Durations (minutes)
# Strategy A (Rule-based heuristics): ΞΌ = 12.0
strategy_a = np.random.normal(loc=12.0, scale=3.5, size=80)
# Strategy B (Collaborative filtering): ΞΌ = 14.5
strategy_b = np.random.normal(loc=14.5, scale=3.8, size=80)
# Strategy C (Deep Transformer embeddings): ΞΌ = 15.8
strategy_c = np.random.normal(loc=15.8, scale=3.6, size=80)
# 1. One-Way ANOVA Omnibus Test
f_stat, p_val = stats.f_oneway(strategy_a, strategy_b, strategy_c)
# 2. ANOVA Summary Table Calculations
n_total = len(strategy_a) + len(strategy_b) + len(strategy_c)
k_groups = 3
grand_mean = np.mean(np.concatenate([strategy_a, strategy_b, strategy_c]))
ss_between = (
len(strategy_a) * (np.mean(strategy_a) - grand_mean)**2 +
len(strategy_b) * (np.mean(strategy_b) - grand_mean)**2 +
len(strategy_c) * (np.mean(strategy_c) - grand_mean)**2
)
ss_within = (
np.sum((strategy_a - np.mean(strategy_a))**2) +
np.sum((strategy_b - np.mean(strategy_b))**2) +
np.sum((strategy_c - np.mean(strategy_c))**2)
)
df_between = k_groups - 1
df_within = n_total - k_groups
ms_between = ss_between / df_between
ms_within = ss_within / df_within
eta_squared = ss_between / (ss_between + ss_within) # Effect size
print("=" * 65)
print("ONE-WAY ANOVA VARIANCE DECOMPOSITION")
print("=" * 65)
print(f"Group A (Heuristics) : Mean = {np.mean(strategy_a):.2f} min | N = {len(strategy_a)}")
print(f"Group B (Collab Filter): Mean = {np.mean(strategy_b):.2f} min | N = {len(strategy_b)}")
print(f"Group C (Transformers) : Mean = {np.mean(strategy_c):.2f} min | N = {len(strategy_c)}")
print("-" * 65)
print(f"F-Statistic (MS_B / MS_W) : F = {f_stat:.4f} (df1={df_between}, df2={df_within})")
print(f"Omnibus p-value : {p_val:.4e} (Reject H0 at Ξ±=0.05)")
print(f"Eta-Squared (Ξ·Β²) Effect Size : {eta_squared:.4f} ({eta_squared*100:.1f}% variance explained)")
import numpy as np
import pandas as pd
from statsmodels.stats.multicomp import pairwise_tukeyhsd
# Combine arrays into a structured dataframe
df_experiment = pd.DataFrame({
"session_duration": np.concatenate([strategy_a, strategy_b, strategy_c]),
"algorithm_arm": (
["Heuristics"] * len(strategy_a) +
["CollabFilter"] * len(strategy_b) +
["Transformers"] * len(strategy_c)
)
})
# Execute Tukey's HSD Post-Hoc Test across all group combinations
tukey = pairwise_tukeyhsd(
endog=df_experiment["session_duration"],
groups=df_experiment["algorithm_arm"],
alpha=0.05
)
print("=" * 65)
print("TUKEY'S HONESTLY SIGNIFICANT DIFFERENCE (HSD) POST-HOC")
print("=" * 65)
print(tukey.summary())
print("\nEngineering Action:")
for row in tukey.summary().data[1:]:
g1, g2, diff, p_adj, lower, upper, reject = row
if reject:
print(f" β’ Significant: {g2} outperforms {g1} by {diff:+.2f} min (p_adj = {p_adj:.4f})")
import numpy as np
import pandas as pd
from scipy import stats
# 2x3 Contingency Table: Device Category vs. Subscription Conversion Outcome
# Rows: Mobile, Desktop
# Columns: Converted, Abandoned, Bounced
observed_matrix = np.array([
[420, 1150, 830], # Mobile visitors
[680, 920, 400] # Desktop visitors
])
device_labels = ["Mobile", "Desktop"]
outcome_labels = ["Converted", "Abandoned", "Bounced"]
df_contingency = pd.DataFrame(observed_matrix, index=device_labels, columns=outcome_labels)
# Execute Pearson's Chi-Square Test of Independence
chi2_stat, p_val, dof, expected_matrix = stats.chi2_contingency(observed_matrix)
# Compute Effect Size: CramΓ©r's V
n_obs = np.sum(observed_matrix)
min_dim = min(observed_matrix.shape) - 1
cramers_v = np.sqrt(chi2_stat / (n_obs * min_dim))
print("=" * 65)
print("CHI-SQUARE (ΟΒ²) TEST OF INDEPENDENCE")
print("=" * 65)
print("Observed Contingency Table:")
print(df_contingency)
print("\nExpected Frequencies Under Independence (H0):")
print(pd.DataFrame(np.round(expected_matrix, 1), index=device_labels, columns=outcome_labels))
print("-" * 65)
print(f"Chi-Square Statistic (ΟΒ²) : {chi2_stat:.4f}")
print(f"Degrees of Freedom (r-1)(c-1): {dof}")
print(f"Asymptotic p-value : {p_val:.4e} (Reject Independence)")
print(f"CramΓ©r's V (Effect Size) : {cramers_v:.4f} (Moderate Association)")
from statsmodels.stats.power import TTestIndPower
import numpy as np
# A/B Experiment Parameters
baseline_conversion_mean = 250.0 # Average spend (INR)
target_minimum_lift = 15.0 # Desired Minimum Detectable Effect (MDE = +βΉ15)
historical_std = 60.0 # Known metric standard deviation
# Standardized Effect Size: Cohen's d = MDE / Ο
effect_size_d = target_minimum_lift / historical_std
alpha_risk = 0.05 # Type I Error (5% False Positive Rate)
target_power = 0.80 # Power = 1 - Ξ² (80% probability of detecting true effect)
power_engine = TTestIndPower()
# Solve for required sample size per variant
required_sample_per_arm = power_engine.solve_power(
effect_size=effect_size_d,
alpha=alpha_risk,
power=target_power,
ratio=1.0, # 1:1 Traffic Allocation
alternative="two-sided"
)
# Reverse Calculation: What power do we achieve if limited to 150 users per arm?
achieved_power_small_sample = power_engine.power(
effect_size=effect_size_d,
nobs1=150,
alpha=alpha_risk,
ratio=1.0
)
print("=" * 65)
print("EXPERIMENTATION POWER ANALYSIS & SAMPLE SIZING")
print("=" * 65)
print(f"Target Lift (MDE) : +βΉ {target_minimum_lift:.2f} (from βΉ{baseline_conversion_mean:.2f})")
print(f"Standardized Cohen's d : {effect_size_d:.4f}")
print(f"Significance Level (Ξ±) : {alpha_risk} (95% Confidence)")
print(f"Desired Power (1 - Ξ²) : {target_power * 100:.1f}%")
print("-" * 65)
print(f"Required Sample per Arm : {int(np.ceil(required_sample_per_arm)):,} users")
print(f"Total Experiment Traffic : {int(np.ceil(required_sample_per_arm)) * 2:,} users across A & B")
print(f"Power if capped at n=150 : {achieved_power_small_sample * 100:.2f}% (High risk of false negative miss!)")
import numpy as np
import pandas as pd
from statsmodels.stats.multitest import multipletests
# Simulated raw p-values from 10 feature-selection significance tests
feature_names = [f"Feature_{i:02d}" for i in range(1, 11)]
raw_p_values = np.array([
0.0012, 0.0084, 0.0152, 0.0381, 0.0492, # All 5 appear significant at raw Ξ±=0.05!
0.1240, 0.2481, 0.4512, 0.7812, 0.9124
])
alpha_threshold = 0.05
# 1. Bonferroni Correction (Strict FWER Control: p_adj = min(1.0, p * m))
bonf_reject, bonf_p_adjusted, _, _ = multipletests(
raw_p_values, alpha=alpha_threshold, method="bonferroni"
)
# 2. Benjamini-Hochberg Correction (False Discovery Rate - FDR Control)
fdr_reject, fdr_p_adjusted, _, _ = multipletests(
raw_p_values, alpha=alpha_threshold, method="fdr_bh"
)
df_corrections = pd.DataFrame({
"Feature": feature_names,
"Raw p-value": raw_p_values,
"Bonferroni p-adj": np.round(bonf_p_adjusted, 4),
"Bonferroni Reject": bonf_reject,
"FDR p-adj": np.round(fdr_p_adjusted, 4),
"FDR Reject": fdr_reject
})
print("=" * 65)
print("MULTIPLE TESTING ADJUSTMENT COMPARISON (m=10 Tests)")
print("=" * 65)
print(df_corrections.to_string(index=False))
import numpy as np
from scipy import stats
np.random.seed(42)
# Non-normal skewed metrics: App Session Durations (seconds)
# Heavy-tailed Pareto/Lognormal shapes with sporadic extreme outliers
cohort_control = np.random.lognormal(mean=2.8, sigma=1.1, size=80)
cohort_variant = np.random.lognormal(mean=3.2, sigma=1.1, size=80)
# Inject extreme outliers (e.g., app left open in background)
cohort_control = np.append(cohort_control, [18000.0, 24000.0])
# 1. Parametric Welch's t-Test (Vulnerable to outlier leverage)
t_stat, p_ttest = stats.ttest_ind(cohort_variant, cohort_control, equal_var=False)
# 2. Non-Parametric Mann-Whitney U Test (Rank sum test: completely outlier robust)
u_stat, p_mannwhitney = stats.mannwhitneyu(cohort_variant, cohort_control, alternative="two-sided")
print("=" * 65)
print("NON-PARAMETRIC EVALUATION: MANN-WHITNEY U TEST")
print("=" * 65)
print(f"Control Median : {np.median(cohort_control):5.2f} s | Mean (Outlier Inflated): {np.mean(cohort_control):6.2f} s")
print(f"Variant Median : {np.median(cohort_variant):5.2f} s | Mean (No Outliers) : {np.mean(cohort_variant):6.2f} s")
print("-" * 65)
print(f"Parametric Welch's t-test p-value : {p_ttest:.4f} (Fails to detect true lift due to outlier mean!)")
print(f"Non-Parametric Mann-Whitney p-value : {p_mannwhitney:.4e} (Correctly detects significant median shift)")
STATISTICAL HYPOTHESIS TESTING REFERENCE MATRIX:
ββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββ
β Test Name β Data Type & Prerequisites β SciPy / Statsmodels β Primary Data Science Use Case β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β Independent Student's t-Test β 2 Continuous Groups (Equal Ο)β stats.ttest_ind(..., β A/B testing with homogeneous β
β β Normal distributions β equal_var=True) β conversion variance β
β Welch's t-Test β 2 Continuous Groups (Unequal)β stats.ttest_ind(..., β Default A/B testing standard β
β β Normal, unequal variances β equal_var=False) β (Protects against Type I errors) β
β Paired t-Test β 2 Repeated Measures on Same β stats.ttest_rel(a, b)β Before/After model optimization, β
β β entities (Correlated pairs) β β offline inference latency diffs β
β One-Way ANOVA (F-Test) β 3+ Continuous Groups β stats.f_oneway(a,b,c)β Multi-arm bandit / Multi-variant β
β β Homogeneous variance, Normal β β A/B/C recommendation testing β
β Tukey's HSD Post-Hoc β Post-ANOVA pairwise tests β pairwise_tukeyhsd(...)β Pinpointing specific winner β
β β Controls FWER family-wise β β cohorts after ANOVA rejection β
β Chi-Square (ΟΒ²) Independence β 2 Categorical Features β stats.chi2_conting.. β Feature selection, CTR dependenceβ
β β Contingency frequencies >= 5 β β on device / geographic region β
β Mann-Whitney U Test β 2 Independent Groups β stats.mannwhitneyu(..)β Non-normal heavy-tailed revenue β
β β Non-normal continuous/ordinalβ β and dwell-time rank comparisons β
β Benjamini-Hochberg (FDR) β Multiple p-values array β multipletests(..., β High-dimensional feature pruning β
β β Controls False Discovery Rateβ method="fdr_bh") β without hyper-conservative loss β
ββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββ€
β EXPERIMENTAL INFERENCE BEST PRACTICES: β
β 1. Default to Welch's t-test (`equal_var=False`) in A/B testing; equal variance is rarely guaranteed in production. β
β 2. Conduct a priori statistical power calculations to size experiment runtimes before running online A/B tests. β
β 3. Never run repeated pairwise t-tests across 3+ groups without Tukey HSD or Bonferroni multiple testing corrections. β
β 4. If data is heavily skewed or contains extreme outliers, use the non-parametric Mann-Whitney U test. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ