π Understanding Distributions: Summarizing Reality & Inferring Populations
Data science operates at the boundary between observed empirical data and unknown population realities. Descriptive Statistics provides mathematical summaries of observed distributionsβcharacterizing location (mean, median, trimmed means), dispersion (variance, standard deviation, interquartile range), and distributional geometry (skewness and excess kurtosis). Inferential Statistics bridges the gap from observed samples to universal parameters using probability theory. Anchored by the Central Limit Theorem (CLT) and Bessel's sample correction ($N-1$), inferential methods quantify uncertainty via standard errors, construct parametric and non-parametric bootstrap confidence intervals, and validate whether patterns reflect true statistical signal or random sampling artifacts.
STATISTICAL INFERENCE ARCHITECTURAL TOPOGRAPHY:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 1. POPULATION (Unknown Target Distribution: Parameters ΞΌ, Ο, N -> β) β
β All retail credit card transactions, all sensor streams, all user visits β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ (Random Sampling with Replacement/Stratification)
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 2. OBSERVED SAMPLE DATA (Sample Statistics: xΜ, s, n) β
β Empirical finite dataset ingested into Pandas DataFrame / NumPy Matrix β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βββββββββββββββββββββββ΄ββββββββββββββββββββββ
βΌ βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β DESCRIPTIVE STATISTICS (Exploratory Data)ββ INFERENTIAL STATISTICS (Decision Engine) β
β * Central Tendency: Mean, Median, Mode ββ * Standard Error: SE = s / sqrt(n) β
β * Dispersion: Variance (ddof=1), IQR, MADββ * Central Limit Theorem (CLT) Normality β
β * Shape Moments: Fisher Skew & Kurtosis ββ * Confidence Intervals: [xΜ Β± z*(s/βn)] β
β * Outlier Detection: Tukey's 1.5*IQR Ruleββ * Bootstrap Sampling: Empiric CI bounds β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β β
βββββββββββββββββββββββ¬ββββββββββββββββββββββ
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β MACHINE LEARNING APPLICATION: Feature Engineering, Leakage Guards, A/B Testsβ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
import numpy as np
from scipy import stats
# Baseline engineering salaries (in INR Lakhs per annum)
np.random.seed(42)
baseline_salaries = np.random.normal(loc=18.0, scale=3.5, size=50)
# Introduce 3 extreme high-net-worth outliers (e.g., executive equity buyouts)
outlier_salaries = np.append(baseline_salaries, [350.0, 480.0, 720.0])
def summarize_central_tendency(data: np.ndarray, label: str) -> None:
mean_val = np.mean(data)
median_val = np.median(data)
# Trim top and bottom 10% extreme values
trimmed_mean = stats.trim_mean(data, proportiontocut=0.10)
print(f"[{label}] (N={len(data)})")
print(f" Arithmetic Mean : βΉ {mean_val:6.2f} Lakhs (Breakdown Point = 0%)")
print(f" 50th Median : βΉ {median_val:6.2f} Lakhs (Breakdown Point = 50%)")
print(f" 10% Trimmed Mean: βΉ {trimmed_mean:6.2f} Lakhs (Robust compromise)")
print("=" * 65)
print("ROBUSTNESS & BREAKDOWN POINT AUDIT")
print("=" * 65)
summarize_central_tendency(baseline_salaries, "Clean Baseline Sample")
print("-" * 65)
summarize_central_tendency(outlier_salaries, "Contaminated Sample (+3 Outliers)")
ddof=1 applies Bessel's correction to restore unbiasedness.
import numpy as np
# True population parameters
true_pop = np.random.normal(loc=100.0, scale=15.0, size=100000)
true_variance = np.var(true_pop, ddof=0)
# Repeatedly take small sample batches (n=10) and evaluate variance bias
num_simulations = 5000
sample_size = 10
biased_variances = []
unbiased_variances = []
for _ in range(num_simulations):
sample = np.random.choice(true_pop, size=sample_size, replace=False)
# ddof=0 -> Divides by N (Biased MLE)
biased_variances.append(np.var(sample, ddof=0))
# ddof=1 -> Divides by N - 1 (Unbiased Bessel's correction)
unbiased_variances.append(np.var(sample, ddof=1))
mean_biased = np.mean(biased_variances)
mean_unbiased = np.mean(unbiased_variances)
print("=" * 65)
print("BESSEL'S CORRECTION EMPIRICAL CONVERGENCE")
print("=" * 65)
print(f"True Population Variance (ΟΒ²) : {true_variance:.4f}")
print(f"Average Biased Variance (N) : {mean_biased:.4f} (Underestimates by {(1 - mean_biased/true_variance)*100:.1f}%)")
print(f"Average Unbiased Variance (N-1) : {mean_unbiased:.4f} (Expected: {(sample_size/(sample_size-1))*mean_biased:.4f})")
print(f"Relative Unbiased Error : {abs(mean_unbiased - true_variance)/true_variance * 100:.2f}%")
import numpy as np
from scipy import stats
np.random.seed(42)
# Generate 3 distinct feature distributions
gaussian_dist = np.random.normal(loc=0.0, scale=1.0, size=5000)
skewed_lognormal = np.random.lognormal(mean=1.5, sigma=0.75, size=5000)
heavy_tailed_t = np.random.standard_t(df=3, size=5000) # Student-t with heavy tails
def audit_moments(data: np.ndarray, name: str) -> None:
skew_val = stats.skew(data)
# fisher=True computes Excess Kurtosis (Normal Distribution = 0.0)
kurt_val = stats.kurtosis(data, fisher=True)
if abs(skew_val) < 0.5:
skew_type = "Symmetric (Bell-shaped)"
elif skew_val >= 0.5:
skew_type = "Right-skewed (Positive Tail)"
else:
skew_type = "Left-skewed (Negative Tail)"
if kurt_val > 1.0:
tail_type = "Leptokurtic (Heavy Tails / Outlier-Prone)"
elif kurt_val < -1.0:
tail_type = "Platykurtic (Light Tails / Flat)"
else:
tail_type = "Mesokurtic (Normal-like Tails)"
print(f"Feature: {name:22s}")
print(f" Fisher Skewness : {skew_val:+6.3f} -> {skew_type}")
print(f" Excess Kurtosis : {kurt_val:+6.3f} -> {tail_type}\n")
print("=" * 65)
print("HIGHER DISTRIBUTION MOMENTS AUDIT")
print("=" * 65)
audit_moments(gaussian_dist, "Standard Gaussian")
audit_moments(skewed_lognormal, "Log-Normal Income")
audit_moments(heavy_tailed_t, "Student-t Returns")
import numpy as np
from scipy import stats
np.random.seed(42)
# Underlying non-normal population: Exponential Distribution (e.g., website session durations)
scale_lambda = 10.0 # True Population Mean = 10.0, True Population Std = 10.0
population = np.random.exponential(scale=scale_lambda, size=200000)
pop_mean = np.mean(population)
pop_std = np.std(population, ddof=0)
num_experiments = 4000
sample_sizes = [2, 5, 30, 100]
print("=" * 65)
print("CENTRAL LIMIT THEOREM (CLT) SAMPLING DISTRIBUTION")
print("=" * 65)
print(f"Parent Population Shape: Heavily Right-Skewed Exponential (Ξ»={scale_lambda})")
print(f"True Population Mean (ΞΌ): {pop_mean:.4f} | True Population Std (Ο): {pop_std:.4f}\n")
for n in sample_sizes:
# Generate distribution of sample means
sample_means = np.array([
np.mean(np.random.choice(population, size=n, replace=False))
for _ in range(num_experiments)
])
mean_of_means = np.mean(sample_means)
std_of_means = np.std(sample_means, ddof=1)
theoretical_se = pop_std / np.sqrt(n)
clt_skewness = stats.skew(sample_means)
# Normality test (Shapiro-Wilk on a subsample)
_, p_value = stats.shapiro(sample_means[:500])
print(f"Sample Size (n = {n:3d}):")
print(f" Mean of Sample Means (xΜ) : {mean_of_means:.4f} (Approaches ΞΌ = {pop_mean:.2f})")
print(f" Observed Standard Error : {std_of_means:.4f}")
print(f" Theoretical SE (Ο / βn) : {theoretical_se:.4f}")
print(f" Skewness of Sampling Dist: {clt_skewness:+.4f} (0.0 = Normal)")
print(f" Normality p-value (n>=30): {p_value:.4e} {'[NORMAL]' if p_value > 0.05 else '[NON-NORMAL]'}\n")
import numpy as np
from scipy import stats
# E-commerce checkout conversion latency (seconds)
np.random.seed(42)
sample_latencies = np.random.normal(loc=1.85, scale=0.45, size=25)
n = len(sample_latencies)
sample_mean = np.mean(sample_latencies)
sample_std = np.std(sample_latencies, ddof=1)
confidence_level = 0.95
alpha = 1.0 - confidence_level
# 1. Standard Error of the Mean (SE)
se = sample_std / np.sqrt(n)
# 2. Student's t-distribution (Unknown population Ο, small n)
df = n - 1
t_critical = stats.t.ppf(1.0 - alpha / 2.0, df=df)
margin_of_error_t = t_critical * se
ci_t_lower = sample_mean - margin_of_error_t
ci_t_upper = sample_mean + margin_of_error_t
# 3. Z-distribution (Theoretical Large-Sample Asymptotic bound)
z_critical = stats.norm.ppf(1.0 - alpha / 2.0)
margin_of_error_z = z_critical * se
ci_z_lower = sample_mean - margin_of_error_z
ci_z_upper = sample_mean + margin_of_error_z
print("=" * 65)
print("95% PARAMETRIC CONFIDENCE INTERVAL BOUNDS")
print("=" * 65)
print(f"Sample Size (n) : {n} observations")
print(f"Sample Mean (xΜ) : {sample_mean:.4f} seconds")
print(f"Sample Std Dev (s) : {sample_std:.4f} seconds")
print(f"Standard Error (SE) : {se:.4f}")
print("-" * 65)
print(f"Student's t Critical (df={df}) : {t_critical:.4f}")
print(f"t-Confidence Interval (95%) : [{ci_t_lower:.4f}, {ci_t_upper:.4f}] s (Margin: Β±{margin_of_error_t:.4f})")
print(f"Z-Confidence Interval (95%) : [{ci_z_lower:.4f}, {ci_z_upper:.4f}] s (Margin: Β±{margin_of_error_z:.4f})")
print(f"Width Expansion from t vs Z : +{((margin_of_error_t / margin_of_error_z) - 1)*100:.2f}% (Accounts for parameter uncertainty)")
import numpy as np
# Non-normal skewed dataset: API response durations with sporadic long-tail timeouts
np.random.seed(42)
api_durations = np.concatenate([
np.random.normal(loc=120, scale=15, size=180),
np.random.exponential(scale=350, size=20)
])
def bootstrap_percentile_ci(data: np.ndarray, metric_fn, num_bootstraps: int = 5000,
ci: float = 0.95) -> tuple:
"""
Computes non-parametric empirical bootstrap confidence interval.
"""
n = len(data)
bootstrap_estimates = np.empty(num_bootstraps)
for i in range(num_bootstraps):
# Sample with replacement (same sample size n)
boot_sample = np.random.choice(data, size=n, replace=True)
bootstrap_estimates[i] = metric_fn(boot_sample)
alpha = (1.0 - ci) / 2.0
lower_bound = np.percentile(bootstrap_estimates, 100 * alpha)
upper_bound = np.percentile(bootstrap_estimates, 100 * (1.0 - alpha))
bootstrap_se = np.std(bootstrap_estimates, ddof=1)
return lower_bound, upper_bound, bootstrap_se, bootstrap_estimates
# Estimate 90th percentile (P90) latency SLA
p90_metric = lambda d: np.percentile(d, 90)
point_estimate_p90 = p90_metric(api_durations)
b_lower, b_upper, b_se, b_dist = bootstrap_percentile_ci(
api_durations, metric_fn=p90_metric, num_bootstraps=5000, ci=0.95
)
print("=" * 65)
print("NON-PARAMETRIC BOOTSTRAP SLA ESTIMATION")
print("=" * 65)
print(f"Sample Observations (N) : {len(api_durations)}")
print(f"Point Estimate (P90 Latency) : {point_estimate_p90:.2f} ms")
print(f"Bootstrap Standard Error : {b_se:.2f} ms")
print(f"95% Bootstrap Empirical CI : [{b_lower:.2f}, {b_upper:.2f}] ms")
print(f"SLA Compliance Verdict : Upper bound {b_upper:.2f} ms must satisfy SLA targets.")
import numpy as np
from scipy import stats
# Feature with extreme recording errors
np.random.seed(42)
raw_readings = np.concatenate([
np.random.normal(loc=50.0, scale=5.0, size=95),
np.array([120.0, 145.0, -40.0, 310.0, 280.0]) # Severe outliers
])
# 1. Method A: Tukey's IQR Rule
q25, q75 = np.percentile(raw_readings, [25, 75])
iqr = q75 - q25
iqr_lower = q25 - 1.5 * iqr
iqr_upper = q75 + 1.5 * iqr
iqr_outliers = raw_readings[(raw_readings < iqr_lower) | (raw_readings > iqr_upper)]
# 2. Method B: Median Absolute Deviation (MAD) -> 1.4826 scale factor for normal consistency
median_val = np.median(raw_readings)
mad = stats.median_abs_deviation(raw_readings, scale="normal")
modified_z_scores = 0.6745 * (raw_readings - median_val) / stats.median_abs_deviation(raw_readings)
mad_outliers = raw_readings[np.abs(modified_z_scores) > 3.5]
print("=" * 65)
print("OUTLIER DETECTION: TUKEY IQR VS. ROBUST MAD")
print("=" * 65)
print(f"Dataset Size : {len(raw_readings)}")
print(f"Tukey IQR Fences : Lower = {iqr_lower:.2f} | Upper = {iqr_upper:.2f}")
print(f"Tukey Outliers ({len(iqr_outliers)}): {np.sort(iqr_outliers)}")
print("-" * 65)
print(f"Median = {median_val:.2f} | Normal-Scaled MAD = {mad:.2f}")
print(f"MAD Outliers ({len(mad_outliers)}): {np.sort(mad_outliers)}")
import numpy as np
from scipy import stats
np.random.seed(42)
x = np.linspace(1.0, 10.0, 100)
# Monotonic non-linear relationship: y = exp(x) (Perfect monotonic, highly non-linear)
y_monotonic = np.exp(x * 0.5)
# Calculate Pearson (Linear) and Spearman (Rank Monotonic)
r_linear, p_linear = stats.pearsonr(x, y_monotonic)
rho_rank, p_rank = stats.spearmanr(x, y_monotonic)
# Introduce 1 single corrupt extreme outlier point
x_corrupt = np.append(x, [10.0])
y_corrupt = np.append(y_monotonic, [-500.0])
r_corrupt, _ = stats.pearsonr(x_corrupt, y_corrupt)
rho_corrupt, _ = stats.spearmanr(x_corrupt, y_corrupt)
print("=" * 65)
print("PEARSON VS. SPEARMAN CORRELATION BENCHMARK")
print("=" * 65)
print("Clean Non-Linear Exponential Dataset:")
print(f" Pearson r (Linearity Metric) : {r_linear:.4f} (Degraded due to exponential curve)")
print(f" Spearman Ο (Monotonic Metric) : {rho_rank:.4f} (Perfect 1.0 rank order invariant!)")
print("-" * 65)
print("Contaminated with 1 Single Leverage Outlier:")
print(f" Pearson r after Outlier : {r_corrupt:.4f} (Severe degradation)")
print(f" Spearman Ο after Outlier : {rho_corrupt:.4f} (Preserved rank order stability)")
STATISTICAL FOUNDATIONS REFERENCE MATRIX:
ββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββ
β Parameter / Metric β Mathematical Formulation β Robustness / Breakdownβ Machine Learning Application β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β Sample Mean (xΜ) β Ξ£ x_i / n β 0% (Extremely Fragile) Loss functions (MSE), Imputation β
β Median β 50th Percentile β 50% (Maximal Robust) Robust centering, MAE loss bounds β
β Sample Variance (sΒ²) β Ξ£ (x_i - xΜ)Β² / (n - 1) β 0% (Bessel ddof=1) Unbiased dispersion, Scalers β
β Interquartile Range (IQR) β Q3 - Q1 (75th - 25th) β 25% Robust RobustScaler, Tukey outlier filter β
β Median Absolute Dev (MAD) β median(|x_i - median(x)|) β 50% Robust Reliable anomaly z-score filter β
β Skewness (Fisher) β m3 / sΒ³ β Sensitive to Tails Feature transform triggers (BoxCox)β
β Excess Kurtosis β (m4 / sβ΄) - 3 β Heavy-Tail Metric Fat-tail risk, Student-t priors β
β Standard Error (SE) β s / βn β N-dependent Scale A/B test significance, Power checksβ
β Student's t-Interval β xΜ Β± t_(Ξ±/2, n-1) * (s / βn) β Requires Normality Confidence bounds on small samples β
β Bootstrap Resampling CI β Empirical Percentiles(B_dist)β Fully Distribution-Free Arbitrary metric SLAs (P90/P99) β
ββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββ€
β STATISTICAL BEST PRACTICES: β
β 1. Always enforce `ddof=1` when computing sample variance in NumPy (`np.var(x, ddof=1)`) to avoid bias. β
β 2. When features display high skewness (|skew| > 1.0), evaluate Median/IQR rather than Mean/Std for feature scaling. β
β 3. For non-linear relationships or data with outliers, select Spearman's rank correlation over Pearson's linear r. β
β 4. When estimating confidence intervals for non-normal metrics (e.g., P99 latencies), use non-parametric bootstrap. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ