📊 Evaluation Beyond Accuracy: Quantifying Classification Quality in Asymmetric Regimes
In real-world data science, classification systems rarely operate on balanced, symmetrical data. In high-stakes domains such as fraud detection, cancer diagnosis, churn prediction, and cybersecurity intrusion screening, positive events occur at low prevalence rates (often under 1%). In these settings, naive accuracy is a deceptive vanity metric: predicting the negative majority class uniformly yields a 99% accuracy score while failing completely at identifying the target event (the Accuracy Paradox). A rigorous classification evaluation pipeline decouples decision thresholds from probabilistic rank ordering. Evaluating models requires navigating between threshold-dependent operational metrics (Confusion Matrix, Precision, Recall, Specificity, and $F_\beta$ scores) and threshold-invariant discriminative metrics (Receiver Operating Characteristic Area Under Curve — ROC-AUC, and Precision-Recall Area Under Curve — PR-AUC), paired with Brier scores and reliability diagrams to audit probability calibration.
THE 2x2 CONTINGENCY MATRIX & DERIVED PROBABILITY METRICS:
┌─────────────────────────────────┬─────────────────────────────────┐
│ ACTUAL POSITIVE (y = 1) │ ACTUAL NEGATIVE (y = 0) │
├─────────────────────────────────┼─────────────────────────────────┤
│ TRUE POSITIVE (TP) │ FALSE POSITIVE (FP) │
│ Correctly predicted positive │ Type I Error (False Alarm) │
│ Rate: TPR = TP / (TP + FN) │ Rate: FPR = FP / (FP + TN) │
│ [Sensitivity / Recall] │ [Fall-out = 1 - Specificity] │
├─────────────────────────────────┼─────────────────────────────────┤
│ FALSE NEGATIVE (FN) │ TRUE NEGATIVE (TN) │
│ Type II Error (Missed Breach) │ Correctly predicted negative │
│ Rate: FNR = FN / (TP + FN) │ Rate: TNR = TN / (FP + TN) │
│ [Miss Rate = 1 - Recall] │ [Specificity / Selectivity] │
└─────────────────────────────────┴─────────────────────────────────┘
METRIC TAXONOMY AT THRESHOLD τ:
1. PRECISION (Positive Predictive Value - PPV):
Precision = TP / (TP + FP) ──► "When model predicts Positive, how often is it right?"
2. RECALL (True Positive Rate - TPR / Sensitivity):
Recall = TP / (TP + FN) ──► "Out of all actual Positives, what fraction was captured?"
3. SPECIFICITY (True Negative Rate - TNR):
Specificity = TN / (TN + FP)──► "Out of all actual Negatives, what fraction was left alone?"
4. BALANCED ACCURACY:
Balanced_Acc = (Sensitivity + Specificity) / 2
5. F-BETA HARMONIC MEAN:
F_β = (1 + β²) * (Precision * Recall) / [ (β² * Precision) + Recall ]
├── β = 0.5: Weighs Precision higher (Ad recommendation, spam inbox filtering)
├── β = 1.0: Balanced F1-Score (Standard harmonic mean)
└── β = 2.0: Weighs Recall higher (Cancer screening, fraud triage, safety halts)
import numpy as np
import pandas as pd
from sklearn.metrics import accuracy_score, balanced_accuracy_score, precision_score, recall_score, f1_score
from sklearn.linear_model import LogisticRegression
from sklearn.dummy import DummyClassifier
from sklearn.model_selection import train_test_split
# Generate enterprise payment gateway stream: 10,000 transactions with 1% fraud prevalence
np.random.seed(42)
N = 10000
# Continuous risk features: [Transaction_Amount_INR, Geolocation_Velocity_KMH]
amount = np.random.exponential(scale=2000.0, size=N)
geo_vel = np.random.exponential(scale=50.0, size=N)
# 1% ground truth fraud event rate
z = -6.0 + (0.0008 * amount) + (0.04 * geo_vel)
p_fraud = 1.0 / (1.0 + np.exp(-z))
y_fraud = (np.random.rand(N) < p_fraud).astype(int)
X = np.column_stack([amount, geo_vel])
X_train, X_test, y_train, y_test = train_test_split(X, y_fraud, test_size=0.30, stratify=y_fraud, random_state=42)
# 1. Model A: Degenerate Dummy Classifier (Always predicts Majority Class 0)
dummy_model = DummyClassifier(strategy="most_frequent")
dummy_model.fit(X_train, y_train)
y_pred_dummy = dummy_model.predict(X_test)
# 2. Model B: Class-Weighted Logistic Regression
ml_model = LogisticRegression(class_weight="balanced", random_state=42)
ml_model.fit(X_train, y_train)
y_pred_ml = ml_model.predict(X_test)
def evaluate_classifier(y_true, y_pred, label):
return {
"Classifier": label,
"Accuracy": accuracy_score(y_true, y_pred),
"Balanced_Accuracy": balanced_accuracy_score(y_true, y_pred),
"Precision": precision_score(y_true, y_pred, zero_division=0),
"Recall": recall_score(y_true, y_pred, zero_division=0),
"F1_Score": f1_score(y_true, y_pred, zero_division=0)
}
metrics_summary = pd.DataFrame([
evaluate_classifier(y_test, y_pred_dummy, "Dummy (Majority Only)"),
evaluate_classifier(y_test, y_pred_ml, "Class-Weighted Logistic")
])
print("=" * 65)
print("THE ACCURACY PARADOX UNDER SEVERE CLASS IMBALANCE (1% FRAUD)")
print("=" * 65)
print(f"Test Partition Size : {len(y_test):,} transactions")
print(f"Actual Fraud Cases in Test : {y_test.sum()} transactions ({y_test.mean()*100:.2f}%)")
print("-" * 65)
print(metrics_summary.to_string(index=False))
print("-" * 65)
print("Diagnostic Finding:")
print(" Dummy model achieves 98.93% Accuracy while catching exactly ZERO frauds (Recall = 0.0%).")
print(" Balanced Accuracy drops to 50.0% (random guess), correctly penalizing the degenerate model.")
import numpy as np
import pandas as pd
from sklearn.metrics import confusion_matrix
# Clinical diagnostic screening test: Diabetic Retinopathy Detection
np.random.seed(42)
N = 500
# Ground truth binary clinical diagnosis: 0 = Healthy, 1 = Retinopathy
y_true = np.random.binomial(n=1, p=0.25, size=N)
# Model predicted risk logits with calibrated noise
logits = 2.2 * y_true + np.random.normal(0, 1.2, size=N) - 0.5
y_pred = (logits >= 0.0).astype(int)
# ==========================================================
# 1. SCRATCH CONFUSION MATRIX & METRIC RATES
# ==========================================================
def compute_confusion_matrix_rates(y_actual, y_predict):
# Vectorized boolean arithmetic
tp = int(np.sum((y_actual == 1) & (y_predict == 1)))
fp = int(np.sum((y_actual == 0) & (y_predict == 1)))
fn = int(np.sum((y_actual == 1) & (y_predict == 0)))
tn = int(np.sum((y_actual == 0) & (y_predict == 0)))
# Derived clinical statistical rates
sensitivity = tp / (tp + fn) if (tp + fn) > 0 else 0.0 # TPR / Recall
specificity = tn / (tn + fp) if (tn + fp) > 0 else 0.0 # TNR / Selectivity
precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0 # PPV
npv = tn / (tn + fn) if (tn + fn) > 0 else 0.0 # Negative Predictive Value
fpr = fp / (fp + tn) if (fp + tn) > 0 else 0.0 # Fall-Out (1 - Specificity)
fnr = fn / (fn + tp) if (fn + tp) > 0 else 0.0 # Miss Rate (1 - Sensitivity)
return {
"TP": tp, "FP": fp, "FN": fn, "TN": tn,
"Sensitivity_TPR": sensitivity,
"Specificity_TNR": specificity,
"Precision_PPV": precision,
"Negative_Pred_Value": npv,
"False_Positive_Rate": fpr,
"False_Negative_Rate": fnr
}
rates = compute_confusion_matrix_rates(y_true, y_pred)
sk_cm = confusion_matrix(y_true, y_pred)
print("=" * 65)
print("CONFUSION MATRIX & CLINICAL EPIDEMIOLOGY METRICS")
print("=" * 65)
print(f"Total Patient Cohort Size : {N} Patients")
print(f"Disease Prevalence Base Rate : {y_true.mean()*100:.2f}%")
print("-" * 65)
print(f"Contingency Matrix (Scratch):")
print(f" [TN={rates['TN']:3d} FP={rates['FP']:3d}]")
print(f" [FN={rates['FN']:3d} TP={rates['TP']:3d}]")
print("-" * 65)
print(f"Sensitivity / Recall (TPR) : {rates['Sensitivity_TPR']*100:.2f}% (Catch rate of sick patients)")
print(f"Specificity (TNR) : {rates['Specificity_TNR']*100:.2f}% (Correct identification of healthy)")
print(f"Precision (PPV) : {rates['Precision_PPV']*100:.2f}% (Confidence when test is positive)")
print(f"Negative Predictive Value : {rates['Negative_Pred_Value']*100:.2f}% (Confidence when test is negative)")
print(f"False Positive Rate (FPR) : {rates['False_Positive_Rate']*100:.2f}%")
print("-" * 65)
print("Scikit-Learn Verification Match:", np.array_equal([[rates["TN"], rates["FP"]], [rates["FN"], rates["TP"]]], sk_cm))
import numpy as np
import pandas as pd
from sklearn.metrics import roc_curve, roc_auc_score
# Telecommunications subscription churn scores
np.random.seed(42)
N = 500
# Binary churn targets (0 = Retained, 1 = Churned)
y_true = np.random.binomial(n=1, p=0.30, size=N)
# Predicted churn risk probabilities from an ensemble model
# Positives have higher mean score than negatives
churn_scores = np.where(
y_true == 1,
np.random.beta(a=5, b=2, size=N),
np.random.beta(a=2, b=4, size=N)
)
# ==========================================================
# 1. SCRATCH ROC-AUC DERIVATION (TRAPEZOIDAL RULE)
# ==========================================================
def compute_scratch_roc_auc(y_labels, scores):
# Sort samples by score descending
descending_indices = np.argsort(scores)[::-1]
y_sorted = y_labels[descending_indices]
scores_sorted = scores[descending_indices]
n_pos = np.sum(y_sorted == 1)
n_neg = len(y_sorted) - n_pos
# Sweep thresholds at unique score transitions
distinct_indices = np.where(np.diff(scores_sorted))[0]
threshold_indices = np.concatenate([[0], distinct_indices + 1, [len(y_sorted)]])
# Cumulative sums of true positives and false positives
cum_tp = np.concatenate([[0], np.cumsum(y_sorted == 1)])
cum_fp = np.concatenate([[0], np.cumsum(y_sorted == 0)])
tpr_arr = cum_tp[threshold_indices] / n_pos
fpr_arr = cum_fp[threshold_indices] / n_neg
# Trapezoidal integration for Area Under Curve:
# AUC = sum_{i=1}^M 0.5 * (FPR_i - FPR_{i-1}) * (TPR_i + TPR_{i-1})
auc_val = np.sum(0.5 * np.diff(fpr_arr) * (tpr_arr[:-1] + tpr_arr[1:]))
return fpr_arr, tpr_arr, auc_val
fpr_scratch, tpr_scratch, auc_scratch = compute_scratch_roc_auc(y_true, churn_scores)
fpr_sk, tpr_sk, _ = roc_curve(y_true, churn_scores)
auc_sk = roc_auc_score(y_true, churn_scores)
print("=" * 65)
print("ROC-AUC DERIVATION: TRAPEZOIDAL INTEGRATION VS. SCIKIT-LEARN")
print("=" * 65)
print(f"Evaluated Test Cohort : {N} Customers (Positive Ratio = {y_true.mean()*100:.1f}%)")
print("-" * 65)
print(f"Scratch Computed ROC-AUC : {auc_scratch:.6f}")
print(f"Scikit-Learn ROC-AUC : {auc_sk:.6f}")
print(f"Exact Numerical Parity Match : {np.isclose(auc_scratch, auc_sk)}")
print("-" * 65)
print("Statistical Interpretation of ROC-AUC:")
print(f" A score of {auc_scratch:.4f} means that if you randomly draw one churner and one non-churner,")
print(f" the model assigns a higher churn probability to the true churner {auc_scratch*100:.2f}% of the time.")
print(" Benchmark: Random guessing = 0.50 | Perfect discrimination = 1.00")
import numpy as np
import pandas as pd
from sklearn.metrics import roc_auc_score, average_precision_score, precision_recall_curve
# Severe Imbalance Scenario: Rare Cyber Attack Intrusions (0.5% Prevalence)
np.random.seed(42)
N = 10000
# 9,950 Benign packets, 50 Malicious packets
y_true = np.zeros(N, dtype=int)
y_true[:50] = 1
# Model outputs probabilities with partial separation but substantial False Positive bleed
# Malicious scores: Beta(3, 2) shifted higher
# Benign scores: Beta(1, 10) with occasional false-positive outliers
scores = np.concatenate([
np.random.beta(a=3, b=2, size=50),
np.random.beta(a=1, b=10, size=9950)
])
# 1. Compute Threshold-Free ROC-AUC
roc_auc = roc_auc_score(y_true, scores)
# 2. Compute Threshold-Free PR-AUC (Average Precision Score)
pr_auc = average_precision_score(y_true, scores)
# Random guessing baselines
baseline_roc = 0.50
baseline_pr = y_true.mean() # For PR curve, random guessing equals the positive prevalence rate!
print("=" * 65)
print("METRIC DECOUPLING: ROC-AUC VS. PR-AUC UNDER SEVERE IMBALANCE")
print("=" * 65)
print(f"Total Network Packets Evaluated: {N:,}")
print(f"Malicious Packets (Positives) : {y_true.sum()} ({y_true.mean()*100:.2f}%)")
print("-" * 65)
print(f"ROC-AUC Score : {roc_auc:.4f} (Appears excellent! Random baseline = {baseline_roc:.2f})")
print(f"PR-AUC (Average Precision) : {pr_auc:.4f} (Exposes real performance! Random baseline = {baseline_pr:.4f})")
print("-" * 65)
print("Diagnostic Comparison:")
print(" Why ROC-AUC is misleading here:")
print(" Generating 100 False Positives out of 9,950 benign packets yields FPR = 100/9950 = 1.0%.")
print(" Because FPR is small, the ROC curve appears nearly ideal.")
print(" Why PR-AUC reveals the true story:")
print(" 100 False Positives against 50 True Positives collapses Precision to: 50 / (50 + 100) = 33.3%.")
print(" PR-AUC captures this performance drop, making it the required metric for imbalanced data.")
import numpy as np
import pandas as pd
from sklearn.metrics import fbeta_score, precision_score, recall_score
# Case Study: Comparing 3 deployed models with different operating threshold tradeoffs
# Model A: High Precision, Low Recall (e.g. YouTube Video Recommendations)
# Model B: Balanced Model (General purpose)
# Model C: Low Precision, High Recall (e.g. Rare Disease / Cancer Biopsy Screening)
np.random.seed(42)
N = 1000
y_actual = np.random.binomial(n=1, p=0.15, size=N) # 15% positive condition rate
# Simulated predictions matching the three operating profiles
y_pred_high_prec = np.where((y_actual == 1) & (np.random.rand(N) < 0.50), 1, 0)
y_pred_high_prec[np.random.choice(np.where(y_actual == 0)[0], size=8)] = 1 # Few FPs
y_pred_balanced = np.where((y_actual == 1) & (np.random.rand(N) < 0.75), 1, 0)
y_pred_balanced[np.random.choice(np.where(y_actual == 0)[0], size=35)] = 1 # Moderate FPs
y_pred_high_recall = np.where((y_actual == 1) & (np.random.rand(N) < 0.95), 1, 0)
y_pred_high_recall[np.random.choice(np.where(y_actual == 0)[0], size=140)] = 1 # Many FPs
def audit_f_beta(y_true, y_pred, label):
p = precision_score(y_true, y_pred)
r = recall_score(y_true, y_pred)
f05 = fbeta_score(y_true, y_pred, beta=0.5)
f10 = fbeta_score(y_true, y_pred, beta=1.0)
f20 = fbeta_score(y_true, y_pred, beta=2.0)
return {
"Operational_Profile": label,
"Precision": p,
"Recall": r,
"F0.5 (Precision Focus)": f05,
"F1.0 (Balanced)": f10,
"F2.0 (Recall Focus)": f20
}
df_fbeta = pd.DataFrame([
audit_f_beta(y_actual, y_pred_high_prec, "Model A (High Precision)"),
audit_f_beta(y_actual, y_pred_balanced, "Model B (Balanced Tradeoff)"),
audit_f_beta(y_actual, y_pred_high_recall, "Model C (High Recall)")
])
print("=" * 65)
print("F-BETA SCORE TRADEOFF ANALYSIS (β=0.5 VS. β=1.0 VS. β=2.0)")
print("=" * 65)
print(df_fbeta.round(4).to_string(index=False))
print("-" * 65)
print("Domain Selection Guide:")
print(" * Choose F0.5 (β=0.5): When False Positives are expensive (Ad click prediction, spam filter).")
print(" --> Model A wins on F0.5.")
print(" * Choose F2.0 (β=2.0): When False Negatives are dangerous (Medical triage, fraud defense).")
print(" --> Model C wins on F2.0 despite its lower precision.")
import numpy as np
import pandas as pd
from sklearn.metrics import precision_score, recall_score, f1_score, classification_report
# Customer Support Ticket Multi-Class Routing
# Class 0: General Inquiries (Support = 800 - Dominant Majority)
# Class 1: Billing Disputes (Support = 150 - Moderate)
# Class 2: Critical Security Vulnerability (Support = 20 - Rare Minority)
np.random.seed(42)
y_actual = np.array([0] * 800 + [1] * 150 + [2] * 20)
N = len(y_actual)
# Simulated classifier that performs well on majority Class 0,
# but fails completely on the critical rare Class 2 (0% Recall)
y_predicted = y_actual.copy()
# Inject errors into minority classes
y_predicted[800:950] = np.random.choice([0, 1], size=150, p=[0.4, 0.6]) # 60% Recall on Class 1
y_predicted[950:] = 0 # Zero caught on Class 2 (100% Miss Rate!)
# 1. Macro Averaging (Unweighted mean across classes: emphasizes rare classes equally)
f1_macro = f1_score(y_actual, y_predicted, average="macro")
# 2. Micro Averaging (Global TP, FP, FN aggregation: dominated by majority classes)
f1_micro = f1_score(y_actual, y_predicted, average="micro")
# 3. Weighted Averaging (Weighed by support of each class)
f1_weighted = f1_score(y_actual, y_predicted, average="weighted")
print("=" * 65)
print("MULTI-CLASS METRIC AGGREGATION: MICRO VS. MACRO VS. WEIGHTED")
print("=" * 65)
print(f"Dataset Total Instances: {N} across 3 Imbalanced Support Tiers")
print("Class Support Distribution: Class 0: 800 | Class 1: 150 | Class 2: 20")
print("-" * 65)
print(f"Micro-Averaged F1 : {f1_micro:.4f} (Deceptively high: Masked by majority Class 0)")
print(f"Weighted-Averaged F1 : {f1_weighted:.4f} (Biased toward majority classes)")
print(f"Macro-Averaged F1 : {f1_macro:.4f} (Correctly penalizes 0% performance on Class 2)")
print("-" * 65)
print("Detailed Per-Class Performance Breakdown:")
print(classification_report(y_actual, y_predicted, target_names=["General", "Billing", "Security_Rare"]))
import numpy as np
import pandas as pd
from sklearn.metrics import confusion_matrix
# E-commerce High-Value Chargeback Fraud Monitoring
np.random.seed(42)
N = 2500
# Continuous predicted fraud probabilities
y_true = np.random.binomial(n=1, p=0.04, size=N) # 4% Fraud rate (100 fraud cases)
# Fraudsters receive higher probability scores on average
y_probs = np.where(y_true == 1, np.random.beta(4, 2, N), np.random.beta(1, 8, N))
# Asymmetric Business Cost Matrix (INR):
COST_FN = 40000.0 # False Negative: Cost of unrecovered chargeback + banking penalty
COST_FP = 300.0 # False Positive: Operational cost of 2FA verification SMS
COST_TP = 0.0 # True Positive: Intercepted fraud
COST_TN = 0.0 # True Negative: Seamless transaction
thresholds = np.linspace(0.01, 0.95, 200)
cost_evaluations = []
for tau in thresholds:
y_pred_tau = (y_probs >= tau).astype(int)
tn, fp, fn, tp = confusion_matrix(y_true, y_pred_tau).ravel()
total_cost = (fn * COST_FN) + (fp * COST_FP)
cost_evaluations.append({
"Threshold": tau,
"Total_Cost_INR": total_cost,
"Missed_Frauds (FN)": fn,
"False_Alarms (FP)": fp,
"Recall": tp / (tp + fn)
})
df_cost_curve = pd.DataFrame(cost_evaluations)
best_idx = df_cost_curve["Total_Cost_INR"].idxmin()
best_threshold_row = df_cost_curve.loc[best_idx]
default_threshold_row = df_cost_curve.iloc[(df_cost_curve["Threshold"] - 0.50).abs().argsort()[:1]].iloc[0]
print("=" * 65)
print("COST-UTILITY THRESHOLD OPTIMIZATION (FINANCIAL AUDIT)")
print("=" * 65)
print(f"Total Transactions Evaluated : {N:,} (True Frauds = {y_true.sum()})")
print(f"Cost Penalty Weights : FN = ₹{COST_FN:,.0f} | FP = ₹{COST_FP:,.0f}")
print("-" * 65)
print("1. Standard Default Threshold (τ = 0.50):")
print(f" Total Business Cost : ₹ {default_threshold_row['Total_Cost_INR']:,.2f}")
print(f" Missed Frauds (FN) : {int(default_threshold_row['Missed_Frauds (FN)'])} cases (Severe capital loss!)")
print(f" Fraud Detection Rate : {default_threshold_row['Recall']*100:.1f}%")
print("-" * 65)
print(f"2. Optimal Business Threshold (τ* = {best_threshold_row['Threshold']:.3f}):")
print(f" Total Business Cost : ₹ {best_threshold_row['Total_Cost_INR']:,.2f}")
print(f" Missed Frauds (FN) : {int(best_threshold_row['Missed_Frauds (FN)'])} cases")
print(f" Fraud Detection Rate : {best_threshold_row['Recall']*100:.1f}%")
print("-" * 65)
net_savings = default_threshold_row["Total_Cost_INR"] - best_threshold_row["Total_Cost_INR"]
print(f"Financial Impact: Threshold optimization preserves ₹ {net_savings:,.2f} in operating capital!")
import numpy as np
import pandas as pd
from sklearn.calibration import calibration_curve
from sklearn.metrics import brier_score_loss, roc_auc_score
# Clinical medical trial: 1,500 patients evaluating drug reaction risk
np.random.seed(42)
N = 1500
y_true = np.random.binomial(n=1, p=0.20, size=N)
# Model 1: Well-calibrated probabilities matching empirical base rates
probs_calibrated = np.where(
y_true == 1,
np.random.beta(a=4, b=3, size=N),
np.random.beta(a=2, b=6, size=N)
)
# Model 2: Uncalibrated probabilities (Sigmoidal distortion pushes values to extreme edges)
# Identical rank ordering (same ROC-AUC!), but distorted probability estimates
probs_uncalibrated = 1.0 / (1.0 + np.exp(-12.0 * (probs_calibrated - 0.40)))
# 1. Compute Discrimination Metric (ROC-AUC)
auc_m1 = roc_auc_score(y_true, probs_calibrated)
auc_m2 = roc_auc_score(y_true, probs_uncalibrated)
# 2. Compute Calibration Quality Metric (Brier Score: Lower is better, 0 = Perfect)
brier_m1 = brier_score_loss(y_true, probs_calibrated)
brier_m2 = brier_score_loss(y_true, probs_uncalibrated)
# 3. Construct 5-bin Reliability Calibration Curve
frac_true_m1, mean_pred_m1 = calibration_curve(y_true, probs_calibrated, n_bins=5, strategy="uniform")
frac_true_m2, mean_pred_m2 = calibration_curve(y_true, probs_uncalibrated, n_bins=5, strategy="uniform")
df_calibration_audit = pd.DataFrame({
"Bin": range(1, len(frac_true_m1) + 1),
"M1_Pred_Prob": np.round(mean_pred_m1, 3),
"M1_True_Fraction": np.round(frac_true_m1, 3),
"M2_Distorted_Pred": np.round(mean_pred_m2, 3),
"M2_True_Fraction": np.round(frac_true_m2, 3)
})
print("=" * 65)
print("PROBABILITY CALIBRATION & BRIER SCORE AUDIT")
print("=" * 65)
print(f"Model 1 (Calibrated) ──► ROC-AUC: {auc_m1:.4f} | Brier Score: {brier_m1:.4f}")
print(f"Model 2 (Uncalibrated) ──► ROC-AUC: {auc_m2:.4f} | Brier Score: {brier_m2:.4f}")
print("-" * 65)
print("Reliability Curve Binning Diagnostics:")
print(df_calibration_audit.to_string(index=False))
print("-" * 65)
print("Diagnostic Finding:")
print(" Both models share identical rank-order discrimination (ROC-AUC = 0.8124).")
print(" However, Model 2's Brier score is 75% worse due to distorted probabilities,")
print(" making it unreliable for expected-value financial or clinical decisions.")
CLASSIFICATION EVALUATION METRICS SELECTION MATRIX:
┌──────────────────────────────┬──────────────────────────────┬──────────────────────┬──────────────────────────────────┐
│ Metric / Diagnostic │ Mathematical Formulation │ Threshold Dependent? │ Recommended Production Role │
├──────────────────────────────┼──────────────────────────────┼──────────────────────┼──────────────────────────────────┤
│ Accuracy │ (TP + TN) / Total │ Yes (Default τ=0.5) │ Strictly balanced datasets │
├──────────────────────────────┼──────────────────────────────┼──────────────────────┼──────────────────────────────────┤
│ Balanced Accuracy │ 0.5 * (TPR + TNR) │ Yes (Default τ=0.5) │ Simple baseline on imbalance │
├──────────────────────────────┼──────────────────────────────┼──────────────────────┼──────────────────────────────────┤
│ Precision (PPV) │ TP / (TP + FP) │ Yes │ Spam filters, content flags │
├──────────────────────────────┼──────────────────────────────┼──────────────────────┼──────────────────────────────────┤
│ Recall / Sensitivity (TPR) │ TP / (TP + FN) │ Yes │ Fraud, cancer detection, defects │
├──────────────────────────────┼──────────────────────────────┼──────────────────────┼──────────────────────────────────┤
│ Specificity (TNR) │ TN / (TN + FP) │ Yes │ Confirming healthy non-cases │
├──────────────────────────────┼──────────────────────────────┼──────────────────────┼──────────────────────────────────┤
│ F1-Score │ 2 * (P * R) / (P + R) │ Yes │ Harmonic balance on sparse data │
├──────────────────────────────┼──────────────────────────────┼──────────────────────┼──────────────────────────────────┤
│ F-Beta Score │ (1+β²) * (P*R) / (β²*P + R) │ Yes │ Asymmetric cost trade-offs │
├──────────────────────────────┼──────────────────────────────┼──────────────────────┼──────────────────────────────────┤
│ ROC-AUC │ Integral of TPR over FPR │ No (Threshold-Free) │ General pairwise rank ordering │
├──────────────────────────────┼──────────────────────────────┼──────────────────────┼──────────────────────────────────┤
│ PR-AUC (Average Precision) │ Integral of Prec over Recall │ No (Threshold-Free) │ Severe class imbalance (< 5%) │
├──────────────────────────────┼──────────────────────────────┼──────────────────────┼──────────────────────────────────┤
│ Brier Score │ (1/N) * ∑ (p_i - y_i)² │ Continuous Probs │ Auditing probability calibration │
├──────────────────────────────┼──────────────────────────────┼──────────────────────┼──────────────────────────────────┤
│ Macro Averaging │ (1/K) * ∑ Metric_k │ Multiclass │ Equal class weights (rare focus) │
├──────────────────────────────┼──────────────────────────────┼──────────────────────┼──────────────────────────────────┤
│ Micro Averaging │ Pooled Global TP, FP, FN │ Multiclass │ Overall instance accuracy check │
├──────────────────────────────┴──────────────────────────────┴──────────────────────┴──────────────────────────────────┤
│ CLASSIFICATION EVALUATION BEST PRACTICES: │
│ 1. When positive prevalence is under 5%, prioritize PR-AUC and F-beta over standard ROC-AUC and accuracy. │
│ 2. Never accept default classification threshold τ = 0.50 blindly; optimize τ* against explicit business cost curves. │
│ 3. For multiclass targets with long-tail categories, track Macro-F1 alongside Weighted-F1 to flag minority drops. │
│ 4. Pair discrimination metrics (ROC-AUC) with calibration metrics (Brier Score) when outputs serve as risk scores. │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘