π¬ From Raw Records to Decision Intelligence: The Data Science Architecture
Data science is the systematic convergence of domain expertise, mathematical statistics, and scalable software engineering. Modern enterprise data platforms do not simply train algorithms in isolation; they execute an end-to-end, iterative journey defined by the industry-standard CRISP-DM (Cross-Industry Standard Process for Data Mining) framework. From framing key commercial KPIs to executing vectorized transformations, identifying feature leakage, optimizing model decision boundaries, and monitoring inference distributions in production, Python provides the canonical scientific ecosystem: NumPy for low-level memory buffers, Pandas for relational tabular analytics, Scikit-Learn for statistical learning estimators, and Matplotlib for visual diagnostic signals.
THE END-TO-END DATA SCIENCE LIFECYCLE (CRISP-DM):
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 1. BUSINESS UNDERSTANDING β
β Define core commercial KPIs: Churn Reduction, Underwriting Risk, LTV β
β Translate domain targets into statistical loss functions and thresholds β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 2. DATA UNDERSTANDING β
β Ingest raw tabular, sensor, and JSON feeds across transactional OLTP DBs β
β Audit distributions, skewness, missing values, and data type integrity β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 3. DATA PREPARATION (The Heavy Engine ~70% of Pipeline) β
β Handling missing values (MICE/KNN), One-Hot/Target Encoding, Scaling β
β Outlier winsorization, interaction feature synthesis, leakage prevention β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 4. MODELING β
β Train candidate estimators: Regularized GLMs, Random Forests, XGBoost β
β Hyperparameter optimization via Bayesian tuning (Optuna/GridSearchCV) β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 5. EVALUATION β
β Assess out-of-sample generalization: ROC-AUC, Precision-Recall, F1-Score β
β Conduct error analysis and business cost-benefit payoff matrix checks β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β 6. DEPLOYMENT & MONITORING β
β Serialize models (ONNX / Joblib) and deploy REST APIs via FastAPI/Docker β
β Continuous drift detection: Population Stability Index (PSI) & KS-Tests β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
import numpy as np
def calculate_business_impact(y_true: np.ndarray, y_pred: np.ndarray,
cost_fn: float, cost_fp: float,
gain_tp: float, gain_tn: float) -> dict:
"""
Evaluates classification utility using a customized financial cost-benefit payoff matrix.
Domain: Loan Default Risk / Fraud Detection
"""
tp = np.sum((y_true == 1) & (y_pred == 1))
fp = np.sum((y_true == 0) & (y_pred == 1))
fn = np.sum((y_true == 1) & (y_pred == 0))
tn = np.sum((y_true == 0) & (y_pred == 0))
# Compute financial balance sheet impact
total_revenue = (tp * gain_tp) + (tn * gain_tn)
total_loss = (fn * cost_fn) + (fp * cost_fp)
net_economic_value = total_revenue - total_loss
return {
"Confusion Matrix": {"TP": int(tp), "FP": int(fp), "FN": int(fn), "TN": int(tn)},
"Total Revenue": round(total_revenue, 2),
"Total Loss": round(total_loss, 2),
"Net Economic Value": round(net_economic_value, 2)
}
# Ground truth defaults (1 = Default, 0 = Non-Default) across 1,000 customers
np.random.seed(42)
actual_outcomes = np.random.choice([0, 1], size=1000, p=[0.85, 0.15])
# Model A (High Recall, Aggressive Intervention)
predictions_a = actual_outcomes.copy()
# Introduce realistic misclassifications
predictions_a[np.random.choice(1000, size=50, replace=False)] = 1
# Financial stakes: False Negative (Charge-off loss) = βΉ 2,50,000; False Positive (Friction/Review) = βΉ 2,000
report = calculate_business_impact(
y_true=actual_outcomes,
y_pred=predictions_a,
cost_fn=250000.0,
cost_fp=2000.0,
gain_tp=50000.0,
gain_tn=15000.0
)
print(f"Confusion Matrix : {report['Confusion Matrix']}")
print(f"Net Economic Gain: βΉ {report['Net Economic Value']:,.2f}")
import pandas as pd
import numpy as np
def generate_data_health_profile(df: pd.DataFrame) -> pd.DataFrame:
"""
Computes diagnostic health metrics across features for initial EDA audits.
"""
profile_records = []
for col in df.columns:
col_series = df[col]
dtype = col_series.dtype
null_count = col_series.isnull().sum()
null_pct = (null_count / len(df)) * 100
unique_count = col_series.nunique()
is_numeric = np.issubdtype(dtype, np.number)
skewness = round(col_series.skew(), 2) if is_numeric else np.nan
memory_usage_kb = col_series.memory_usage(deep=True) / 1024
profile_records.append({
"Feature": col,
"Type": str(dtype),
"Null Count": null_count,
"Null (%)": round(null_pct, 2),
"Cardinality": unique_count,
"Skewness": skewness,
"Memory (KB)": round(memory_usage_kb, 2)
})
return pd.DataFrame(profile_records).sort_values(by="Null (%)", ascending=False)
# Simulated enterprise applicant transactional feed
raw_data = pd.DataFrame({
"applicant_id": [f"APP-{i:05d}" for i in range(1000)],
"age": np.random.choice([25, 32, 45, np.nan, 58], size=1000, p=[0.3, 0.3, 0.2, 0.1, 0.1]),
"annual_income_inr": np.random.exponential(scale=800000, size=1000),
"city_tier": np.random.choice(["Tier-1", "Tier-2", "Tier-3", None], size=1000, p=[0.5, 0.3, 0.15, 0.05]),
"credit_score": np.random.normal(loc=720, scale=40, size=1000),
"default_flag": np.random.choice([0, 1], size=1000, p=[0.9, 0.1])
})
health_audit = generate_data_health_profile(raw_data)
print(health_audit.to_string(index=False))
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
# Separate feature matrix X and target vector y
X = raw_data.drop(columns=["applicant_id", "default_flag"])
y = raw_data["default_flag"]
# 1. Stratified Partitioning (Prevents test distribution skew)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.20, random_state=42, stratify=y
)
# 2. Define Feature Subsets
numeric_features = ["age", "annual_income_inr", "credit_score"]
categorical_features = ["city_tier"]
# 3. Build Sub-Pipelines
numeric_transformer = Pipeline(steps=[
("imputer", SimpleImputer(strategy="median")),
("scaler", StandardScaler())
])
categorical_transformer = Pipeline(steps=[
("imputer", SimpleImputer(strategy="constant", fill_value="UNKNOWN")),
("encoder", OneHotEncoder(handle_unknown="ignore", sparse_output=False))
])
# 4. Integrate into Master ColumnTransformer
preprocessor = ColumnTransformer(transformers=[
("num", numeric_transformer, numeric_features),
("cat", categorical_transformer, categorical_features)
])
# Fit ONLY on X_train to prevent feature leakage
X_train_processed = preprocessor.fit_transform(X_train)
X_test_processed = preprocessor.transform(X_test)
print(f"X_train Processed Matrix Shape: {X_train_processed.shape}")
print(f"X_test Processed Matrix Shape : {X_test_processed.shape}")
print(f"Transformed Feature Columns : {preprocessor.get_feature_names_out()}")
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score, average_precision_score
# Candidate Model A: L2 Regularized Logistic Regression
clf_logistic = Pipeline(steps=[
("preprocessor", preprocessor),
("classifier", LogisticRegression(class_weight="balanced", random_state=42))
])
# Candidate Model B: Random Forest Ensemble
clf_rf = Pipeline(steps=[
("preprocessor", preprocessor),
("classifier", RandomForestClassifier(n_estimators=100, max_depth=6, class_weight="balanced", random_state=42))
])
# Fit both models
clf_logistic.fit(X_train, y_train)
clf_rf.fit(X_train, y_train)
# Predict probabilities for the positive class (Default = 1)
y_prob_log = clf_logistic.predict_proba(X_test)[:, 1]
y_prob_rf = clf_rf.predict_proba(X_test)[:, 1]
# Evaluate discriminatory performance
results = {
"Logistic Regression": {
"ROC-AUC": round(roc_auc_score(y_test, y_prob_log), 4),
"PR-AUC": round(average_precision_score(y_test, y_prob_log), 4)
},
"Random Forest": {
"ROC-AUC": round(roc_auc_score(y_test, y_prob_rf), 4),
"PR-AUC": round(average_precision_score(y_test, y_prob_rf), 4)
}
}
for model_name, metrics in results.items():
print(f"[{model_name}] ROC-AUC: {metrics['ROC-AUC']} | PR-AUC: {metrics['PR-AUC']}")
from sklearn.metrics import precision_recall_curve, f1_score
def find_optimal_threshold(y_true: np.ndarray, y_probs: np.ndarray) -> tuple:
"""
Identifies the probability threshold that maximizes the F1-Score on out-of-fold data.
"""
precisions, recalls, thresholds = precision_recall_curve(y_true, y_probs)
# Calculate harmonic mean (F1) across all evaluated threshold steps
f1_scores = (2 * precisions * recalls) / (precisions + recalls + 1e-10)
best_idx = np.argmax(f1_scores)
return thresholds[best_idx], precisions[best_idx], recalls[best_idx], f1_scores[best_idx]
opt_threshold, best_prec, best_rec, best_f1 = find_optimal_threshold(y_test.values, y_prob_rf)
print(f"Default Threshold (0.50) F1-Score : {round(f1_score(y_test, y_prob_rf >= 0.5), 4)}")
print(f"Optimal Tuned Threshold : {round(opt_threshold, 4)}")
print(f" Precision at Optimal Threshold : {round(best_prec, 4)}")
print(f" Recall at Optimal Threshold : {round(best_rec, 4)}")
print(f" Maximized F1-Score : {round(best_f1, 4)}")
import joblib
import json
import os
import tempfile
from datetime import datetime
artifact_dir = os.path.join(tempfile.gettempdir(), "credit_risk_model_v1")
os.makedirs(artifact_dir, exist_ok=True)
model_filepath = os.path.join(artifact_dir, "model_pipeline.joblib")
metadata_filepath = os.path.join(artifact_dir, "metadata.json")
# 1. Serialize Unified Pipeline (Preprocessor + Random Forest)
joblib.dump(clf_rf, model_filepath, compress=3)
# 2. Generate Audit Metadata
metadata = {
"model_name": "CreditRiskUnderwriting",
"version": "1.0.4",
"trained_at": datetime.utcnow().isoformat() + "Z",
"optimal_threshold": round(float(opt_threshold), 4),
"features_required": list(X.columns),
"metrics": {
"test_roc_auc": round(float(results["Random Forest"]["ROC-AUC"]), 4),
"test_pr_auc": round(float(results["Random Forest"]["PR-AUC"]), 4)
}
}
with open(metadata_filepath, "w") as f:
json.dump(metadata, f, indent=2)
print(f"Model Pipeline Stored: {model_filepath} ({os.path.getsize(model_filepath)} bytes)")
print(f"Metadata Manifest : {metadata_filepath}")
class CreditRiskInferenceEngine:
"""Production wrapper loading serialized pipeline artifacts for low-latency inference."""
def __init__(self, model_path: str, meta_path: str):
self.model = joblib.load(model_path)
with open(meta_path, "r") as f:
self.metadata = json.load(f)
self.threshold = self.metadata["optimal_threshold"]
def predict_risk(self, applicant_record: dict) -> dict:
# Convert dictionary to DataFrame for Pipeline compatibility
df_input = pd.DataFrame([applicant_record])
# Calculate raw default probability
prob_default = float(self.model.predict_proba(df_input)[:, 1][0])
decision = "DECLINE_OR_SCRUTINY" if prob_default >= self.threshold else "AUTO_APPROVE"
return {
"applicant_name": applicant_record.get("applicant_name", "Anonymous"),
"default_probability": round(prob_default, 4),
"threshold_applied": self.threshold,
"decision": decision,
"risk_tier": "HIGH" if prob_default >= 0.6 else ("MEDIUM" if prob_default >= self.threshold else "LOW")
}
# Instantiate inference client
engine = CreditRiskInferenceEngine(model_filepath, metadata_filepath)
# Score real-world payload
live_applicant = {
"applicant_name": "Aarav Sharma",
"age": 28,
"annual_income_inr": 1250000.0,
"city_tier": "Tier-1",
"credit_score": 780.0
}
scoring_result = engine.predict_risk(live_applicant)
print("Production Scoring Outcome:")
for key, value in scoring_result.items():
print(f" {key:20s}: {value}")
def calculate_psi(baseline: np.ndarray, current: np.ndarray, num_buckets: int = 10) -> float:
"""
Calculates the Population Stability Index (PSI) between baseline and production distributions.
Thresholds: PSI < 0.10 (Stable), 0.10 <= PSI < 0.25 (Moderate Shift), PSI >= 0.25 (Action Required)
"""
# Create reference quantile breakpoints from baseline
quantiles = np.linspace(0, 100, num_buckets + 1)
bucket_bounds = np.percentile(baseline, quantiles)
# Adjust outer bounds to capture floating extremes
bucket_bounds[0] -= 1e-5
bucket_bounds[-1] += 1e-5
# Bucket frequency counts
base_counts, _ = np.histogram(baseline, bins=bucket_bounds)
curr_counts, _ = np.histogram(current, bins=bucket_bounds)
# Convert to proportions with smoothing epsilon to prevent division by zero
eps = 1e-4
base_pct = (base_counts / len(baseline)) + eps
curr_pct = (curr_counts / len(current)) + eps
# Re-normalize
base_pct /= np.sum(base_pct)
curr_pct /= np.sum(curr_pct)
# PSI calculation: sum((Current - Base) * ln(Current / Base))
psi_vector = (curr_pct - base_pct) * np.log(curr_pct / base_pct)
return float(np.sum(psi_vector))
# Baseline credit scores vs. Drifted live traffic (e.g., market economic downturn)
baseline_scores = np.random.normal(loc=720, scale=40, size=5000)
drifted_production_scores = np.random.normal(loc=675, scale=55, size=2000)
psi_metric = calculate_psi(baseline_scores, drifted_production_scores)
print(f"Calculated Feature PSI: {round(psi_metric, 4)}")
if psi_metric >= 0.25:
print("ALERT: Severe feature drift detected! Automatic model retraining triggered.")
elif psi_metric >= 0.10:
print("WARNING: Moderate distribution shift observed. Monitor inference calibration.")
else:
print("STATUS: Feature distribution remains stable.")
DATA SCIENCE LIFECYCLE REFERENCE MATRIX:
ββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β CRISP-DM Stage β Core Python Technologies β Primary Deliverables & Architectural Focus β
ββββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β 1. Business Understanding β SciPy, Statsmodels, NumPy β Financial cost-benefit matrix, metric definition (PR-AUC) β
β 2. Data Understanding β Pandas, Seaborn, Polars β Profiling reports, cardinality audits, skewness tests β
β 3. Data Preparation β Scikit-Learn (ColumnTrans..) β Leakage-free preprocessing pipelines, encoders, scalers β
β 4. Modeling β Scikit-Learn, XGBoost, Optunaβ Regularized baselines, ensemble models, hyper-tuning β
β 5. Model Evaluation β Yellowbrick, Shap, Matplotlibβ Confusion matrix, optimal threshold tuning, ROC curves β
β 6. Deployment & Monitoring β Joblib, FastAPI, Evidently β Serialized pipeline bundles, low-latency REST APIs, PSI β
ββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ΄ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β PRODUCTION DATA SCIENCE BEST PRACTICES: β
β 1. Always encapsulate preprocessors and estimators into a single Scikit-Learn Pipeline to eliminate leakage. β
β 2. Optimize classification thresholds based on business economic costs rather than default 0.50 probabilities. β
β 3. Log data schemas and version artifacts alongside serialized weights to ensure reproducible inferences. β
β 4. Track input distribution drift (PSI / KS-Test) continuously to alert when model retraining is required. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ