π§ From Biological Synapses to Deep Function Approximators
Every modern deep learning architectureβfrom computer vision convolutional backbones to generative transformer modelsβoriginates from the fundamental computational model of an artificial neuron. Introduced by Frank Rosenblatt in 1957, the Perceptron mathematically formalizes dendritic summation and axonal firing: it computes an inner product of continuous input signals with learnable synaptic weights, applies a scalar bias, and evaluates a step threshold function. While single-layer perceptrons are mathematically limited to linearly separable decision boundaries (a constraint proven by Marvin Minsky and Seymour Papert via the XOR problem in 1969), stacking neurons into cascaded hidden layers with non-linear activation functions produces the Multi-Layer Perceptron (MLP) / Feedforward Neural Network (FNN). By Cybenko and Hornik's Universal Approximation Theorem, an MLP featuring even a single finite non-linear hidden layer can approximate any continuous multivariate function on compact subsets of $\mathbb{R}^D$ to arbitrary precision. Training these deep computational graphs relies on Backpropagationβan efficient reverse-mode automatic differentiation technique using the multivariable calculus chain rule to calculate exact loss gradients ($\frac{\partial \mathcal{L}}{\partial \mathbf{W}}$) across millions of parameters.
THE ARTIFICIAL NEURON (PERCEPTRON) COMPUTATIONAL GRAPH:
Inputs (Dendrites) Weights (Synapses) Summation & Bias Activation (Axon)
xβ βββββββββββββββΊ [ wβ ] ββββ
xβ βββββββββββββββΊ [ wβ ] ββββΌβββΊ [ z = β(w_j * x_j) + b ] βββΊ [ f(z) ] βββΊ Output Ε·
x_D ββββββββββββββΊ [ w_D ] βββ
(Bias unit: b) βββββββββββββββ
1. LINEAR PREDICTOR (Net Input):
z = wβxβ + wβxβ + ... + w_D x_D + b = w^T x + b
2. HEAVISIDE STEP ACTIVATION FUNCTION:
Ε· = f(z) = { +1 if z β₯ 0
{ 0 if z < 0 (or -1 in bipolar notation)
3. ROSENBLATT WEIGHT UPDATE RULE (Delta Learning Rule):
For each training sample (x_i, y_i):
w_j <-- w_j + Ξ· * (y_i - Ε·_i) * x_{ij}
b <-- b + Ξ· * (y_i - Ε·_i)
* If prediction is correct (y_i - Ε·_i = 0), weights remain unchanged.
* If output is 0 but target is 1: w moves toward x_i (positive reinforcement).
* If output is 1 but target is 0: w moves away from x_i (negative reinforcement).
THE MINSKY-PAPERT XOR LINEAR SEPARABILITY FAILURE (1969):
AND GATE (Linearly Separable): XOR GATE (Linearly Non-Separable):
xβ xβ
β² β²
1 β β(0,1) β (1,1) 1 β β (0,1) β(1,1)
β β
β \ Hyperplane Separator β No single straight line
β \ β can isolate the β from β!
0 β β(0,0) \ β(1,0) 0 β β(0,0) β (1,0)
ββββββββββββ\βββββββββββΊ xβ ββββββββββββββββββββββββΊ xβ
0 1 0 1
Single Perceptron: CONVERGES Single Perceptron: CYCLES FOREVER!
import numpy as np
import pandas as pd
# ==========================================================
# 1. SCRATCH PERCEPTRON CLASSIFIER
# ==========================================================
class PerceptronScratch:
def __init__(self, learning_rate=0.1, max_epochs=50):
self.lr = learning_rate
self.max_epochs = max_epochs
self.weights = None
self.bias = 0.0
self.converged = False
self.epochs_to_converge = 0
def fit(self, X, y):
n_samples, n_features = X.shape
self.weights = np.zeros(n_features)
self.bias = 0.0
for epoch in range(1, self.max_epochs + 1):
errors = 0
for idx in range(n_samples):
x_i = X[idx]
y_i = y[idx]
# Forward pass: Linear dot product + Heaviside step threshold
linear_output = np.dot(x_i, self.weights) + self.bias
y_pred = 1 if linear_output >= 0.0 else 0
# Weight update rule: w <-- w + Ξ· * (y - Ε·) * x
update = self.lr * (y_i - y_pred)
if update != 0.0:
self.weights += update * x_i
self.bias += update
errors += 1
# If zero errors across all samples, hyperplane has achieved separation
if errors == 0:
self.converged = True
self.epochs_to_converge = epoch
break
if not self.converged:
self.epochs_to_converge = self.max_epochs
def predict(self, X):
linear_output = np.dot(X, self.weights) + self.bias
return np.where(linear_output >= 0.0, 1, 0)
# Logical Truth Tables
X_inputs = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y_and = np.array([0, 0, 0, 1]) # Linearly separable
y_xor = np.array([0, 1, 1, 0]) # Linearly non-separable
# Train on AND Gate
p_and = PerceptronScratch(learning_rate=0.2, max_epochs=50)
p_and.fit(X_inputs, y_and)
# Train on XOR Gate
p_xor = PerceptronScratch(learning_rate=0.2, max_epochs=50)
p_xor.fit(X_inputs, y_xor)
print("=" * 65)
print("ROSENBLATT'S PERCEPTRON: CONVERGENCE VS. XOR FAILURE")
print("=" * 65)
print("Task 1: Logical AND Gate (Linearly Separable):")
print(f" Converged Successfully : {p_and.converged} (Halted at Epoch {p_and.epochs_to_converge})")
print(f" Final Learned Weights : wβ={p_and.weights[0]:.2f}, wβ={p_and.weights[1]:.2f}, b={p_and.bias:.2f}")
print(f" Predictions on Truth : {p_and.predict(X_inputs)} (Actual: {y_and})")
print("-" * 65)
print("Task 2: Logical XOR Gate (Linearly Non-Separable):")
print(f" Converged Successfully : {p_xor.converged} (Oscillated for all {p_xor.epochs_to_converge} Epochs!)")
print(f" Final Stalled Weights : wβ={p_xor.weights[0]:.2f}, wβ={p_xor.weights[1]:.2f}, b={p_xor.bias:.2f}")
print(f" Predictions on Truth : {p_xor.predict(X_inputs)} (Actual: {y_xor})")
print("-" * 65)
print("Historic Implication: The inability of single-layer perceptrons to solve XOR")
print("triggered the first AI Winter until multi-layer backpropagation emerged.")
import numpy as np
import pandas as pd
# The Boolean identity: A XOR B = (A OR B) AND (A NAND B)
# We can construct a 2-layer Feedforward Network with:
# Hidden Neuron 1 (h1) acting as an OR Gate
# Hidden Neuron 2 (h2) acting as a NAND Gate
# Output Neuron (y) combining h1 and h2 via an AND Gate
X_xor = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
# Layer 1 Weights (Hidden Layer: 2 inputs -> 2 hidden neurons)
# Neuron h1 (OR Gate): w1=1.0, w2=1.0, bias=-0.5
# Neuron h2 (NAND Gate): w1=-1.0, w2=-1.0, bias=1.5
W1 = np.array([
[1.0, -1.0], # Weights from input x1 to [h1, h2]
[1.0, -1.0] # Weights from input x2 to [h1, h2]
])
b1 = np.array([-0.5, 1.5])
# Layer 2 Weights (Output Layer: 2 hidden inputs -> 1 output neuron)
# Output Neuron (AND Gate): w1=1.0, w2=1.0, bias=-1.5
W2 = np.array([[1.0], [1.0]])
b2 = np.array([-1.5])
# Threshold activation function
def step_activation(z):
return (z >= 0.0).astype(int)
# ==========================================================
# FEEDFORWARD PASS ACROSS THE NETWORK
# ==========================================================
# Step 1: Compute Hidden Layer Pre-Activations and Outputs
z_hidden = np.dot(X_xor, W1) + b1
h_output = step_activation(z_hidden)
# Step 2: Compute Output Layer Pre-Activation and Final Decision
z_output = np.dot(h_output, W2) + b2
y_pred = step_activation(z_output).flatten()
df_xor_pipeline = pd.DataFrame({
"Input_x1": X_xor[:, 0],
"Input_x2": X_xor[:, 1],
"Hidden_h1 (OR)": h_output[:, 0],
"Hidden_h2 (NAND)": h_output[:, 1],
"Output_Ε· (XOR)": y_pred,
"Target_y": [0, 1, 1, 0]
})
print("=" * 65)
print("MULTI-LAYER PERCEPTRON (MLP): EXACT ANALYTICAL XOR SOLUTION")
print("=" * 65)
print(df_xor_pipeline.to_string(index=False))
print("-" * 65)
print("Geometric Explanation of Non-Linear Representation:")
print(" In the raw input space (x1, x2), the XOR points are not linearly separable.")
print(" The hidden layer maps input (x1, x2) into latent space (h1, h2):")
print(" Input (0, 0) βββΊ Latent (0, 1)")
print(" Input (0, 1) βββΊ Latent (1, 1)")
print(" Input (1, 0) βββΊ Latent (1, 1)")
print(" Input (1, 1) βββΊ Latent (1, 0)")
print(" In this transformed space, a single linear hyperplane cleanly separates the classes!")
import numpy as np
import pandas as pd
# Credit risk classification: 400 applicants, 2 continuous features
# Features: [Income_Normalized, Debt_Ratio_Normalized]
np.random.seed(42)
N = 400
X = np.random.uniform(-2.0, 2.0, size=(N, 2))
# Non-linear circular risk boundary
y = ((X[:, 0]**2 + X[:, 1]**2) < 1.2).astype(int).reshape(-1, 1)
# ==========================================================
# 2-LAYER NEURAL NETWORK (2 Inputs -> 4 Hidden Neurons -> 1 Output)
# ==========================================================
class NeuralNetworkScratch:
def __init__(self, input_dim=2, hidden_dim=4, output_dim=1, lr=0.1):
self.lr = lr
# Xavier initialization
self.W1 = np.random.randn(input_dim, hidden_dim) * np.sqrt(2.0 / input_dim)
self.b1 = np.zeros((1, hidden_dim))
self.W2 = np.random.randn(hidden_dim, output_dim) * np.sqrt(2.0 / hidden_dim)
self.b2 = np.zeros((1, output_dim))
self.loss_history = []
def _sigmoid(self, z):
return 1.0 / (1.0 + np.exp(-np.clip(z, -500.0, 500.0)))
def _sigmoid_derivative(self, a):
# Derivative in terms of the activation output: Ο'(z) = a * (1 - a)
return a * (1.0 - a)
def forward(self, X):
# Hidden Layer forward pass
self.z1 = np.dot(X, self.W1) + self.b1
self.a1 = self._sigmoid(self.z1)
# Output Layer forward pass
self.z2 = np.dot(self.a1, self.W2) + self.b2
self.a2 = self._sigmoid(self.z2)
return self.a2
def backward(self, X, y):
m = X.shape[0]
# 1. Output Layer Gradient:
# Binary Cross-Entropy Loss: L = - (y*log(a2) + (1-y)*log(1-a2))
# dL/dz2 = a2 - y
delta2 = (self.a2 - y) # Shape: (m, 1)
# Gradients for W2 and b2:
dW2 = (1.0 / m) * np.dot(self.a1.T, delta2) # (hidden_dim, 1)
db2 = (1.0 / m) * np.sum(delta2, axis=0, keepdims=True)
# 2. Backpropagate error to Hidden Layer via Chain Rule:
# delta1 = (delta2 @ W2^T) * Ο'(z1)
delta1 = np.dot(delta2, self.W2.T) * self._sigmoid_derivative(self.a1) # (m, hidden_dim)
# Gradients for W1 and b1:
dW1 = (1.0 / m) * np.dot(X.T, delta1) # (input_dim, hidden_dim)
db1 = (1.0 / m) * np.sum(delta1, axis=0, keepdims=True)
# 3. Gradient Descent Parameter Updates:
self.W2 -= self.lr * dW2
self.b2 -= self.lr * db2
self.W1 -= self.lr * dW1
self.b1 -= self.lr * db1
def train(self, X, y, epochs=1500):
for epoch in range(1, epochs + 1):
y_pred = self.forward(X)
self.backward(X, y)
# Record Binary Cross-Entropy Loss
eps = 1e-15
loss = -np.mean(y * np.log(y_pred + eps) + (1.0 - y) * np.log(1.0 - y_pred + eps))
self.loss_history.append(loss)
nn = NeuralNetworkScratch(input_dim=2, hidden_dim=8, output_dim=1, lr=0.5)
nn.train(X, y, epochs=2000)
final_preds = (nn.forward(X) >= 0.5).astype(int)
accuracy = np.mean(final_preds == y) * 100.0
print("=" * 65)
print("BACKPROPAGATION ENGINE FROM FIRST PRINCIPLES (NUMPY)")
print("=" * 65)
print(f"Network Architecture : [2 Inputs βββΊ 8 Hidden Neurons βββΊ 1 Output]")
print(f"Dataset Size : {N} Samples (Non-linear circular boundary)")
print("-" * 65)
print(f"Initial Training Loss (Ep 1) : {nn.loss_history[0]:.4f}")
print(f"Midway Training Loss (Ep 500): {nn.loss_history[499]:.4f}")
print(f"Final Converged Loss (Ep 2000: {nn.loss_history[-1]:.4f}")
print(f"Final Classification Accuracy: {accuracy:.2f}%")
print("-" * 65)
print("Chain Rule Verification: Gradients propagated from output back to input layer,")
print("allowing an 8-neuron hidden layer to warp the non-linear circular risk boundary.")
import numpy as np
import pandas as pd
# Activation functions and their first derivatives
def sigmoid(z):
return 1.0 / (1.0 + np.exp(-np.clip(z, -500.0, 500.0)))
def sigmoid_deriv(z):
s = sigmoid(z)
return s * (1.0 - s)
def tanh_func(z):
return np.tanh(z)
def tanh_deriv(z):
return 1.0 - np.tanh(z) ** 2
def relu(z):
return np.maximum(0.0, z)
def relu_deriv(z):
return np.where(z > 0.0, 1.0, 0.0)
def leaky_relu(z, alpha=0.01):
return np.where(z > 0.0, z, alpha * z)
def leaky_relu_deriv(z, alpha=0.01):
return np.where(z > 0.0, 1.0, alpha)
# Evaluate activations and gradient flow across varied input regimes
test_inputs = np.array([-10.0, -2.0, 0.0, 2.0, 10.0])
df_activations = pd.DataFrame({
"Input_z": test_inputs,
"Sigmoid_Ο": np.round(sigmoid(test_inputs), 4),
"Sigmoid_Grad": np.round(sigmoid_deriv(test_inputs), 4),
"Tanh_Grad": np.round(tanh_deriv(test_inputs), 4),
"ReLU_Val": np.round(relu(test_inputs), 4),
"ReLU_Grad": np.round(relu_deriv(test_inputs), 4),
"LeakyReLU_Grad": np.round(leaky_relu_deriv(test_inputs), 4)
})
# ==========================================================
# DYING RELU DIAGNOSTIC SIMULATION
# When large negative gradient updates push a neuron's bias so low
# that it outputs z <= 0 for all dataset inputs, its gradient becomes 0.0 forever.
# ==========================================================
np.random.seed(42)
N_BATCH = 1000
X_inputs = np.random.normal(0, 1.0, size=(N_BATCH, 5))
# Neuron A (Healthy ReLU): positive bias
z_healthy = np.dot(X_inputs, np.random.randn(5)) + 1.2
active_pct_healthy = np.mean(z_healthy > 0.0) * 100.0
# Neuron B (Dying ReLU): knocked into negative saturation via aggressive learning rate
z_dead = np.dot(X_inputs, np.random.randn(5)) - 4.5
active_pct_dead = np.mean(z_dead > 0.0) * 100.0
print("=" * 65)
print("ACTIVATION FUNCTIONS & GRADIENT VANISHING AUDIT")
print("=" * 65)
print(df_activations.to_string(index=False))
print("-" * 65)
print("Vanishing Gradient Demonstration:")
print(f" At z = 10.0, Sigmoid Gradient collapses to {sigmoid_deriv(np.array([10.0]))[0]:.6f} (Saturates to 0).")
print(f" At z = 10.0, ReLU Gradient remains constant at {relu_deriv(np.array([10.0]))[0]:.1f} (No vanishing!).")
print("-" * 65)
print("Dying ReLU Pathology Simulation:")
print(f" Healthy Neuron Activation Rate: {active_pct_healthy:.1f}% positive outputs (Gradients propagate)")
print(f" Dead Neuron Activation Rate : {active_pct_dead:.1f}% positive outputs (0.0 gradient forever!)")
print(" Remediation: Use LeakyReLU (slope=0.01) or GELU to ensure small non-zero gradients.")
import numpy as np
import pandas as pd
# Forward propagation variance benchmark through a 10-layer deep network
np.random.seed(42)
N_SAMPLES = 1000
DIM = 512
N_LAYERS = 10
X_init = np.random.randn(N_SAMPLES, DIM) # Initial variance = 1.0
def simulate_deep_layer_propagation(X_input, init_type="random_small", activation="relu"):
current_act = X_input.copy()
layer_variances = [np.var(current_act)]
for _ in range(N_LAYERS):
fan_in = current_act.shape[1]
fan_out = DIM
# Select initialization variance
if init_type == "random_small":
# Naive small normal distribution
W = np.random.randn(fan_in, fan_out) * 0.01
elif init_type == "xavier_glorot":
# Xavier/Glorot normal: Var = 2 / (fan_in + fan_out)
std = np.sqrt(2.0 / (fan_in + fan_out))
W = np.random.randn(fan_in, fan_out) * std
elif init_type == "kaiming_he":
# He/Kaiming normal: Var = 2 / fan_in (Preserves ReLU variance)
std = np.sqrt(2.0 / fan_in)
W = np.random.randn(fan_in, fan_out) * std
# Linear forward projection
z = np.dot(current_act, W)
# Apply activation
if activation == "relu":
current_act = np.maximum(0.0, z)
elif activation == "tanh":
current_act = np.tanh(z)
layer_variances.append(np.var(current_act))
return layer_variances
# Run simulations across 10 layers
var_small_relu = simulate_deep_layer_propagation(X_init, "random_small", "relu")
var_xavier_relu = simulate_deep_layer_propagation(X_init, "xavier_glorot", "relu")
var_he_relu = simulate_deep_layer_propagation(X_init, "kaiming_he", "relu")
df_init = pd.DataFrame({
"Layer": range(N_LAYERS + 1),
"Naive_Small_Var (0.01)": [f"{v:.4e}" for v in var_small_relu],
"Xavier_Glorot_Var": [f"{v:.4f}" for v in var_xavier_relu],
"He_Kaiming_Var": [f"{v:.4f}" for v in var_he_relu]
})
print("=" * 65)
print("10-LAYER DEEP NETWORK: ACTIVATION VARIANCE PROPAGATION (ReLU)")
print("=" * 65)
print(df_init.iloc[[0, 1, 3, 5, 10]].to_string(index=False))
print("-" * 65)
print("Theoretical Breakdown:")
print(" * Naive Small (0.01): Variance completely collapses to 0.0 by Layer 3 (Vanishing signal).")
print(" * Xavier/Glorot: Variance halves at each layer (0.50 -> 0.25 -> 0.007) because")
print(" it does not account for ReLU zeroing out half of the negative distributions.")
print(" * He/Kaiming: Variance remains stable near ~1.0 across all 10 layers,")
print(" enabling deep networks to train without gradient decay.")
import torch
import torch.nn as nn
import torch.optim as optim
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
# High-dimensional noisy dataset: 1,200 samples, 40 features (only 10 informative)
np.random.seed(42)
torch.manual_seed(42)
X_np, y_np = make_classification(
n_samples=1200, n_features=40, n_informative=10, n_redundant=15, flip_y=0.08, random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(X_np, y_np, test_size=0.30, random_state=42)
X_tr = torch.tensor(X_train, dtype=torch.float32)
y_tr = torch.tensor(y_train, dtype=torch.float32).unsqueeze(1)
X_te = torch.tensor(X_test, dtype=torch.float32)
y_te = torch.tensor(y_test, dtype=torch.float32).unsqueeze(1)
# Architecture Definition
class MLPRegularized(nn.Module):
def __init__(self, use_dropout=False, dropout_rate=0.3):
super(MLPRegularized, self).__init__()
self.use_dropout = use_dropout
self.fc1 = nn.Linear(40, 128)
self.relu1 = nn.ReLU()
self.drop1 = nn.Dropout(p=dropout_rate)
self.fc2 = nn.Linear(128, 64)
self.relu2 = nn.ReLU()
self.drop2 = nn.Dropout(p=dropout_rate)
self.out = nn.Linear(64, 1)
def forward(self, x):
x = self.relu1(self.fc1(x))
if self.use_dropout:
x = self.drop1(x)
x = self.relu2(self.fc2(x))
if self.use_dropout:
x = self.drop2(x)
return self.out(x) # Returns raw logits
def train_and_evaluate_mlp(use_dropout=False, weight_decay=0.0, epochs=150):
model = MLPRegularized(use_dropout=use_dropout)
criterion = nn.BCEWithLogitsLoss()
optimizer = optim.Adam(model.parameters(), lr=0.005, weight_decay=weight_decay)
for epoch in range(epochs):
model.train()
optimizer.zero_grad()
logits = model(X_tr)
loss = criterion(logits, y_tr)
loss.backward()
optimizer.step()
# Evaluation mode disables dropout layers
model.eval()
with torch.no_grad():
train_preds = (torch.sigmoid(model(X_tr)) >= 0.5).int()
test_preds = (torch.sigmoid(model(X_te)) >= 0.5).int()
acc_train = accuracy_score(y_train, train_preds.numpy())
acc_test = accuracy_score(y_test, test_preds.numpy())
return acc_train, acc_test
# 1. Unregularized Deep MLP (Overfits training noise)
tr_unreg, te_unreg = train_and_evaluate_mlp(use_dropout=False, weight_decay=0.0)
# 2. L2 Weight Decay Regularization (L2 Penalty = 0.01)
tr_l2, te_l2 = train_and_evaluate_mlp(use_dropout=False, weight_decay=0.01)
# 3. Dropout Regularization (p = 0.3)
tr_drop, te_drop = train_and_evaluate_mlp(use_dropout=True, weight_decay=0.0)
# 4. Hybrid (Dropout + Weight Decay)
tr_hyb, te_hyb = train_and_evaluate_mlp(use_dropout=True, weight_decay=0.01)
df_reg_audit = pd.DataFrame([
{"Strategy": "Unregularized MLP", "Train_Accuracy": f"{tr_unreg*100:.2f}%", "Test_Accuracy": f"{te_unreg*100:.2f}%", "Overfit_Gap": f"{(tr_unreg - te_unreg)*100:+.2f}%"},
{"Strategy": "L2 Weight Decay (Ξ»=0.01)", "Train_Accuracy": f"{tr_l2*100:.2f}%", "Test_Accuracy": f"{te_l2*100:.2f}%", "Overfit_Gap": f"{(tr_l2 - te_l2)*100:+.2f}%"},
{"Strategy": "Dropout (p=0.30)", "Train_Accuracy": f"{tr_drop*100:.2f}%", "Test_Accuracy": f"{te_drop*100:.2f}%", "Overfit_Gap": f"{(tr_drop - te_drop)*100:+.2f}%"},
{"Strategy": "Hybrid (Dropout + L2)", "Train_Accuracy": f"{tr_hyb*100:.2f}%", "Test_Accuracy": f"{te_hyb*100:.2f}%", "Overfit_Gap": f"{(tr_hyb - te_hyb)*100:+.2f}%"}
])
print("=" * 65)
print("NEURAL NETWORK REGULARIZATION AUDIT (150 EPOCHS)")
print("=" * 65)
print(df_reg_audit.to_string(index=False))
print("-" * 65)
print("Mechanics:")
print(" * Unregularized MLP memorizes label noise (100% train accuracy, 83.6% test).")
print(" * Dropout prevents co-adaptation by dropping 30% of activations at each step.")
print(" * Hybrid regularization narrows the overfit gap to 2.74% while achieving peak test accuracy.")
Dataset and DataLoader batching, He/Kaiming initialization, AdamW optimizer with decoupled weight decay, and automatic validation early stopping.
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import roc_auc_score, brier_score_loss
# Enterprise Customer Subscription Churn: 3,000 instances, 8 continuous features
np.random.seed(42)
torch.manual_seed(42)
N = 3000
P = 8
X_raw = np.random.randn(N, P)
# Complex non-linear churn risk probability
logit = 1.5 * X_raw[:, 0] - 2.0 * (X_raw[:, 1]**2) + 0.8 * X_raw[:, 2] * X_raw[:, 3]
p_churn = 1.0 / (1.0 + np.exp(-logit))
y_raw = (np.random.rand(N) < p_churn).astype(np.float32)
# 1. STRICT ISOLATION: Partition dataset BEFORE scaling
X_train_raw, X_test_raw, y_train, y_test = train_test_split(
X_raw, y_raw, test_size=0.25, stratify=y_raw, random_state=42
)
X_tr_sub, X_val_raw, y_tr_sub, y_val = train_test_split(
X_train_raw, y_train, test_size=0.20, stratify=y_train, random_state=42
)
# 2. Fit Scaler strictly on sub-training split
scaler = StandardScaler()
X_tr_scaled = scaler.fit_transform(X_tr_sub)
X_val_scaled = scaler.transform(X_val_raw)
X_te_scaled = scaler.transform(X_test_raw)
# 3. PyTorch Custom Dataset
class TabularDataset(Dataset):
def __init__(self, X, y):
self.X = torch.tensor(X, dtype=torch.float32)
self.y = torch.tensor(y, dtype=torch.float32).unsqueeze(1)
def __len__(self):
return len(self.X)
def __getitem__(self, idx):
return self.X[idx], self.y[idx]
train_loader = DataLoader(TabularDataset(X_tr_scaled, y_tr_sub), batch_size=64, shuffle=True)
val_loader = DataLoader(TabularDataset(X_val_scaled, y_val), batch_size=128, shuffle=False)
test_loader = DataLoader(TabularDataset(X_te_scaled, y_test), batch_size=128, shuffle=False)
# 4. Neural Network Architecture with Kaiming Initialization
class ProductionANN(nn.Module):
def __init__(self, in_features):
super(ProductionANN, self).__init__()
self.net = nn.Sequential(
nn.Linear(in_features, 64),
nn.BatchNorm1d(64),
nn.ReLU(),
nn.Dropout(0.25),
nn.Linear(64, 32),
nn.BatchNorm1d(32),
nn.ReLU(),
nn.Dropout(0.20),
nn.Linear(32, 1) # Raw logit output
)
self._init_weights()
def _init_weights(self):
for m in self.modules():
if isinstance(m, nn.Linear):
nn.init.kaiming_normal_(m.weight, nonlinearity="relu")
nn.init.zeros_(m.bias)
def forward(self, x):
return self.net(x)
model = ProductionANN(in_features=P)
criterion = nn.BCEWithLogitsLoss()
optimizer = torch.optim.AdamW(model.parameters(), lr=0.003, weight_decay=0.01)
# 5. Training Loop with Early Stopping Guard
PATIENCE = 10
best_val_loss = float("inf")
patience_counter = 0
best_model_weights = None
for epoch in range(1, 101):
model.train()
for batch_x, batch_y in train_loader:
optimizer.zero_grad()
loss = criterion(model(batch_x), batch_y)
loss.backward()
optimizer.step()
# Validation Phase
model.eval()
val_loss = 0.0
with torch.no_grad():
for bx, by in val_loader:
val_loss += criterion(model(bx), by).item() * len(by)
val_loss /= len(val_loader.dataset)
# Early stopping check
if val_loss < best_val_loss:
best_val_loss = val_loss
best_model_weights = model.state_dict().copy()
patience_counter = 0
else:
patience_counter += 1
if patience_counter >= PATIENCE:
break
# Load optimal weights
model.load_state_dict(best_model_weights)
model.eval()
# 6. Final Evaluation on Held-Out Test Set
with torch.no_grad():
all_logits = []
for bx, _ in test_loader:
all_logits.append(model(bx))
test_probs = torch.sigmoid(torch.cat(all_logits)).numpy().flatten()
test_auc = roc_auc_score(y_test, test_probs)
brier_loss = brier_score_loss(y_test, test_probs)
print("=" * 65)
print("PYTORCH PRODUCTION FEEDFORWARD ANN PIPELINE")
print("=" * 65)
print(f"Dataset Size : {N} records (Train=1800, Val=450, Test=750)")
print(f"Early Stopping Best Epoch : Epoch {epoch - patience_counter} (Halted at {epoch})")
print(f"Best Validation BCE Loss : {best_val_loss:.4f}")
print("-" * 65)
print(f"Held-Out Test ROC-AUC Score : {test_auc:.4f}")
print(f"Held-Out Test Brier Loss : {brier_loss:.4f} (Calibrated probabilities)")
print("-" * 65)
print("Pipeline Integrity: Feature scaler parameters (ΞΌ, Ο) fit strictly on train split;")
print("batch normalization and dropout toggled between model.train() and model.eval().")
nn.CrossEntropyLoss.
import torch
import torch.nn as nn
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report, accuracy_score
# High-dimensional customer service ticket routing: 3 categories (K = 3)
# [0: Billing, 1: Technical Bug, 2: Account Security]
np.random.seed(42)
torch.manual_seed(42)
N = 1500
K_CLASSES = 3
X_raw, y_raw = make_classification(
n_samples=N,
n_features=12,
n_informative=8,
n_classes=K_CLASSES,
n_clusters_per_class=1,
random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(X_raw, y_raw, test_size=0.25, random_state=42, stratify=y_raw)
scaler = StandardScaler()
X_tr_scaled = torch.tensor(scaler.fit_transform(X_train), dtype=torch.float32)
y_tr_tensor = torch.tensor(y_train, dtype=torch.long)
X_te_scaled = torch.tensor(scaler.transform(X_test), dtype=torch.float32)
y_te_tensor = torch.tensor(y_test, dtype=torch.long)
# Multi-Class MLP Architecture
class MultiClassMLP(nn.Module):
def __init__(self, in_features, num_classes):
super(MultiClassMLP, self).__init__()
self.network = nn.Sequential(
nn.Linear(in_features, 64),
nn.ReLU(),
nn.Dropout(0.20),
nn.Linear(64, 32),
nn.ReLU(),
nn.Linear(32, num_classes) # K raw logits output (No explicit Softmax layer!)
)
def forward(self, x):
return self.network(x)
model = MultiClassMLP(in_features=12, num_classes=K_CLASSES)
# PyTorch's nn.CrossEntropyLoss combines LogSoftmax and NLLLoss in one numerically stable function:
# Loss = - ln [ exp(z_true) / sum_j exp(z_j) ] = - z_true + ln( sum_j exp(z_j) )
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
# Training Loop
for epoch in range(1, 101):
model.train()
optimizer.zero_grad()
logits = model(X_tr_scaled)
loss = criterion(logits, y_tr_tensor)
loss.backward()
optimizer.step()
# Inference Phase
model.eval()
with torch.no_grad():
raw_test_logits = model(X_te_scaled)
# Apply explicit Softmax to extract normalized class probabilities P(y = k | x)
predicted_probabilities = torch.softmax(raw_test_logits, dim=1).numpy()
predicted_classes = np.argmax(predicted_probabilities, axis=1)
test_accuracy = accuracy_score(y_test, predicted_classes)
print("=" * 65)
print("PYTORCH MULTI-CLASS SOFTMAX ANN CLASSIFICATION")
print("=" * 65)
print(f"Total Service Tickets Evaluated : {N} across K={K_CLASSES} categories")
print(f"Final Multi-Class Test Accuracy : {test_accuracy * 100:.2f}%")
print("-" * 65)
print("Sample Inspection (First 3 Classified Tickets):")
for i in range(3):
probs = predicted_probabilities[i]
pred_cls = predicted_classes[i]
true_cls = y_test[i]
print(f" Ticket {i+1}: True={true_cls} | Predicted={pred_cls} | Probs: [C0={probs[0]:.3f}, C1={probs[1]:.3f}, C2={probs[2]:.3f}] (Sum={probs.sum():.1f})")
print("-" * 65)
print("Classification Performance Breakdown:")
print(classification_report(y_test, predicted_classes, target_names=["Billing", "Technical", "Security"]))
NEURAL NETWORK ARCHITECTURAL COMPONENTS MATRIX:
ββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββ
β Component / Paradigm β Mathematical Formulation β Key Hyperparameters β Primary Production Role β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β Single Perceptron (Rosenblattβ Ε· = step( w^T x + b ) β learning_rate (Ξ·) β Linearly separable binary tasks; β
β β Error update: Ξw = Ξ·(y-Ε·)x β β historical foundation of ANNs β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β Multi-Layer Perceptron (MLP) β f(x) = Wβ Ο(Wβ x + bβ) + bβ β hidden_layer_sizes, β Universal function approximation; β
β β Feedforward non-linear stack β activation, depth β complex tabular & embedding data β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β Backpropagation β Ξ΄_l = (Ξ΄_{l+1} W_{l+1}^T) β learning_rate, β Reverse-mode automatic gradient β
β β β Ο'(z_l) β batch_size β differentiation via chain rule β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β ReLU Activation β f(z) = max(0, z) β None β Default hidden layer activation; β
β β Gradient = 1 if z > 0 β β prevents vanishing gradients β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β LeakyReLU / GELU β f(z) = max(Ξ±z, z) β negative_slope (Ξ±) β Eliminates the "Dying ReLU" trap; β
β β Continuous leaky gradient β β smoother non-linear optimization β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β He / Kaiming Initialization β W ~ N(0, 2 / fan_in) β nonlinearity='relu' β Preserves activation variance β
β β Gaussian / Uniform β β across deep ReLU architectures β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β Glorot / Xavier Init β W ~ N(0, 2/(fan_in+fan_out)) β nonlinearity='tanh' β Preserves variance for symmetric β
β β Balanced variance scaling β β activations (Sigmoid, Tanh) β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β Dropout Regularization β r ~ Bernoulli(1 - p) β dropout_rate (p) β Prevents co-adaptation of hidden β
β β x_drop = (r β x) / (1 - p) β β feature detectors during trainingβ
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β Softmax Output Layer β P(y=k) = e^(z_k) / β e^(z_j) β temperature (T) β Multi-class mutually exclusive β
β β Normalized probabilities β β probability distribution β
ββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ€
β PRODUCTION BEST PRACTICES: β
β 1. ALWAYS scale input features (StandardScaler); unscaled inputs distort loss surface gradients into narrow ravines. β
β 2. Pair ReLU activations with He/Kaiming initialization to prevent activation variance decay across deep layers. β
β 3. For multi-class classification in PyTorch, use `nn.CrossEntropyLoss()` directly on raw logits without explicit softmax β
β 4. Always toggle `model.train()` and `model.eval()` to control Dropout and Batch Normalization behaviors correctly. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ