ποΈ Spatial Inductive Biases: Exploiting Locality and Translation Equivariance
Feeding high-resolution image matrices directly into standard Multi-Layer Perceptrons (MLPs) causes two fundamental failures: parameter explosion (flattening a modest $224 \times 224 \times 3$ RGB image into a 1024-neuron hidden layer requires over 154 million parameters in the first weight matrix alone) and a complete loss of 2D spatial grid topology. Convolutional Neural Networks (CNNs) resolve this by encoding two biological vision priors: Local Receptive Fields (sparsity of connections) and Stationary Spatial Statistics (weight sharing). Instead of full matrix multiplication across all pixels, learnable parameter kernels slide over spatial neighborhoods using discrete cross-correlation. Shifting a pattern across input coordinates simply shifts its feature map representationβa property termed Translation Equivariance. Downsampling layers (pooling or strided convolutions) aggregate localized features into increasingly broad Effective Receptive Fields (ERF), constructing an abstraction hierarchy that progresses from low-level oriented edges and color gradients up to complex semantic object representations.
THE 2D DISCRETE CROSS-CORRELATION OPERATION:
Input Feature Tensor X (C_in x H_in x W_in) convolved with Kernel Tensor K (C_out x C_in x K_h x K_w):
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Output Activation Map S at spatial coordinate (i, j) for channel k: β
β S(k, i, j) = (X * K)(k, i, j) + b_k β
β = β_{c=1}^{C_in} β_{m=0}^{K_h - 1} β_{n=0}^{K_w - 1} X(c, i*s + m, j*s + n) * K(k, c, m, n) + b_k β
β where s is the stride and b_k is the learnable scalar bias for filter k. β
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
OUTPUT SPATIAL DIMENSION EQUATIONS:
Given: Input size (H_in, W_in), Kernel size (K_h, K_w), Padding (P_h, P_w),
Stride (S_h, S_w), and Dilation (D_h, D_w):
H_out = floor[ ( H_in + 2*P_h - D_h*(K_h - 1) - 1 ) / S_h ] + 1
W_out = floor[ ( W_in + 2*P_w - D_w*(K_w - 1) - 1 ) / S_w ] + 1
COMMON PADDING SCHEMES (Assuming S = 1, D = 1, Square K):
1. VALID PADDING (P = 0):
* No borders added. Map shrinks: H_out = H_in - K + 1
2. SAME PADDING (P = (K - 1) / 2):
* Ensures output spatial resolution equals input resolution: H_out = H_in
3. FULL PADDING (P = K - 1):
* Slides kernel across every boundary pixel: H_out = H_in + K - 1
EFFECTIVE RECEPTIVE FIELD (ERF) PROPAGATION:
Layer l receptive field grows linearly with previous receptive field and stride:
RF_l = RF_{l-1} + (K_l - 1) * β_{i=1}^{l-1} S_i
* Two stacked 3x3 convolutions with stride 1 achieve the receptive field of
one 5x5 convolution: RF = 1 + (3 - 1) + (3 - 1) = 5
* Parameters: 2 * (3Β² * CΒ²) = 18*CΒ² VS. 1 * (5Β² * CΒ²) = 25*CΒ² [28% Parameter Savings!]
torch.nn.functional.conv2d.
import numpy as np
import pandas as pd
import torch
import torch.nn.functional as F
# Sample image feature map: Batch=1, Channels=1, Height=5, Width=5
np.random.seed(42)
X_input = np.array([[
[1.0, 2.0, 0.0, 1.0, 1.0],
[0.0, 1.0, 2.0, 0.0, 1.0],
[2.0, 3.0, 1.0, 2.0, 0.0],
[0.0, 1.0, 2.0, 1.0, 0.0],
[1.0, 0.0, 1.0, 2.0, 1.0]
]], dtype=np.float32).reshape(1, 1, 5, 5)
# Kernel filter: 1 Output Channel, 1 Input Channel, Kernel 3x3 (Edge detector)
K_filter = np.array([[
[ 1.0, 0.0, -1.0],
[ 1.0, 0.0, -1.0],
[ 1.0, 0.0, -1.0]
]], dtype=np.float32).reshape(1, 1, 3, 3)
bias = np.array([0.5], dtype=np.float32)
STRIDE = 1
PADDING = 1
# ==========================================================
# 1. SCRATCH IM2COL 2D CONVOLUTION (GEMM TRANSFORMATION)
# ==========================================================
def im2col_indices(x, kh, kw, padding=1, stride=1):
# Zero-pad the input
p = padding
x_padded = np.pad(x, ((0, 0), (0, 0), (p, p), (p, p)), mode="constant")
b, c, h_pad, w_pad = x_padded.shape
out_h = int((h_pad - kh) / stride + 1)
out_w = int((w_pad - kw) / stride + 1)
# Extract sliding window patches
cols = []
for i in range(0, out_h * stride, stride):
for j in range(0, out_w * stride, stride):
patch = x_padded[:, :, i:i+kh, j:j+kw]
cols.append(patch.reshape(b, -1))
# Matrix shape: (Kernel_Elements, Total_Spatial_Patches)
return np.array(cols).transpose(1, 2, 0), out_h, out_w
def conv2d_scratch(x, k, b, stride=1, padding=1):
batch_size, c_in, h_in, w_in = x.shape
c_out, _, kh, kw = k.shape
# Reshape kernel into 2D weight matrix: (C_out, C_in * Kh * Kw)
W_row = k.reshape(c_out, -1)
# Unroll image patches into column matrix: (Batch, C_in * Kh * Kw, H_out * W_out)
X_col, out_h, out_w = im2col_indices(x, kh, kw, padding, stride)
# Batch General Matrix Multiply (GEMM): W_row @ X_col + bias
out = np.zeros((batch_size, c_out, out_h * out_w), dtype=np.float32)
for bi in range(batch_size):
out[bi] = W_row @ X_col[bi] + b[:, None]
return out.reshape(batch_size, c_out, out_h, out_w)
conv_scratch = conv2d_scratch(X_input, K_filter, bias, stride=STRIDE, padding=PADDING)
# ==========================================================
# 2. PYTORCH F.CONV2D COMPARISON
# ==========================================================
t_X = torch.tensor(X_input)
t_K = torch.tensor(K_filter)
t_b = torch.tensor(bias)
conv_torch = F.conv2d(t_X, t_K, bias=t_b, stride=STRIDE, padding=PADDING).numpy()
print("=" * 65)
print("2D CONVOLUTION: SCRATCH IM2COL (GEMM) VS. PYTORCH F.CONV2D")
print("=" * 65)
print(f"Input Feature Shape : {X_input.shape} (1x1x5x5)")
print(f"Kernel Filter Dimensions : {K_filter.shape} (3x3 Vertical Sobel Edge)")
print(f"Padding: {PADDING} | Stride: {STRIDE} βββΊ Output Shape: {conv_scratch.shape}")
print("-" * 65)
print("Scratch Output Feature Map (2D Grid):")
print(np.round(conv_scratch[0, 0], 2))
print("-" * 65)
print(f"Exact Numerical Parity Match : {np.allclose(conv_scratch, conv_torch, atol=1e-5)}")
print("Engine Note: im2col maps sliding receptive windows into contiguous matrix rows,")
print("enabling BLAS / cuBLAS hardware accelerators to perform convolution as pure GEMM.")
import torch
import torch.nn as nn
import pandas as pd
# Feature map representing activations from a defect inspection camera
# Shape: (Batch=1, Channels=1, Height=4, Width=4)
feature_map = torch.tensor([[
[0.2, 0.8, 0.1, 0.0],
[0.1, 9.5, 0.4, 0.2], # Strong feature spike (e.g. scratch anomaly = 9.5)
[0.3, 0.2, 1.1, 0.9],
[0.0, 0.1, 0.8, 1.4]
]], dtype=torch.float32).unsqueeze(0)
# 1. 2D Max Pooling (Kernel=2, Stride=2): Captures sharp peak activations
max_pool = nn.MaxPool2d(kernel_size=2, stride=2)
out_max = max_pool(feature_map)
# 2. 2D Average Pooling (Kernel=2, Stride=2): Averages neighborhood activations
avg_pool = nn.AvgPool2d(kernel_size=2, stride=2)
out_avg = avg_pool(feature_map)
# 3. Global Average Pooling (GAP - Lin et al., Network in Network): Collapses H x W to 1 x 1
gap_pool = nn.AdaptiveAvgPool2d((1, 1))
out_gap = gap_pool(feature_map)
print("=" * 65)
print("POOLING SUBSAMPLING & INVARIANCE AUDIT")
print("=" * 65)
print("Input Activation Map (4x4 Matrix with Isolated Spike 9.5):")
print(feature_map[0, 0].numpy())
print("-" * 65)
print(f"2x2 Max Pooling Output Shape : {tuple(out_max.shape)}")
print(out_max[0, 0].numpy())
print(f" --> Top-Left Cell Value : {out_max[0, 0, 0, 0].item():.2f} (Preserves isolated spike intact!)")
print("-" * 65)
print(f"2x2 Avg Pooling Output Shape : {tuple(out_avg.shape)}")
print(out_avg[0, 0].numpy())
print(f" --> Top-Left Cell Value : {out_avg[0, 0, 0, 0].item():.2f} (Spike diluted by 72% via averaging)")
print("-" * 65)
print(f"Global Average Pooling (GAP) : {out_gap.squeeze().item():.4f} (Single vector scalar)")
print("Architectural Trend: Modern architectures (ResNet, EfficientNet) use GAP")
print("before the final classifier to eliminate millions of FC layer parameters.")
import pandas as pd
# Layer specifications: (Layer Name, Kernel Size, Stride, Padding)
network_spec = [
("Conv2d_L1 (Input Layer)", 3, 1, 1),
("Conv2d_L2 (Stride 2)", 3, 2, 1),
("Conv2d_L3 (Stride 1)", 3, 1, 1),
("MaxPool2d (Stride 2)", 2, 2, 0),
("Conv2d_L4 (Stride 1)", 3, 1, 1)
]
def calculate_receptive_fields(layers):
rf_current = 1 # Initial receptive field of input pixel
cumulative_stride = 1 # Cumulative stride jump
audit_records = []
for name, k, s, p in layers:
# Receptive Field Recurrence Relation:
# RF_new = RF_prev + (k - 1) * Cumulative_Stride_prev
rf_new = rf_current + (k - 1) * cumulative_stride
audit_records.append({
"Layer_Stage": name,
"Kernel (k)": k,
"Stride (s)": s,
"Padding (p)": p,
"Jump_Stride (J_l)": cumulative_stride,
"Receptive_Field (Pixels)": rf_new
})
# Update trackers for subsequent layer
rf_current = rf_new
cumulative_stride *= s
return pd.DataFrame(audit_records)
df_rf = calculate_receptive_fields(network_spec)
print("=" * 65)
print("CNN RECEPTIVE FIELD EXPANSION ACROSS NETWORK DEPTH")
print("=" * 65)
print(df_rf.to_string(index=False))
print("-" * 65)
print("Key Architectural Findings:")
print(f" * Final Layer Neuron observes a {df_rf.iloc[-1]['Receptive_Field (Pixels)']}x{df_rf.iloc[-1]['Receptive_Field (Pixels)']} input pixel patch.")
print(" * Strided convolutions and pooling layers accelerate RF growth by multiplying")
print(" the jump step (J_l), allowing deep networks to capture global context.")
import torch
import torch.nn as nn
import pandas as pd
# ResNet BasicBlock Implementation
class ResNetBlock(nn.Module):
def __init__(self, channels):
super(ResNetBlock, self).__init__()
self.conv1 = nn.Conv2d(channels, channels, kernel_size=3, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(channels)
self.relu = nn.ReLU(inplace=True)
self.conv2 = nn.Conv2d(channels, channels, kernel_size=3, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(channels)
def forward(self, x):
# Identity shortcut connection: x is preserved intact!
identity = x
# Residual branch: F(x)
out = self.relu(self.bn1(self.conv1(x)))
out = self.bn2(self.conv2(out))
# Skip addition: H(x) = F(x) + x
out += identity
return self.relu(out)
# Plain Conv Block without Skip Connection
class PlainBlock(nn.Module):
def __init__(self, channels):
super(PlainBlock, self).__init__()
self.conv1 = nn.Conv2d(channels, channels, kernel_size=3, padding=1, bias=False)
self.bn1 = nn.BatchNorm2d(channels)
self.relu = nn.ReLU(inplace=True)
self.conv2 = nn.Conv2d(channels, channels, kernel_size=3, padding=1, bias=False)
self.bn2 = nn.BatchNorm2d(channels)
def forward(self, x):
out = self.relu(self.bn1(self.conv1(x)))
out = self.relu(self.bn2(self.conv2(out)))
return out
# Build deep 12-block networks (24 convolutional layers)
torch.manual_seed(42)
CHANNELS = 32
X_test_input = torch.randn(4, CHANNELS, 16, 16, requires_grad=True)
# 1. Plain 24-Layer Network
plain_layers = [PlainBlock(CHANNELS) for _ in range(12)]
net_plain = nn.Sequential(*plain_layers)
# 2. ResNet 24-Layer Network
resnet_layers = [ResNetBlock(CHANNELS) for _ in range(12)]
net_resnet = nn.Sequential(*resnet_layers)
# Measure gradient magnitude reaching input tensor
# Forward and Backward on Plain
out_plain = net_plain(X_test_input).sum()
out_plain.backward()
grad_norm_plain = X_test_input.grad.norm().item()
# Reset input gradients
X_test_input.grad.zero_()
# Forward and Backward on ResNet
out_resnet = net_resnet(X_test_input).sum()
out_resnet.backward()
grad_norm_resnet = X_test_input.grad.norm().item()
print("=" * 65)
print("DEEP 24-LAYER NETWORK: PLAIN VS. RESIDUAL GRADIENT FLOW")
print("=" * 65)
print(f"Input Tensor Shape : {tuple(X_test_input.shape)}")
print(f"Network Depth : 12 Consecutive Blocks (24 Conv Layers)")
print("-" * 65)
print(f"Plain Network Input Gradient : {grad_norm_plain:.6f} (Suffers severe degradation)")
print(f"ResNet Input Gradient : {grad_norm_resnet:.6f} (Strong gradient flow)")
print(f"Relative Gradient Strength : ResNet preserves {grad_norm_resnet / grad_norm_plain:.1f}x stronger signal!")
print("-" * 65)
print("Mathematical Reason: In ResNet, dL/dx = dL/dOut * (dF/dx + 1).")
print("The '+ 1' term acts as a gradient superhighway, allowing updates to bypass")
print("weight multiplications without decaying.")
import torch
import torch.nn as nn
import pandas as pd
# Depthwise Separable Convolution Block
class DepthwiseSeparableConv(nn.Module):
def __init__(self, in_channels, out_channels, kernel_size=3, padding=1):
super(DepthwiseSeparableConv, self).__init__()
# 1. Depthwise Convolution: groups=in_channels enforces 1 filter per channel
self.depthwise = nn.Conv2d(
in_channels, in_channels, kernel_size=kernel_size,
padding=padding, groups=in_channels, bias=False
)
# 2. Pointwise Convolution: 1x1 kernel mixes cross-channel projections
self.pointwise = nn.Conv2d(
in_channels, out_channels, kernel_size=1, bias=False
)
def forward(self, x):
return self.pointwise(self.depthwise(x))
IN_CHANNELS = 64
OUT_CHANNELS = 128
KERNEL = 3
H, W = 32, 32
# 1. Standard Convolution Layer
conv_standard = nn.Conv2d(IN_CHANNELS, OUT_CHANNELS, kernel_size=KERNEL, padding=1, bias=False)
# 2. Depthwise Separable Convolution Block
conv_separable = DepthwiseSeparableConv(IN_CHANNELS, OUT_CHANNELS, kernel_size=KERNEL, padding=1)
# Parameter Count Calculation:
# Standard : K * K * C_in * C_out = 3 * 3 * 64 * 128 = 73,728
# Depthwise: K * K * C_in = 3 * 3 * 64 = 576
# Pointwise: 1 * 1 * C_in * C_out = 1 * 1 * 64 * 128 = 8,192
# Separable Total: 576 + 8,192 = 8,768 (8.4x fewer parameters!)
params_std = sum(p.numel() for p in conv_standard.parameters())
params_sep = sum(p.numel() for p in conv_separable.parameters())
# Theoretical FLOPs:
flops_std = params_std * H * W
flops_sep = params_sep * H * W
df_efficiency = pd.DataFrame([
{
"Layer_Design": "Standard Conv2d (3x3)",
"Parameters": f"{params_std:,}",
"Computational_FLOPs": f"{flops_std:,}",
"Efficiency_Ratio": "1.0x (Baseline)"
},
{
"Layer_Design": "Depthwise Separable (MobileNet)",
"Parameters": f"{params_sep:,}",
"Computational_FLOPs": f"{flops_sep:,}",
"Efficiency_Ratio": f"{params_std / params_sep:.1f}x More Efficient!"
}
])
print("=" * 65)
print("EFFICIENCY BENCHMARK: STANDARD CONV VS. DEPTHWISE SEPARABLE")
print("=" * 65)
print(df_efficiency.to_string(index=False))
print("-" * 65)
print("Theoretical Ratio Formula:")
print(f" Cost_Sep / Cost_Std = 1/C_out + 1/KΒ² = 1/{OUT_CHANNELS} + 1/{KERNEL}Β² = {1/OUT_CHANNELS + 1/(KERNEL**2):.4f}")
print(f" Matches empirical computation savings: {flops_sep / flops_std:.4f} of original cost.")
import torch
import torch.nn as nn
import numpy as np
import pandas as pd
# Toy CNN Architecture for Industrial Defect Classification
class DefectInspectionCNN(nn.Module):
def __init__(self):
super(DefectInspectionCNN, self).__init__()
self.features = nn.Sequential(
nn.Conv2d(1, 16, kernel_size=3, padding=1),
nn.ReLU(),
nn.Conv2d(16, 32, kernel_size=3, padding=1), # Target layer for Grad-CAM
nn.ReLU()
)
self.classifier = nn.Sequential(
nn.AdaptiveAvgPool2d((1, 1)),
nn.Flatten(),
nn.Linear(32, 2) # [0: Clean, 1: Surface_Crack]
)
self.gradients = None
self.activations = None
def activations_hook(self, grad):
self.gradients = grad
def forward(self, x):
h = self.features[0](x)
h = self.features[1](h)
h = self.features[2](h)
# Register backward hook on activations of the final convolutional layer
if x.requires_grad:
h.register_hook(self.activations_hook)
self.activations = h
h = self.features[3](h)
return self.classifier(h)
# Initialize network and synthetic surface defect image (28x28 grayscale)
torch.manual_seed(42)
model = DefectInspectionCNN()
input_image = torch.randn(1, 1, 28, 28, requires_grad=True)
# 1. Forward Pass: Extract raw class logits
logits = model(input_image)
target_class = 1 # We want to explain class 1 ("Surface_Crack")
target_score = logits[0, target_class]
# 2. Backward Pass: Compute gradients of target class score w.r.t. activation maps
model.zero_grad()
target_score.backward()
# 3. Grad-CAM Calculation:
# Extract captured gradients and forward activations: Shape (1, 32, 28, 28)
grads = model.gradients[0]
acts = model.activations[0]
# Global average pool the gradients over spatial dimensions to get channel weights: Ξ±_k
alpha_k = torch.mean(grads, dim=(1, 2)) # Shape: (32,)
# Weighted sum of activation maps: L_cam = sum(Ξ±_k * A^k)
cam = torch.zeros(acts.shape[1:], dtype=torch.float32)
for k, w in enumerate(alpha_k):
cam += w * acts[k]
# Apply ReLU to retain features that contribute positively to target class
cam = torch.relu(cam)
# Normalize heatmap between [0, 1] for visualization
cam_norm = (cam - cam.min()) / (cam.max() - cam.min() + 1e-8)
print("=" * 65)
print("EXPLAINABLE AI: GRAD-CAM SALIENCY EXTRACTION")
print("=" * 65)
print(f"Network Output Logits : Clean={logits[0,0].item():.4f} | Crack={logits[0,1].item():.4f}")
print(f"Target Explanation Class : Class {target_class} (Surface_Crack)")
print(f"Captured Gradients Shape : {tuple(grads.shape)} (32 Channels x 28x28 Spatial)")
print("-" * 65)
print("Channel Importance Weights (Top 3 Ξ±_k Drivers):")
top_channels = torch.topk(alpha_k, 3)
for rank, (val, idx) in enumerate(zip(top_channels.values, top_channels.indices), start=1):
print(f" Rank {rank}: Feature Channel {idx.item():2d} βββΊ Importance Weight Ξ± = {val.item():+.6f}")
print("-" * 65)
print(f"Generated Heatmap Resolution : {tuple(cam_norm.shape)} (Aligned with input grid)")
print(f"Heatmap Intensity Range : Min = {cam_norm.min().item():.2f} | Max = {cam_norm.max().item():.2f}")
print("System Insight: Grad-CAM reveals which spatial pixel coordinates triggered the")
print("surface crack classification without requiring model architecture modifications.")
Dataset pipeline.
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
# Industrial vision defect dataset: 1,000 synthetic images (1 Channel, 28x28)
np.random.seed(42)
torch.manual_seed(42)
N = 1000
# Raw pixel intensity arrays spanning [0 to 255]
images_raw = np.random.uniform(0.0, 255.0, size=(N, 1, 28, 28)).astype(np.float32)
# Binary defect label
labels_raw = (images_raw.mean(axis=(1, 2, 3)) > 130.0).astype(np.int64)
# 1. STRICT PARTITION: Train/Test split BEFORE any normalization occurs
X_train_raw, X_test_raw, y_train, y_test = train_test_split(
images_raw, labels_raw, test_size=0.25, stratify=labels_raw, random_state=42
)
# 2. Compute channel statistics STRICTLY on training split
mean_tr = np.mean(X_train_raw)
std_tr = np.std(X_train_raw)
# 3. Custom PyTorch Dataset with Isolated Transform Application
class NormalizedVisionDataset(Dataset):
def __init__(self, images, labels, mean, std):
# Apply standard z-score normalization: (x - ΞΌ_train) / Ο_train
norm_images = (images - mean) / (std + 1e-8)
self.X = torch.tensor(norm_images, dtype=torch.float32)
self.y = torch.tensor(labels, dtype=torch.long)
def __len__(self):
return len(self.X)
def __getitem__(self, idx):
return self.X[idx], self.y[idx]
train_dataset = NormalizedVisionDataset(X_train_raw, y_train, mean_tr, std_tr)
test_dataset = NormalizedVisionDataset(X_test_raw, y_test, mean_tr, std_tr)
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=64, shuffle=False)
# 4. Standard Vision CNN Architecture
class IndustrialVisionCNN(nn.Module):
def __init__(self):
super(IndustrialVisionCNN, self).__init__()
self.conv = nn.Sequential(
nn.Conv2d(1, 16, kernel_size=3, padding=1),
nn.BatchNorm2d(16),
nn.ReLU(),
nn.MaxPool2d(2, 2),
nn.Conv2d(16, 32, kernel_size=3, padding=1),
nn.BatchNorm2d(32),
nn.ReLU(),
nn.AdaptiveAvgPool2d((1, 1))
)
self.fc = nn.Linear(32, 2)
def forward(self, x):
return self.fc(self.conv(x).flatten(1))
model = IndustrialVisionCNN()
criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.005)
# Fast 5-epoch training loop
model.train()
for epoch in range(5):
for bx, by in train_loader:
optimizer.zero_grad()
loss = criterion(model(bx), by)
loss.backward()
optimizer.step()
# Evaluation on Unseen Test Split
model.eval()
all_preds = []
with torch.no_grad():
for bx, _ in test_loader:
preds = torch.argmax(model(bx), dim=-1)
all_preds.extend(preds.numpy())
test_acc = accuracy_score(y_test, all_preds)
print("=" * 65)
print("LEAKAGE-FREE PYTORCH VISION PIPELINE AUDIT")
print("=" * 65)
print(f"Training Images Partition : {len(X_train_raw)} images")
print(f"Testing Images Partition : {len(X_test_raw)} images")
print("-" * 65)
print(f"Training Set Mean (ΞΌ) : {mean_tr:.4f}")
print(f"Training Set Std (Ο) : {std_tr:.4f}")
print(f"Held-Out Test Set Accuracy : {test_acc * 100:.2f}%")
print("-" * 65)
print("Integrity Check: Test set was normalized using training parameters (ΞΌ_tr, Ο_tr),")
print("preventing distribution leakage from test evaluation into model training.")
param.requires_grad = False) with Differential Learning Rate Fine-Tuning.
import torch
import torch.nn as nn
from torchvision.models import resnet18, ResNet18_Weights
import pandas as pd
# Target Domain: Medical X-Ray Pulmonary Pathology Classification (3 Classes)
NUM_CLASSES = 3
# ==========================================================
# STRATEGY 1: FIXED FEATURE EXTRACTOR (FROZEN BACKBONE)
# Freeze all feature extraction weights; train only the final linear head
# ==========================================================
model_frozen = resnet18(weights=ResNet18_Weights.DEFAULT)
# Freeze all parameter tensors in backbone
for param in model_frozen.parameters():
param.requires_grad = False
# Replace final fully connected classification head (requires_grad is True by default)
num_ftrs = model_frozen.fc.in_features
model_frozen.fc = nn.Linear(num_ftrs, NUM_CLASSES)
# Parameter audit
total_params_frozen = sum(p.numel() for p in model_frozen.parameters())
trainable_params_frozen = sum(p.numel() for p in model_frozen.parameters() if p.requires_grad)
# ==========================================================
# STRATEGY 2: DIFFERENTIAL LEARNING RATE FINE-TUNING
# Backbone trained with small learning rate; head trained with larger learning rate
# ==========================================================
model_finetune = resnet18(weights=ResNet18_Weights.DEFAULT)
model_finetune.fc = nn.Linear(num_ftrs, NUM_CLASSES)
# Parameter groups with differential learning rates
optimizer_differential = torch.optim.AdamW([
{"params": [p for n, p in model_finetune.named_parameters() if "fc" not in n], "lr": 1e-5},
{"params": model_finetune.fc.parameters(), "lr": 1e-3}
], weight_decay=0.01)
trainable_params_finetune = sum(p.numel() for p in model_finetune.parameters() if p.requires_grad)
df_transfer = pd.DataFrame([
{
"Strategy": "Fixed Feature Extractor",
"Total_Parameters": f"{total_params_frozen:,}",
"Trainable_Parameters": f"{trainable_params_frozen:,}",
"Frozen_Parameters": f"{total_params_frozen - trainable_params_frozen:,}",
"Training_Role": "Small datasets (<1,000 images); low risk of overfitting"
},
{
"Strategy": "Differential Fine-Tuning",
"Total_Parameters": f"{total_params_frozen:,}",
"Trainable_Parameters": f"{trainable_params_finetune:,}",
"Frozen_Parameters": "0",
"Training_Role": "Domain shift; updates features while preserving pre-trained weights"
}
])
print("=" * 65)
print("TRANSFER LEARNING & FINE-TUNING ARCHITECTURAL AUDIT")
print("=" * 65)
print(df_transfer.to_string(index=False))
print("-" * 65)
print("Engineering Workflow:")
print(" * Fixed Feature Extractor updates only 1,539 head parameters (0.01% of model),")
print(" enabling training on modest compute in minutes.")
print(" * Differential Fine-Tuning trains backbone at lr=1e-5 to avoid catastrophic forgetting")
print(" while training the new classification head at lr=1e-3.")
MILESTONE CONVOLUTIONAL ARCHITECTURES REFERENCE MATRIX:
ββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββ
β Architecture / Model β Core Structural Innovation β Parameter Footprint β Primary Production Role β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β LeNet-5 (LeCun et al. 1998) β Conv-Pool-Conv-Pool stack β ~60,000 params β Historical OCR baseline, β
β β Average pooling, tanh/sigmoidβ β digit recognition (MNIST) β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β AlexNet (Krizhevsky et al.) β ReLU activations, Dropout, β ~60 Million params β Breakout ImageNet 2012 model; β
β β GPU training, Max pooling β Heavy FC layers β historical baseline β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β VGGNet (Simonyan & Zisserman)β Stacked homogeneous 3x3 convsβ ~138 Million params β Clean uniform design; popular β
β β Factorized large filters β (VGG-16) β feature extractor backbone β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β Inception / GoogLeNet β Multi-scale parallel kernels β ~7 Million params β High accuracy per parameter; β
β (Szegedy et al.) β 1x1, 3x3, 5x5 bottleneck β Efficient compute β complex multi-branch graph β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β ResNet (He et al. 2015) β Residual identity skip β ~25 Million params β Industry workhorse for vision; β
β β connections (F(x) + x) β (ResNet-50) β solves gradient degradation β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β MobileNet (Howard et al.) β Depthwise Separable Convs β ~4.2 Million params β Mobile edge devices, embedded β
β β (Depthwise + Pointwise 1x1) β High FLOP efficiency β vision, low-latency APIs β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β ConvNeXt (Liu et al. 2022) β Modernized pure CNN matching β ~28 - 198 Million β High-performance modern vision β
β β Vision Transformers (7x7,GELUβ params β competitive with ViT β
ββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββ€
β CNN PRODUCTION BEST PRACTICES: β
β 1. Replace large fully connected layers with Global Average Pooling (`nn.AdaptiveAvgPool2d((1, 1))`) to save memory. β
β 2. Normalize input images using training set channel statistics `(x - ΞΌ_train) / Ο_train` to prevent data leakage. β
β 3. For mobile, IoT, or real-time streaming edge deployments, choose MobileNet or EfficientNet over heavy plain CNNs. β
β 4. Leverage Grad-CAM to visualize activation saliency and verify that model decisions rely on true object features. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ