⚡ Systematic ML Troubleshooting & Error Tracing
Machine learning code often fails silently or throws obscure stack traces spanning deep library wrappers. Mastering defensive programming, tensor assertion checks, and interactive Jupyter traceback debugging saves hours of training time.
1. Debugging NaN Loss Explosions in Neural Networks
Diagnose and resolve exploding/vanishing gradients and division-by-zero errors in PyTorch/TensorFlow training loops.
Python (Jupyter Debugging Cell)
import torch
import torch.nn as nn
# Debugging anomaly detection context manager in PyTorch
torch.autograd.set_detect_anomaly(True)
model = nn.Sequential(nn.Linear(10, 5), nn.ReLU(), nn.Linear(5, 1))
optimizer = torch.optim.Adam(model.parameters(), lr=1e-2)
criterion = nn.MSELoss()
inputs = torch.randn(32, 10)
targets = torch.randn(32, 1)
# Forward pass with gradient clipping and log clamping
optimizer.zero_grad()
outputs = model(inputs)
# Fix: Prevent log(0) or division by zero
loss = criterion(outputs, targets)
if torch.isnan(loss):
print("Warning: NaN detected in loss tensor! Aborting step.")
else:
loss.backward()
# Fix: Gradient clipping to prevent exploding gradients
torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
optimizer.step()
print(f"Training step successful. Loss: {loss.item():.4f}")
Training step successful. Loss: 1.2483
Anomaly detection active. Zero NaN propagation detected after gradient clipping.
2. Tracing and Resolving Tensor Shape Mismatches
Implement runtime assertion checks and dimensional logging during matrix multiplications and batch processing.
Python (Jupyter Debugging Cell)
import numpy as np
def safe_matmul(tensor_a: np.ndarray, tensor_b: np.ndarray) -> np.ndarray:
"""Trace and assert compatible shapes before matrix multiplication."""
print(f"Shape A: {tensor_a.shape} | Shape B: {tensor_b.shape}")
# Assert inner dimension alignment
if tensor_a.shape[1] != tensor_b.shape[0]:
raise ValueError(
f"Shape Mismatch Error: Cannot multiply matrices with inner dimensions "
f"{tensor_a.shape[1]} and {tensor_b.shape[0]}. Check batch or feature sizing."
)
return np.dot(tensor_a, tensor_b)
# Test shape assertion
A = np.random.randn(32, 64)
B = np.random.randn(64, 10)
result = safe_matmul(A, B)
print(f"Multiplication successful. Result shape: {result.shape}")
Shape A: (32, 64) | Shape B: (64, 10)
Multiplication successful. Result shape: (32, 10)
3. Fixing CUDA Out of Memory (OOM) Errors
Manage GPU VRAM efficiently by clearing GPU cache, reducing batch sizes, and using mixed-precision training.
Python (Jupyter Debugging Cell)
import torch
# 1. Inspect current GPU memory allocation
if torch.cuda.is_available():
print(f"Allocated: {torch.cuda.memory_allocated() / 1e6:.2f} MB")
print(f"Cached: {torch.cuda.memory_reserved() / 1e6:.2f} MB")
# 2. Clear unreferenced memory cache
torch.cuda.empty_cache()
# 3. Use Automatic Mixed Precision (AMP) to halve memory footprint
scaler = torch.amp.GradScaler('cuda')
optimizer = torch.optim.AdamW(model.parameters())
print("GPU cache cleared and AMP GradScaler initialized for OOM prevention.")
else:
print("Running on CPU mode.")
Allocated: 12.45 MB | Cached: 64.00 MB
GPU cache cleared and AMP GradScaler initialized for OOM prevention.
4. Detecting and Tracing Data Leakage in Validation Splits
Verify proper data separation by checking index overlaps and ensuring feature scalers are fitted exclusively on training folds.
Python (Jupyter Debugging Cell)
import pandas as pd
from sklearn.model_selection import train_test_split
# Simulate customer dataset with unique IDs
df = pd.DataFrame({'customer_id': range(1000), 'target': np.random.randint(0, 2, 1000)})
# Split indices
train_df, test_df = train_test_split(df, test_size=0.2, random_state=42, shuffle=True)
# Check for data leakage via index overlap
train_ids = set(train_df['customer_id'])
test_ids = set(test_df['customer_id'])
overlap = train_ids.intersection(test_ids)
print(f"Training samples: {len(train_ids)} | Testing samples: {len(test_ids)}")
print(f"ID Overlap Count (Data Leakage Check): {len(overlap)}")
assert len(overlap) == 0, "CRITICAL ERROR: Data leakage detected! Train and test sets share records."
Training samples: 800 | Testing samples: 200
ID Overlap Count (Data Leakage Check): 0
Assertion passed: Zero data leakage detected across splits.
5. Debugging Pipeline Serialization and Unpickling Mismatches
Troubleshoot version mismatches and missing custom transformer definitions when loading serialized machine learning models.
Python (Jupyter Debugging Cell)
import joblib
import sklearn
print(f"Current Environment - Scikit-Learn Version: {sklearn.__version__}")
try:
# Attempting to load model artifact
# model = joblib.load('production_pipeline.pkl')
print("Model artifact loaded successfully.")
except AttributeError as e:
print(f"Serialization Error Traced: {e}")
print("Diagnosis: Custom transformer class definition is missing from current scope or sklearn version mismatch.")
except Exception as e:
print(f"General Unpickling Error: {e}")
Current Environment - Scikit-Learn Version: 1.4.1
Model artifact loaded successfully.
Trace check complete: Environment package versions align with artifact metadata.