π From Sequential State Propagation to Parallel Global Self-Attention
Processing temporal and sequential dataβsuch as financial time series, telemetry streams, and natural language textβdemands models that represent order-dependent temporal dependencies. Recurrent Neural Networks (RNNs) process sequences sequentially through an internal recurrent hidden state $\mathbf{h}_t = \tanh(\mathbf{W}_{hh}\mathbf{h}_{t-1} + \mathbf{W}_{xh}\mathbf{x}_t + \mathbf{b}_h)$, passing historical memory step-by-step. However, unfolded Backpropagation Through Time (BPTT) across long horizons causes repeated multiplications of the recurrent weight matrix $\mathbf{W}_{hh}$, triggering exponential gradient vanishing or explosion ($\|\mathbf{W}_{hh}^T\| \to 0$ or $\infty$). Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU) addressed this failure via additive internal cell state highways regulated by non-linear sigmoid gates (Forget, Input, and Output gates). Yet, recurrent models remained bound to sequential execution ($\mathcal{O}(T)$ sequential steps), preventing hardware parallelization over long sequences. The Transformer architecture (Vaswani et al., 2017) completely abolished recurrence in favor of Scaled Dot-Product Self-Attention, comparing all tokens in a sequence simultaneously in $\mathcal{O}(1)$ sequential operations via query, key, and value tensors ($\text{Attention}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) = \text{softmax}(\frac{\mathbf{Q}\mathbf{K}^T}{\sqrt{d_k}})\mathbf{V}$), unlocking modern LLM and multimodal scale.
THE EVOLUTION OF SEQUENCE MODELING ARCHITECTURES:
1. RECURRENT NEURAL NETWORK (Elman RNN - O(T) Sequential Depth):
h_{t-1} βββΊ [ W_hh * h_{t-1} + W_xh * x_t + b ] βββΊ tanh βββΊ h_t βββΊ y_t
β²
x_t ββββββββββββββββ
* Sequential Constraint: h_t CANNOT be calculated until h_{t-1} completes.
* Path Length Between Step 1 and Step T: O(T) sequential matrix steps.
2. LONG SHORT-TERM MEMORY (LSTM - Constant Additive Cell Highway):
Cell State C_{t-1} ββββββ[ x f_t ]βββββββ( + i_t * c~_t )βββββββββΊ C_t
β² β² β
Hidden State h_{t-1} βββ β β βΌ
βββΊ [Forget Gate] [Input Gate] [Output Gate o_t] βββΊ h_t
Input Vector x_t βββββββ f_t = Ο(...) i_t = Ο(...) o_t = Ο(...)
3. TRANSFORMER SELF-ATTENTION (Vaswani et al. - O(1) Sequential Operations):
Input Sequence X = [x_1, x_2, ..., x_T] (Processed simultaneously in parallel!)
β
ββββββββββββββββββββββββ¬βββββββββββββββββββββββ
βΌ βΌ βΌ
Query (Q = X * W_Q) Key (K = X * W_K) Value (V = X * W_V)
β β β
ββββββββββββΊ [ Q * K^T / βd_k ] β
β β
βΌ β
[ Softmax ] βββΊ (Attention Map) β
β β
βββββββββββββββΊ [ (Map) * V ] βββΊ Output Representation
* Path Length Between ANY Two Tokens: O(1) direct dot product edge!
* Fully parallelizable on GPU hardware during both forward and backward passes.
import numpy as np
import pandas as pd
# Scenario: Financial market price-trend sequential forecasting
# Predict next-step trend based on previous time-series steps
np.random.seed(42)
SEQ_LEN = 15 # 15 temporal steps
INPUT_DIM = 2 # [Normalized_Price_Delta, Volume_Spike]
HIDDEN_DIM = 4 # Latent hidden memory state
# Generate a synthetic temporal sequence
X_seq = np.random.randn(SEQ_LEN, INPUT_DIM)
y_target = np.array([1.0]) # Upward price breakout target at final step
# ==========================================================
# 1. SCRATCH ELMAN RNN CLASS WITH EXPLICIT BPTT
# ==========================================================
class ElmanRNNScratch:
def __init__(self, input_dim, hidden_dim):
# Xavier/Glorot weight initialization
self.W_xh = np.random.randn(hidden_dim, input_dim) * np.sqrt(2.0 / (input_dim + hidden_dim))
self.W_hh = np.random.randn(hidden_dim, hidden_dim) * np.sqrt(2.0 / (hidden_dim + hidden_dim))
self.b_h = np.zeros((hidden_dim, 1))
self.W_hy = np.random.randn(1, hidden_dim) * np.sqrt(2.0 / hidden_dim)
self.b_y = np.zeros((1, 1))
def forward(self, X):
T = X.shape[0]
self.h_states = {}
self.h_states[-1] = np.zeros((self.W_hh.shape[0], 1))
# Forward pass through time steps t = 0 to T-1
for t in range(T):
x_t = X[t].reshape(-1, 1)
# h_t = tanh( W_hh * h_{t-1} + W_xh * x_t + b_h )
linear_h = self.W_hh @ self.h_states[t - 1] + self.W_xh @ x_t + self.b_h
self.h_states[t] = np.tanh(linear_h)
# Final output prediction from the last hidden state h_{T-1}
self.y_pred = self.W_hy @ self.h_states[T - 1] + self.b_y
return self.y_pred
def backward_diagnostics(self, X, y_true):
T = X.shape[0]
# Loss: 0.5 * (y_pred - y_true)^2
dL_dy = (self.y_pred - y_true) # Scalar output error
# Gradient arriving at final hidden state h_{T-1}
dh_T = self.W_hy.T @ dL_dy
# Track gradient magnitude reaching each historical time step
dh_current = dh_T.copy()
step_gradient_norms = []
# Backpropagate through time from t = T-1 down to 0
for t in reversed(range(T)):
# d/d(linear_h) of tanh is (1 - h_t^2)
dtanh = (1.0 - self.h_states[t] ** 2)
dh_raw = dh_current * dtanh
# Record Frobenius norm of gradient at time step t
step_gradient_norms.append((t, np.linalg.norm(dh_raw)))
# Recurrent backpropagation step: dh_{t-1} = W_hh^T @ dh_raw
dh_current = self.W_hh.T @ dh_raw
return sorted(step_gradient_norms, key=lambda x: x[0])
rnn = ElmanRNNScratch(input_dim=INPUT_DIM, hidden_dim=HIDDEN_DIM)
pred = rnn.forward(X_seq)
grad_records = rnn.backward_diagnostics(X_seq, y_target)
df_bptt = pd.DataFrame(grad_records, columns=["Time_Step_t", "Gradient_Norm_at_Step"])
df_bptt["Temporal_Distance_from_Loss"] = SEQ_LEN - 1 - df_bptt["Time_Step_t"]
print("=" * 65)
print("ELMAN RNN: BACKPROPAGATION THROUGH TIME (BPTT) GRADIENT AUDIT")
print("=" * 65)
print(f"Sequence Length (T) : {SEQ_LEN} Time Steps")
print(f"Final Model Output (y_pred) : {pred.item():.4f} (Target: {y_target[0]:.1f})")
print("-" * 65)
print(df_bptt.iloc[[0, 2, 5, 8, 11, 14]][["Time_Step_t", "Temporal_Distance_from_Loss", "Gradient_Norm_at_Step"]].to_string(index=False))
print("-" * 65)
print("Diagnostic Finding:")
recent_grad = df_bptt.loc[df_bptt['Time_Step_t'] == 14, 'Gradient_Norm_at_Step'].values[0]
oldest_grad = df_bptt.loc[df_bptt['Time_Step_t'] == 0, 'Gradient_Norm_at_Step'].values[0]
print(f" Gradient Norm at Step t=14 (Distance = 0) : {recent_grad:.6f}")
print(f" Gradient Norm at Step t=0 (Distance = 14) : {oldest_grad:.6e}")
print(f" Gradient Decay Factor : Collapsed by a factor of {recent_grad / (oldest_grad + 1e-15):.1f}x!")
print(" This empirical vanishing gradient prevents standard RNNs from learning long dependencies.")
import numpy as np
import pandas as pd
# Scenario: IoT industrial boiler pressure anomaly detection
# Evaluate single-step gating operations on incoming pressure telemetry
np.random.seed(42)
def sigmoid(z):
return 1.0 / (1.0 + np.exp(-np.clip(z, -50.0, 50.0)))
class LSTMCellScratch:
def __init__(self, input_dim=3, hidden_dim=4):
self.hidden_dim = hidden_dim
# Concatenated weights for 4 internal gates: [Forget, Input, Candidate, Output]
# Shape: (4 * hidden_dim, input_dim + hidden_dim)
combined_dim = input_dim + hidden_dim
self.W = np.random.randn(4 * hidden_dim, combined_dim) * np.sqrt(2.0 / combined_dim)
self.b = np.zeros((4 * hidden_dim, 1))
# Initialize forget gate biases to +1.0 (Jozefowicz et al. best practice to prevent early forgetting)
self.b[:hidden_dim] = 1.0
def step(self, x_t, h_prev, c_prev):
# Concatenate current input and previous hidden state: (input_dim + hidden_dim, 1)
concat_input = np.vstack([x_t, h_prev])
# Parallel affine transformation for all four gates
gates_raw = self.W @ concat_input + self.b
H = self.hidden_dim
# 1. Forget Gate: f_t = Ο( W_f * [x_t, h_{t-1}] + b_f )
f_t = sigmoid(gates_raw[0:H])
# 2. Input Gate: i_t = Ο( W_i * [x_t, h_{t-1}] + b_i )
i_t = sigmoid(gates_raw[H:2*H])
# 3. Candidate Cell State: C~_t = tanh( W_c * [x_t, h_{t-1}] + b_c )
c_tilde = np.tanh(gates_raw[2*H:3*H])
# 4. Cell State Update (The Constant Additive Error Carousel):
# C_t = f_t β C_{t-1} + i_t β C~_t
c_next = (f_t * c_prev) + (i_t * c_tilde)
# 5. Output Gate: o_t = Ο( W_o * [x_t, h_{t-1}] + b_o )
o_t = sigmoid(gates_raw[3*H:4*H])
# 6. Emitted Hidden State: h_t = o_t β tanh( C_t )
h_next = o_t * np.tanh(c_next)
return h_next, c_next, f_t, i_t, o_t
# Simulate 3 incoming time steps of sensor readings: [Pressure, Temperature, Flow]
lstm = LSTMCellScratch(input_dim=3, hidden_dim=4)
h_state = np.zeros((4, 1))
c_state = np.zeros((4, 1))
incoming_telemetry = [
np.array([[2.5], [110.0], [45.0]]), # Normal baseline
np.array([[2.8], [115.0], [46.0]]), # Steady
np.array([[8.4], [195.0], [12.0]]) # Critical pressure surge anomaly
]
gate_tracking = []
for step_idx, x in enumerate(incoming_telemetry, start=1):
h_state, c_state, f_gate, i_gate, o_gate = lstm.step(x, h_state, c_state)
gate_tracking.append({
"Time_Step": f"Step_{step_idx}",
"Mean_Forget_Gate (f)": round(float(np.mean(f_gate)), 4),
"Mean_Input_Gate (i)": round(float(np.mean(i_gate)), 4),
"Mean_Output_Gate (o)": round(float(np.mean(o_gate)), 4),
"Cell_State_Norm ||C||": round(float(np.linalg.norm(c_state)), 4),
"Hidden_State_Norm ||h||": round(float(np.linalg.norm(h_state)), 4)
})
df_lstm = pd.DataFrame(gate_tracking)
print("=" * 65)
print("LSTM GATING DYNAMICS & CELL HIGHWAY AUDIT")
print("=" * 65)
print(df_lstm.to_string(index=False))
print("-" * 65)
print("Mathematical Gradient Highway Property:")
print(" In the cell state update C_t = f_t * C_{t-1} + i_t * C~_t, the derivative is:")
print(" dC_t / dC_{t-1} = f_t (an additive identity gate).")
print(" When f_t β 1.0, gradients propagate across hundreds of steps without decaying,")
print(" eliminating the exponential vanishing gradient problem that plagues standard RNNs.")
import numpy as np
import pandas as pd
# Scenario: Document semantic query matching
# Sequence Length = 4 tokens, Embedding Head Dimension d_k = 64
np.random.seed(42)
SEQ_LEN = 4
D_K = 64
# Generate synthetic Query, Key, and Value projections
Q = np.random.randn(SEQ_LEN, D_K)
K = np.random.randn(SEQ_LEN, D_K)
V = np.random.randn(SEQ_LEN, D_K)
def softmax(z, axis=-1):
# Numerically stable softmax with max subtraction
e_z = np.exp(z - np.max(z, axis=axis, keepdims=True))
return e_z / np.sum(e_z, axis=axis, keepdims=True)
# 1. Unscaled Attention Scores: S_raw = Q @ K^T
raw_scores = Q @ K.T
# 2. Scaled Attention Scores: S_scaled = (Q @ K^T) / sqrt(d_k)
scale_factor = np.sqrt(D_K)
scaled_scores = raw_scores / scale_factor
# Compute attention probability distribution maps
attn_weights_unscaled = softmax(raw_scores, axis=-1)
attn_weights_scaled = softmax(scaled_scores, axis=-1)
# Compute context representation output: Attention_Map @ V
context_output = attn_weights_scaled @ V
# Theoretical Variance Proof:
# If q_i, k_i ~ i.i.d. N(0, 1), then Var(q Β· k) = Var(β_{i=1}^{d_k} q_i * k_i) = d_k
var_raw = np.var(raw_scores)
var_scaled = np.var(scaled_scores)
print("=" * 65)
print(f"SCALED DOT-PRODUCT ATTENTION AUDIT (d_k = {D_K}, βd_k = {scale_factor:.1f})")
print("=" * 65)
print(f"Raw Dot-Product Variance : {var_raw:.2f} (Theoretical expectation = {D_K}.0)")
print(f"Scaled Dot-Product Variance : {var_scaled:.4f} (Stabilized to ~1.0!)")
print("-" * 65)
print("Unscaled Softmax Row 0 (Saturated / Extreme Polarization):")
print(np.round(attn_weights_unscaled[0], 4))
print("Scaled Softmax Row 0 (Balanced Information Distribution):")
print(np.round(attn_weights_scaled[0], 4))
print("-" * 65)
print(f"Context Output Matrix Shape : {context_output.shape} (SEQ_LEN x D_K)")
print("Theoretical Proof: For large d_k (e.g. 64 or 128), unscaled dot products grow large.")
print("Softmax saturates to 1.0 for the max value and 0.0 elsewhere, causing vanishing gradients.")
print("Dividing by βd_k keeps the variance near 1.0, preserving gradient flow through Softmax.")
import numpy as np
import pandas as pd
# Parameters: Max Sequence Length = 6 tokens, Embedding Model Dimension d_model = 8
MAX_POS = 6
D_MODEL = 8
def generate_sinusoidal_positional_encoding(max_seq_len, d_model):
pe = np.zeros((max_seq_len, d_model))
# pos vector: Shape (max_seq_len, 1)
position = np.arange(max_seq_len)[:, np.newaxis]
# Division term: 10000^(2i / d_model) = exp( 2i * -log(10000) / d_model )
div_term = np.exp(np.arange(0, d_model, 2) * -(np.log(10000.0) / d_model))
# Even dimensions: 2i -> sin
pe[:, 0::2] = np.sin(position * div_term)
# Odd dimensions: 2i+1 -> cos
pe[:, 1::2] = np.cos(position * div_term)
return pe
PE_table = generate_sinusoidal_positional_encoding(MAX_POS, D_MODEL)
# Verify linear shift property: The dot product of PE(pos) and PE(pos + k)
# depends strictly on the relative distance k, regardless of absolute position!
dot_pos0_pos2 = np.dot(PE_table[0], PE_table[2])
dot_pos2_pos4 = np.dot(PE_table[2], PE_table[4])
dot_pos1_pos5 = np.dot(PE_table[1], PE_table[5])
df_pe = pd.DataFrame(
np.round(PE_table, 3),
columns=[f"Dim_{i}" for i in range(D_MODEL)],
index=[f"Token_Pos_{p}" for p in range(MAX_POS)]
)
print("=" * 65)
print(f"SINUSOIDAL POSITIONAL ENCODING MATRIX ({MAX_POS} TOKENS, D_MODEL = {D_MODEL})")
print("=" * 65)
print(df_pe.to_string())
print("-" * 65)
print("Relative Positional Invariance Proof (Dot Product Inner Products):")
print(f" Similarity between Pos 0 and Pos 2 (Ξ = 2): {dot_pos0_pos2:.4f}")
print(f" Similarity between Pos 2 and Pos 4 (Ξ = 2): {dot_pos2_pos4:.4f}")
print(f" Similarity between Pos 1 and Pos 5 (Ξ = 4): {dot_pos1_pos5:.4f}")
print("-" * 65)
print("Key Property: Tokens separated by identical relative distances (Ξ=2) maintain")
print(f"nearly identical geometric inner products ({dot_pos0_pos2:.2f} β {dot_pos2_pos4:.2f}),")
print("enabling the network to attend by relative offset rather than absolute position.")
import torch
import torch.nn as nn
import torch.nn.functional as F
import pandas as pd
class MultiHeadAttentionScratch(nn.Module):
def __init__(self, d_model=64, num_heads=4):
super(MultiHeadAttentionScratch, self).__init__()
assert d_model % num_heads == 0, "d_model must be divisible by num_heads"
self.d_model = d_model
self.num_heads = num_heads
self.d_k = d_model // num_heads # Dimension per individual head
# Linear projection matrices for Q, K, V and final output
self.W_q = nn.Linear(d_model, d_model, bias=False)
self.W_k = nn.Linear(d_model, d_model, bias=False)
self.W_v = nn.Linear(d_model, d_model, bias=False)
self.W_o = nn.Linear(d_model, d_model, bias=False)
def forward(self, x, is_causal=False):
batch_size, seq_len, _ = x.shape
# 1. Project inputs: (Batch, SeqLen, d_model)
Q = self.W_q(x)
K = self.W_k(x)
V = self.W_v(x)
# 2. Reshape and transpose for multi-head computation:
# (Batch, SeqLen, num_heads, d_k) -> (Batch, num_heads, SeqLen, d_k)
Q = Q.view(batch_size, seq_len, self.num_heads, self.d_k).transpose(1, 2)
K = K.view(batch_size, seq_len, self.num_heads, self.d_k).transpose(1, 2)
V = V.view(batch_size, seq_len, self.num_heads, self.d_k).transpose(1, 2)
# 3. Scaled Dot-Product Attention: (Q @ K^T) / sqrt(d_k)
scores = torch.matmul(Q, K.transpose(-2, -1)) / np.sqrt(self.d_k)
# 4. Apply Autoregressive Causal Masking (Upper-triangle filled with -inf)
if is_causal:
# Upper triangular matrix with diagonal=1
causal_mask = torch.triu(torch.ones(seq_len, seq_len, device=x.device), diagonal=1).bool()
# Fills future look-ahead positions with -1e9 so softmax becomes 0.0
scores = scores.masked_fill(causal_mask, float("-inf"))
# 5. Softmax attention distribution
attn_weights = F.softmax(scores, dim=-1)
# 6. Contract with Values: (Batch, num_heads, SeqLen, d_k)
context = torch.matmul(attn_weights, V)
# 7. Concatenate heads back into d_model:
# (Batch, SeqLen, num_heads * d_k = d_model)
context = context.transpose(1, 2).contiguous().view(batch_size, seq_len, self.d_model)
# 8. Final linear projection
output = self.W_o(context)
return output, attn_weights
# Instantiate module: Batch=1, Sequence Length=4, d_model=16, Heads=2 (d_k=8)
torch.manual_seed(42)
mha = MultiHeadAttentionScratch(d_model=16, num_heads=2)
dummy_input = torch.randn(1, 4, 16)
# Forward pass with causal masking enabled
out, weights = mha(dummy_input, is_causal=True)
# Inspect Head 0 Causal Attention Weight Map
head_0_map = weights[0, 0].detach().numpy()
df_causal = pd.DataFrame(
np.round(head_0_map, 4),
index=[f"Token_{i} (Query)" for i in range(4)],
columns=[f"Token_{j} (Key)" for j in range(4)]
)
print("=" * 65)
print("MULTI-HEAD ATTENTION (MHA): CAUSAL AUTOREGRESSIVE MASK AUDIT")
print("=" * 65)
print(f"Input Shape: (1, 4, 16) βββΊ Output Shape: {tuple(out.shape)}")
print("-" * 65)
print("Head 0 Attention Probability Distribution Map (Post-Softmax):")
print(df_causal.to_string())
print("-" * 65)
print("Causal Verification:")
print(" Token 0 can attend ONLY to Token 0 (100% self-attention).")
print(" Token 1 attends to Tokens 0 and 1, with zero weight on Tokens 2 and 3.")
print(" The upper triangle contains exact 0.0 values, preventing future token leakage.")
import time
import torch
import torch.nn as nn
import pandas as pd
# Benchmark configuration: 128 sequences, length 64, embedding dimension 64
BATCH_SIZE = 128
SEQ_LEN = 64
EMBED_DIM = 64
NUM_CLASSES = 2
N_BENCH_RUNS = 25
torch.manual_seed(42)
X_batch = torch.randn(BATCH_SIZE, SEQ_LEN, EMBED_DIM)
# ==========================================================
# 1. BI-DIRECTIONAL LSTM CLASSIFIER
# ==========================================================
class LSTMClassifier(nn.Module):
def __init__(self, embed_dim, hidden_dim, num_classes):
super(LSTMClassifier, self).__init__()
self.lstm = nn.LSTM(
embed_dim, hidden_dim, num_layers=2,
batch_first=True, bidirectional=True
)
self.fc = nn.Linear(hidden_dim * 2, num_classes)
def forward(self, x):
# Sequential recurrent step execution
out, (h_n, _) = self.lstm(x)
# Concatenate forward and backward final hidden states
h_pool = torch.cat([h_n[-2], h_n[-1]], dim=-1)
return self.fc(h_pool)
# ==========================================================
# 2. TRANSFORMER ENCODER CLASSIFIER (PARALLEL ATTENTION)
# ==========================================================
class TransformerClassifier(nn.Module):
def __init__(self, embed_dim, num_heads, num_classes):
super(TransformerClassifier, self).__init__()
encoder_layer = nn.TransformerEncoderLayer(
d_model=embed_dim, nhead=num_heads, dim_feedforward=embed_dim * 4,
batch_first=True, dropout=0.1
)
self.transformer = nn.TransformerEncoder(encoder_layer, num_layers=2)
self.fc = nn.Linear(embed_dim, num_classes)
def forward(self, x):
# Parallel attention computation
out = self.transformer(x)
# Global average pool across the sequence length dimension
pooled = out.mean(dim=1)
return self.fc(pooled)
model_lstm = LSTMClassifier(EMBED_DIM, hidden_dim=64, num_classes=NUM_CLASSES)
model_trans = TransformerClassifier(EMBED_DIM, num_heads=4, num_classes=NUM_CLASSES)
# Parameter audit
params_lstm = sum(p.numel() for p in model_lstm.parameters())
params_trans = sum(p.numel() for p in model_trans.parameters())
# Execution Speed Benchmark
model_lstm.eval()
model_trans.eval()
with torch.no_grad():
# Warm-up pass
_ = model_lstm(X_batch)
_ = model_trans(X_batch)
# 1. LSTM Timing
t0 = time.perf_counter()
for _ in range(N_BENCH_RUNS):
_ = model_lstm(X_batch)
time_lstm = (time.perf_counter() - t0) * 1000.0 / N_BENCH_RUNS
# 2. Transformer Timing
t0 = time.perf_counter()
for _ in range(N_BENCH_RUNS):
_ = model_trans(X_batch)
time_trans = (time.perf_counter() - t0) * 1000.0 / N_BENCH_RUNS
df_bench = pd.DataFrame([
{
"Architecture": "Bi-Directional LSTM (2 Layers)",
"Parameters": f"{params_lstm:,}",
"Sequential_Ops": f"O(T) = {SEQ_LEN} sequential steps",
"Latency_Per_Batch_ms": round(time_lstm, 2),
"Speed_Ratio": "1.0x (Baseline)"
},
{
"Architecture": "Transformer Encoder (2 Layers)",
"Parameters": f"{params_trans:,}",
"Sequential_Ops": "O(1) parallel matrix multiplications",
"Latency_Per_Batch_ms": round(time_trans, 2),
"Speed_Ratio": f"{time_lstm / time_trans:.1f}x Faster"
}
])
print("=" * 65)
print("SEQUENCE PROCESSING BENCHMARK: LSTM VS. TRANSFORMER")
print("=" * 65)
print(df_bench.to_string(index=False))
print("-" * 65)
print("System Takeaway: Although the Transformer evaluates full self-attention,")
print("its operations map to parallel matrix multiplications on GPUs, avoiding")
print("the sequential dependency loops of recurrent architectures.")
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import mean_squared_error
# High-frequency electrical power grid load forecasting (Time-series)
np.random.seed(42)
torch.manual_seed(42)
TOTAL_TIMESTEPS = 1000
WINDOW_SIZE = 12 # Predict next step using previous 12 hours
# Synthetic power demand curve: Trend + Seasonal oscillation + Noise
t = np.linspace(0, 50, TOTAL_TIMESTEPS)
power_load = 500.0 + 2.5 * t + 80.0 * np.sin(2 * np.pi * t / 24) + np.random.normal(0, 10, TOTAL_TIMESTEPS)
# 1. STRICT TEMPORAL CHRONOLOGICAL SPLIT (No random shuffle!)
# 75% Historical Training, 25% Future Evaluation
split_idx = int(TOTAL_TIMESTEPS * 0.75)
train_raw = power_load[:split_idx].reshape(-1, 1)
test_raw = power_load[split_idx:].reshape(-1, 1)
# 2. Fit Scaler STRICTLY on historical training period
scaler = StandardScaler()
train_scaled = scaler.fit_transform(train_raw).flatten()
# Transform future test data using historical mean and variance
test_scaled = scaler.transform(test_raw).flatten()
# 3. Create Rolling Temporal Sliding Windows
def create_sliding_windows(data, window_size):
X, y = [], []
for i in range(len(data) - window_size):
X.append(data[i:i + window_size])
y.append(data[i + window_size])
return np.array(X), np.array(y)
X_tr, y_tr = create_sliding_windows(train_scaled, WINDOW_SIZE)
X_te, y_te = create_sliding_windows(test_scaled, WINDOW_SIZE)
# Convert to PyTorch tensors (Shape: Batch, SeqLen, 1)
X_tr_t = torch.tensor(X_tr, dtype=torch.float32).unsqueeze(-1)
y_tr_t = torch.tensor(y_tr, dtype=torch.float32).unsqueeze(-1)
X_te_t = torch.tensor(X_te, dtype=torch.float32).unsqueeze(-1)
# 4. PyTorch LSTM Forecaster
class TemporalForecastingLSTM(nn.Module):
def __init__(self):
super(TemporalForecastingLSTM, self).__init__()
self.lstm = nn.LSTM(input_size=1, hidden_size=32, batch_first=True)
self.fc = nn.Linear(32, 1)
def forward(self, x):
out, _ = self.lstm(x)
return self.fc(out[:, -1, :])
model = TemporalForecastingLSTM()
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
# Train on historical window
model.train()
for epoch in range(40):
optimizer.zero_grad()
loss = criterion(model(X_tr_t), y_tr_t)
loss.backward()
optimizer.step()
# 5. Out-of-Sample Future Evaluation
model.eval()
with torch.no_grad():
scaled_preds = model(X_te_t).numpy()
# Invert scaling back to Megawatts (MW) physical units
y_test_physical = scaler.inverse_transform(y_te.reshape(-1, 1)).flatten()
y_pred_physical = scaler.inverse_transform(scaled_preds).flatten()
test_rmse = np.sqrt(mean_squared_error(y_test_physical, y_pred_physical))
print("=" * 65)
print("TEMPORAL ROLLING-WINDOW PIPELINE FORECASTING AUDIT")
print("=" * 65)
print(f"Total Time-Series Horizon : {TOTAL_TIMESTEPS} Hours")
print(f"Historical Training Range : Hours 0 to {split_idx}")
print(f"Future Unseen Forecast Range : Hours {split_idx} to {TOTAL_TIMESTEPS}")
print("-" * 65)
print(f"Out-of-Sample Test RMSE : {test_rmse:.2f} MW")
print(f"Mean True Load on Test Set : {np.mean(y_test_physical):.2f} MW (Error: {test_rmse/np.mean(y_test_physical)*100:.2f}%)")
print("-" * 65)
print("Leakage Protection Verification:")
print(" Temporal splits preserve the arrow of time; the scaler calculated ΞΌ and Ο")
print(" strictly from the first 750 hours, preventing future grid trends from leaking.")
import torch
import numpy as np
import pandas as pd
# Rotary Positional Embedding (RoPE) Implementation from first principles
def precompute_rope_freqs_cis(dim: int, max_seq_len: int, theta: float = 10000.0):
# Dim must be even: pair consecutive dimensions into 2D rotation planes
freqs = 1.0 / (theta ** (torch.arange(0, dim, 2)[: (dim // 2)].float() / dim))
t = torch.arange(max_seq_len, device=freqs.device)
# Outer product: angle matrix of shape (max_seq_len, dim // 2)
freqs = torch.outer(t, freqs).float()
# Complex exponentials: e^(i * m * ΞΈ) = cos(mΞΈ) + i * sin(mΞΈ)
freqs_cis = torch.polar(torch.ones_like(freqs), freqs)
return freqs_cis
def apply_rotary_emb(x: torch.Tensor, freqs_cis: torch.Tensor) -> torch.Tensor:
# Reshape real tensor x into complex numbers by pairing consecutive dimensions
# Shape: (Batch, SeqLen, Dim) -> (Batch, SeqLen, Dim//2)
x_complex = torch.view_as_complex(x.float().reshape(*x.shape[:-1], -1, 2))
# Align complex frequencies
freqs_cis = freqs_cis[: x.shape[1]]
# Multiply complex numbers (executes 2D coordinate rotation!)
x_rotated = x_complex * freqs_cis
# Convert back to real tensor
x_out = torch.view_as_real(x_rotated).flatten(-2)
return x_out.type_as(x)
# Setup: 2-token query and key embeddings of dimension d = 4
torch.manual_seed(42)
D_HEAD = 4
SEQ_LEN = 2
q = torch.randn(1, SEQ_LEN, D_HEAD)
k = torch.randn(1, SEQ_LEN, D_HEAD)
freqs_cis = precompute_rope_freqs_cis(dim=D_HEAD, max_seq_len=16)
# Apply rotation to Queries and Keys
q_rot = apply_rotary_emb(q, freqs_cis)
k_rot = apply_rotary_emb(k, freqs_cis)
# Inner product before and after rotation
unrotated_dot = (q[:, 0] * k[:, 1]).sum().item()
rotated_dot = (q_rot[:, 0] * k_rot[:, 1]).sum().item()
print("=" * 65)
print("ROTARY POSITIONAL EMBEDDING (RoPE) COMPLEX PLANE AUDIT")
print("=" * 65)
print(f"Head Embedding Dimension : {D_HEAD} (2 Complex Rotation Planes)")
print(f"Token 0 Query Tensor (Pre) : {np.round(q[0, 0].numpy(), 4)}")
print(f"Token 0 Query Tensor (RoPE) : {np.round(q_rot[0, 0].numpy(), 4)}")
print("-" * 65)
print(f"Unrotated Inner Product Qβ Β· Kβ : {unrotated_dot:.4f}")
print(f"RoPE Rotated Inner Product Qβ Β· Kβ: {rotated_dot:.4f}")
print("-" * 65)
print("Architectural Rationale:")
print(" RoPE rotates queries and keys in 2D coordinate pairs by angles proportional to their positions.")
print(" The resulting dot product depends exclusively on relative token distance (m - n),")
print(" providing length extrapolation without requiring fixed positional matrices.")
SEQUENCE MODELING PARADIGMS ARCHITECTURAL COMPARISON:
ββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββ
β Architecture / Model β Temporal Mechanism β Computational Scalingβ Primary Production Role β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β Elman RNN β Sequential hidden state β O(T * DΒ²) Computes β Small sequence baselines (< 20); β
β β h_t = tanh(Wh + Wx + b) β O(T) Sequential Step β historical baseline β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β LSTM (Hochreiter & β Cell state error carousel β O(T * 4DΒ²) Computes β Streaming IoT telemetry, battery β
β Schmidhuber 1997) β Forget/Input/Output gates β O(T) Sequential Step β sensor forecasting on Edge CPUs β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β GRU (Cho et al. 2014) β Simplified gating β O(T * 3DΒ²) Computes β Faster training than LSTM with β
β β Reset and Update gates β O(T) Sequential Step β comparable capacity on audio/NLP β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β Transformer Encoder β Bidirectional self-attention β O(TΒ² * D) Computes β Document classification, protein β
β (BERT / RoBERTa) β Full pairwise token attentionβ O(1) Sequential Step β modeling, sentence embeddings β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β Transformer Decoder β Causal masked self-attention β O(TΒ² * D) Computes β Autoregressive generation (GPT, β
β (GPT-4, LLaMA, Mistral) β Upper-triangle masked to -infβ O(1) Train / O(T) Genβ code completion, agent models) β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β Rotary Embeddings (RoPE) β Complex-plane 2D rotation β O(T * D) Computes β Modern LLM positional encoding; β
β β Injects relative position β Zero parameter overheadβ supports context extension β
ββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββ€
β SEQUENCE PRODUCTION BEST PRACTICES: β
β 1. NEVER use standard random cross-validation on time-series; use chronological rolling-window splits. β
β 2. When scaling sequence length T > 2,048 tokens, use FlashAttention or RoPE to manage O(TΒ²) attention overhead. β
β 3. For low-latency edge deployment on resource-constrained microcontrollers, prefer GRU over heavy Transformers. β
β 4. Always scale inputs using statistics computed strictly from historical training windows to prevent look-ahead bias.β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ