⚡ The Autoregressive Generative Core: From Teacher Forcing to Real-Time Inference
While the Transformer Encoder builds bidirectional representations by examining all tokens simultaneously, the Transformer Decoder is the engine of open-ended generation. Operating under the probability chain rule, the decoder formulates sequence generation as autoregressive prediction: estimating the categorical distribution $P(x_t \mid x_1, \dots, x_{t-1})$. Modern frontier models (GPT-4, LLaMA-3, Mistral, Qwen) are almost exclusively Decoder-Only architectures, discarding the encoder entirely. To train across full sequences in parallel on GPU tensor cores without temporal information leakage, the decoder implements a strict mathematical constraint: Causal Masked Self-Attention. During training, all positions are updated concurrently via Teacher Forcing; during inference, the model transitions into a token-by-token loop where intermediate key and value activations are cached via KV-Caching, transforming an intractable $O(N^2)$ inference recomputation into an optimized $O(N)$ memory-bound operational regime.
CLASSICAL ENCODER-DECODER DECODER (T5 / Whisper): MODERN CAUSAL DECODER-ONLY (GPT / LLaMA):
Contains 3 Sub-Layers with Cross-Attention Contains 2 Sub-Layers (Encoder Completely Removed)
Input Targets: Y_{l-1} ∈ R^{B × S_tgt × d_model} Input Sequence: X_{l-1} ∈ R^{B × S × d_model}
│ │
├─────────────────────────┐ (Residual 1) ├─────────────────────────┐ (Residual 1)
▼ │ ▼ │
┌─────────────────┐ │ ┌─────────────────┐ │
│ LayerNorm / RMS │ │ │ LayerNorm / RMS │ │
└────────┬────────┘ │ └────────┬────────┘ │
▼ │ ▼ │
┌─────────────────┐ │ ┌─────────────────┐ │
│ Causal Masked │ │ │ Causal Masked │ │
│ Self-Attention │ │ │ Self-Attention │ │
└────────┬────────┘ │ └────────┬────────┘ │
▼ │ ▼ │
┌─────────────────┐ │ ┌─────────────────┐ │
│ Element-wise (+)│ ◄──────────────┘ │ Element-wise (+)│ ◄──────────────┘
└────────┬────────┘ └────────┬────────┘
│ │
├─────────────────────────┐ (Residual 2) ├─────────────────────────┐ (Residual 2)
▼ │ ▼ │
┌─────────────────┐ │ ┌─────────────────┐ │
│ LayerNorm / RMS │ │ │ LayerNorm / RMS │ │
└────────┬────────┘ │ └────────┬────────┘ │
▼ │ ▼ │
┌─────────────────┐ │ ┌─────────────────┐ │
│ Cross-Attention │ ◄── Encoder │ │ Gated FFN │ │
│ (K, V from Enc) │ Memory │ │ (SwiGLU / MLP) │ │
└────────┬────────┘ │ └────────┬────────┘ │
▼ │ ▼ │
┌─────────────────┐ │ ┌─────────────────┐ │
│ Element-wise (+)│ ◄──────────────┘ │ Element-wise (+)│ ◄──────────────┘
└────────┬────────┘ └────────┬────────┘
│ │
├─────────────────────────┐ (Residual 3) ▼
▼ │ Output: X_l ∈ R^{B × S × d_model}
┌─────────────────┐ │
│ Feed-Forward Net│ │
└────────┬────────┘ │
▼ │
┌─────────────────┐ │
│ Element-wise (+)│ ◄──────────────┘
└────────┬────────┘
│
▼ Output: Y_l ∈ R^{B × S_tgt × d_model}
THE CAUSAL MASKING DATA TRANSFORMATION PIPELINE:
Sequence Tokens: ["The", "neural", "network", "generates"] (Sequence Length S = 4)
1. RAW ATTENTION DOT-PRODUCT MATRIX: S = (Q · K^T) / √d_k ∈ R^{4 × 4}
Every token position calculates affinity against EVERY other position:
"The" "neural" "network" "generates"
"The" ┌ 3.2 1.8 0.9 2.1 ┐ <-- Token 1 CANNOT see tokens 2, 3, 4!
"neural" │ 1.5 4.1 2.2 0.4 │ <-- Token 2 CANNOT see tokens 3, 4!
"network" │ 0.8 2.0 3.9 1.7 │ <-- Token 3 CANNOT see token 4!
"generates" └ 1.1 0.7 2.8 4.5 ┘
─────────────────────────────────────────────────────────────────────────────────────────
2. ADDITIVE CAUSAL MASK MATRIX M_causal ∈ R^{4 × 4}:
Upper-triangular components (j > i) are strictly set to -inf:
M_{i, j} = 0 if j <= i else -inf
┌ 0 -inf -inf -inf ┐
M_causal = │ 0 0 -inf -inf │
│ 0 0 0 -inf │
└ 0 0 0 0 ┘
─────────────────────────────────────────────────────────────────────────────────────────
3. SUMMATION AND ROW-WISE SOFTMAX:
S_masked = S + M_causal
Applying Softmax: A_{i, j} = exp(S_masked_{i, j}) / ∑_m exp(S_masked_{i, m})
Because exp(-inf) = 0.0, future positions collapse strictly to zero probability mass!
Final Lower-Triangular Attention Probability Matrix A ∈ R^{4 × 4}:
"The" "neural" "network" "generates"
"The" ┌ 1.00 0.00 0.00 0.00 ┐ (Row sum = 1.0)
"neural" │ 0.07 0.93 0.00 0.00 │ (Row sum = 1.0)
"network" │ 0.03 0.12 0.85 0.00 │ (Row sum = 1.0)
"generates" └ 0.02 0.02 0.14 0.82 ┘ (Row sum = 1.0)
CROSS-ATTENTION INFORMATION ROUTING (Seq2Seq / T5 / Diffusion Models):
Decoder Hidden State H_dec ∈ R^{B × S_tgt × d_model} Encoder Memory H_enc ∈ R^{B × S_src × d_model}
│ │
▼ Linear W_Q ├────────────────────────┐
Queries Q ∈ R^{B × S_tgt × d_k} ▼ Linear W_K ▼ Linear W_V
│ Keys K ∈ R^{B × S_src × d_k} Values V ∈ R^{B × S_src × d_v}
│ │ │
└───────────────────────────┬──────────────────────────────┘ │
│ Matrix Multiply: Q · K^T │
▼ │
Raw Affinity S ∈ R^{B × S_tgt × S_src} │
│ │
▼ Scale (1 / √d_k) + Source Padding Mask │
▼ Softmax along S_src dimension │
Cross-Attention Weights A ∈ R^{B × S_tgt × S_src} │
│ │
│ Matrix Multiply: A · V │
└──────────────────────────────┬────────────────────────┘
│
▼
Contextual Output ∈ R^{B × S_tgt × d_v}
THE DUAL EXECUTION MODES OF TRANSFORMER DECODERS:
┌─────────────────────────┬──────────────────────────────────────┬────────────────────────────────────────┐
│ OPERATIONAL PROPERTY │ TRAINING PHASE (TEACHER FORCING) │ INFERENCE GENERATION PHASE │
├─────────────────────────┼──────────────────────────────────────┼────────────────────────────────────────┤
│ Sequence Feeding │ Full sequence fed simultaneously │ One token generated per step │
│ Compute Parallelism │ Fully parallel across all S tokens │ Strictly sequential $O(S)$ serial loop │
│ Attention Matrix │ Materializes full lower-triangular │ Computes single row vector at step t │
│ Target Tokens Input │ Ground-truth target tokens y^*_{0..S}│ Model's own previous predicted tokens │
│ Hardware Bottleneck │ Compute-Bound (FLOPs bound) │ Memory Bandwidth Bound (VRAM I/O) │
│ Wall-Clock Complexity │ $O(1)$ GPU forward kernel dispatches │ $O(N)$ sequential GPU kernel launches │
│ KV-Cache Utilization │ Banned (Redundant; full seq in VRAM) │ Mandatory (Eliminates $O(N^2)$ waste) │
└─────────────────────────┴──────────────────────────────────────┴────────────────────────────────────────┘
WITHOUT KV-CACHE (NAIVE INFERENCE - O(N^2) COMPUTATION):
At step t = 4, the model re-projects and re-calculates all keys and values for tokens 1, 2, 3:
Step 1: Compute Q_1, K_1, V_1 ──► Emit Token 2
Step 2: Compute Q_{1..2}, K_{1..2}, V_{1..2} (Re-computes token 1!) ──► Emit Token 3
Step 3: Compute Q_{1..3}, K_{1..3}, V_{1..3} (Re-computes tokens 1, 2!) ──► Emit Token 4
Total FLOPs scale as: ∑_{t=1}^N t = N(N + 1) / 2 ➔ O(N^2) redundant matrix multiplications!
─────────────────────────────────────────────────────────────────────────────────────────
WITH KV-CACHE (OPTIMAL INFERENCE - O(N) COMPUTATION):
Keys and Values for historical tokens 1..3 are PERMANENTLY STORED in GPU memory.
Only the SINGLE NEW TOKEN (Token 4) is projected!
Step 4 Input: Only token x_4! (Sequence Length S = 1)
1. Project NEW Query: q_4 ∈ R^{1 × d_k}
2. Project NEW Key: k_4 ∈ R^{1 × d_k}
3. Project NEW Value: v_4 ∈ R^{1 × d_v}
4. UPDATE KV-CACHE IN GPU VRAM:
K_cached = Concat( K_cached_{1..3}, k_4 ) ∈ R^{4 × d_k}
V_cached = Concat( V_cached_{1..3}, v_4 ) ∈ R^{4 × d_v}
5. COMPUTE SINGLE-ROW ATTENTION VECTOR:
Scores = ( q_4 · K_cached^T ) / √d_k ∈ R^{1 × 4}
Weights = Softmax( Scores ) ∈ R^{1 × 4}
Context = Weights · V_cached ∈ R^{1 × d_v}
Total FLOPs per step: Strictly O(1) linear projections! Total generation FLOPs: O(N).
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
# ==============================================================================
# 1. CAUSAL SELF-ATTENTION WITH DYNAMIC KV-CACHE
# ==============================================================================
class CausalSelfAttentionWithCache(nn.Module):
def __init__(self, d_model: int, num_heads: int):
super().__init__()
assert d_model % num_heads == 0
self.d_model = d_model
self.num_heads = num_heads
self.head_dim = d_model // num_heads
self.scale = 1.0 / math.sqrt(self.head_dim)
self.q_proj = nn.Linear(d_model, d_model, bias=False)
self.k_proj = nn.Linear(d_model, d_model, bias=False)
self.v_proj = nn.Linear(d_model, d_model, bias=False)
self.out_proj = nn.Linear(d_model, d_model, bias=False)
def forward(self, x: torch.Tensor, kv_cache: dict = None, use_cache: bool = False):
"""
x: Input tensor of shape (Batch, SeqLen, d_model)
kv_cache: Dictionary storing historical {'k': tensor, 'v': tensor}
use_cache: Boolean flag activating incremental single-token decoding
"""
b, s, _ = x.shape
# 1. Project current token representations
q = self.q_proj(x).view(b, s, self.num_heads, self.head_dim).transpose(1, 2)
k = self.k_proj(x).view(b, s, self.num_heads, self.head_dim).transpose(1, 2)
v = self.v_proj(x).view(b, s, self.num_heads, self.head_dim).transpose(1, 2)
# 2. Ingest / Concatenate historical Key and Value representations from KV-Cache
if kv_cache is not None and use_cache:
if 'k' in kv_cache and kv_cache['k'] is not None:
# Concatenate along sequence dimension (dim=2)
k = torch.cat([kv_cache['k'], k], dim=2)
v = torch.cat([kv_cache['v'], v], dim=2)
kv_cache['k'] = k
kv_cache['v'] = v
total_seq_len = k.size(2)
# 3. Compute Attention Affinity Scores: (B, H, S_q, S_k)
scores = torch.matmul(q, k.transpose(-2, -1)) * self.scale
# 4. Apply Causal Mask ONLY if evaluating multi-token sequences (s > 1)
# In single-token cached decoding (s = 1), new token can attend to ALL historical tokens!
if s > 1:
causal_mask = torch.tril(torch.ones(s, total_seq_len, device=x.device, dtype=torch.bool))
scores = scores.masked_fill(~causal_mask.unsqueeze(0).unsqueeze(0), float('-inf'))
# 5. Softmax normalization & Value aggregation
attn_weights = F.softmax(scores, dim=-1)
context = torch.matmul(attn_weights, v) # (B, H, S_q, head_dim)
# 6. Recombine heads and project out
context = context.transpose(1, 2).contiguous().view(b, s, self.d_model)
output = self.out_proj(context)
return output, kv_cache
# ==============================================================================
# 2. COMPLETE TRANSFORMER DECODER LAYER (CAUSAL DECODER-ONLY)
# ==============================================================================
class TransformerDecoderLayer(nn.Module):
def __init__(self, d_model: int, num_heads: int, d_ff: int):
super().__init__()
self.norm1 = nn.LayerNorm(d_model)
self.attn = CausalSelfAttentionWithCache(d_model, num_heads)
self.norm2 = nn.LayerNorm(d_model)
# SwiGLU Gated Feed-Forward Network
self.w_gate = nn.Linear(d_model, d_ff, bias=False)
self.w_up = nn.Linear(d_model, d_ff, bias=False)
self.w_down = nn.Linear(d_ff, d_model, bias=False)
def forward(self, x: torch.Tensor, kv_cache: dict = None, use_cache: bool = False):
# Sub-layer 1: Causal Self-Attention + Residual
norm_x = self.norm1(x)
attn_out, kv_cache = self.attn(norm_x, kv_cache=kv_cache, use_cache=use_cache)
x = x + attn_out
# Sub-layer 2: SwiGLU FFN + Residual
norm_x = self.norm2(x)
ffn_out = self.w_down(F.silu(self.w_gate(norm_x)) * self.w_up(norm_x))
x = x + ffn_out
return x, kv_cache
# ------------------------------------------------------------------------------
# Runtime Execution & Numerical Equivalence Verification Suite
# ------------------------------------------------------------------------------
if __name__ == "__main__":
torch.manual_seed(42)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
B, S, D, H, D_ff = 1, 5, 64, 4, 128
print("[*] Initializing Transformer Decoder KV-Cache Verification Engine...")
decoder_layer = TransformerDecoderLayer(d_model=D, num_heads=H, d_ff=D_ff).to(device)
decoder_layer.eval()
# Synthetic Input Sequence of 5 tokens
input_tokens = torch.randn(B, S, D, device=device)
# 1. PARALLEL FORWARD PASS (TRAINING MODE - NO CACHE, FULL CAUSAL MASK)
with torch.no_grad():
full_out, _ = decoder_layer(input_tokens, use_cache=False)
print(f"[✔] Parallel Full Sequence Pass Computed. Shape: {full_out.shape}")
# 2. INCREMENTAL CACHED GENERATION (INFERENCE MODE - STEP-BY-STEP)
print("[*] Running Incremental KV-Cached Inference Loop...")
cache_store = {}
cached_outputs = []
with torch.no_grad():
for t in range(S):
# Feed exactly ONE token at a time: shape (1, 1, D)
current_token = input_tokens[:, t:t+1, :]
out_step, cache_store = decoder_layer(current_token, kv_cache=cache_store, use_cache=True)
cached_outputs.append(out_step)
# Concatenate all single-step outputs along sequence dimension
incremental_out = torch.cat(cached_outputs, dim=1)
print(f"[✔] Incremental Cached Pass Concluded. Shape: {incremental_out.shape}")
# 3. MATHEMATICAL EQUIVALENCE CHECK
# Check that parallel training pass matches step-by-step cached inference exactly!
max_error = torch.max(torch.abs(full_out - incremental_out)).item()
print(f"[✔] Maximum Absolute Discrepancy: {max_error:.2e}")
assert max_error < 1e-5, "Mathematical Equivalence Failure between Caching and Parallel Pass!"
print("[✔] MATHEMATICAL EQUIVALENCE VERIFIED: KV-Cache produces bit-exact representations.")
PROBABILITY LOGIT SAMPLING TRANSFORMATIONS:
Decoder Output Logits: z ∈ R^{|V|} over Vocabulary V
1. TEMPERATURE SCALING: P(x_i) = exp(z_i / τ) / ∑_j exp(z_j / τ)
• τ = 1.0 : Standard unchanged model calibration.
• τ ➔ 0.0 : Greedy Argmax (Deterministic, repetitive, zero creativity).
• τ > 1.0 : Flattened distribution (Higher entropy, creative, risks hallucinations).
─────────────────────────────────────────────────────────────────────────────────────────
2. TOP-K TRUNCATION (Fan et al., 2018):
Sort logits descendingly. Retain ONLY the top K candidates; set remaining logits to -inf.
Example (K = 3):
Raw Candidates : ["cat" (0.50), "dog" (0.30), "rabbit" (0.15), "car" (0.04), "tree" (0.01)]
Top-K (K=3) : ["cat", "dog", "rabbit"] (Discard "car", "tree")
Flaw: Fixed K cannot adapt to flat vs. peaked confidence distributions.
─────────────────────────────────────────────────────────────────────────────────────────
3. TOP-P / NUCLEUS SAMPLING (Holtzman et al., 2019):
Dynamically selects the smallest subset of tokens whose cumulative probability exceeds P:
∑_{i ∈ V^(p)} P(x_i) >= P (typically P = 0.90)
• Peaked Distribution (High Confidence):
Candidate 1 has P = 0.92 ➔ Nucleus contains ONLY 1 word!
• Flat Distribution (High Uncertainty):
50 words have P = 0.02 each ➔ Nucleus expands dynamically to contain 45 words!
Enforces context-dependent dynamic truncation, eliminating bizarre long-tail tokens.
TRANSFORMER DECODER ANOMALIES & REMEDIATIONS:
1. THE OFF-BY-ONE LABEL SHIFT TRAP IN TEACHER FORCING:
• Symptom: Training loss drops to 0.0 in 2 steps, but the model produces complete gibberish at inference!
• Theoretical Cause: Feeding the targets as their own labels without shifting:
loss = criterion(logits, targets) # FATAL: Model learns identity mapping: x_t predicts x_t!
• Correct Mathematical Slicing:
inputs = tokens[:, :-1] # Feed tokens 0 to S-1
targets = tokens[:, 1:] # Predict tokens 1 to S
loss = criterion(model(inputs), targets)
2. UNINTENTIONAL CAUSAL MASK INVERSION:
• Symptom: Decoder tokens attend ONLY to future tokens; generation degenerates immediately.
• Cause: Using torch.triu() instead of torch.tril() when building boolean attention masks!
torch.triu selects the UPPER triangle (future tokens), allowing the model to cheat during training.
• Correct Code:
causal_mask = torch.tril(torch.ones(S, S, dtype=torch.bool))
3. KV-CACHE MEMORY FRAGMENTATION IN PYTHON LISTS:
• Symptom: GPU Out of Memory after serving 100 requests due to VRAM memory fragmentation.
• Cause: Appending tensors dynamically using 'torch.cat()' inside the decoding loop!
Re-allocating a new tensor every single token step causes thousands of tiny memory fragments.
• Remediation: Pre-allocate static tensor buffers for max_seq_len ahead of time:
k_cache = torch.empty(B, H, max_seq_len, head_dim, device=device)
And maintain a write pointer index: k_cache[:, :, step:step+1, :] = new_k
4. APPLYING CAUSAL MASK DURING SINGLE-TOKEN INFERENCE:
• Symptom: Tokens generated during cached inference have zero context awareness (attend only to self).
• Cause: Applying an S×S causal mask when input length S = 1!
When S = 1, torch.tril(ones(1, total_len)) sets all historical keys to -inf!
• Remediation: In single-token decoding (s = 1), DO NOT apply causal masking.
TRANSFORMER DECODER ARCHITECTURE SPECIFICATION MATRIX:
┌─────────────────────────┬──────────────────────────────┬──────────────────┬───────────────────────────────┐
│ DECODER COMPONENT │ MATHEMATICAL FORMULATION │ RUNTIME PHASE │ PRIMARY OPERATIONAL ROLE │
├─────────────────────────┼──────────────────────────────┼──────────────────┼───────────────────────────────┤
│ Target Shifted Inputs │ X_input = tokens[:, :-1] │ Training (TF) │ Autoregressive input prefix │
│ Target Shifted Labels │ Y_target = tokens[:, 1:] │ Training (TF) │ Ground-truth supervision │
│ Additive Causal Mask │ M_ij = 0 (j <= i) else -inf │ Training / Prefil│ Enforces temporal causality │
│ Cross-Attention Q │ Q = H_dec · W_Q │ Seq2Seq Decode │ Interrogates encoder memory │
│ Cross-Attention K, V │ K = H_enc · W_K, V = H_enc·WV│ Seq2Seq Decode │ External factual payload │
│ KV-Cache Storage │ K_cache = [K_old; K_new] │ Inference (Gen) │ Eliminates O(N^2) recompute │
│ Temperature Scaling │ Softmax(logits / τ) │ Inference (Gen) │ Calibrates sampling entropy │
│ Nucleus Sampling (Top-P)│ Smallest set where ∑ P >= p │ Inference (Gen) │ Dynamic tail truncation │
│ Prefill Phase │ Process prompt tokens in par.│ Inference Start │ Compute-bound initial cache │
│ Decode Phase │ Step-by-step token generation│ Inference Rollout│ Memory-bandwidth bound loop │
├─────────────────────────┴──────────────────────────────┴──────────────────┴───────────────────────────────┤
│ TOP 5 TRANSFORMER DECODER AXIOMS FOR AI ENGINEERS: │
│ 1. Teacher Forcing Decouples Steps: During training, feeding shifted ground-truth labels allows all │
│ sequence tokens to be processed in a single parallel forward pass with zero serial looping. │
│ 2. Causal Masking Preserves Causality: Additive -inf upper-triangular masking ensures token i cannot │
│ peek at future tokens j > i during parallel training. │
│ 3. KV-Cache Slashes Compute: In autoregressive generation, caching historical Keys and Values transforms │
│ quadratic O(N^2) inference time into linear O(N) operations. │
│ 4. Single-Token Attention is Unmasked: In cached inference where S_q = 1, the new query attends to ALL │
│ historical cached keys without a causal mask. │
│ 5. Shift Labels by One: Always slice inputs as tokens[:, :-1] and targets as tokens[:, 1:] to prevent │
│ the model from learning an identity mapping. │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────┘