THE TRANSFORMER DECODER: CAUSAL MASKING, CROSS-ATTENTION & KV-CACHING

Industrial reference and architectural engineering manual for B.Tech Computer Science and AI Engineers: Deep inspection of the Transformer Decoder sub-layer topology, autoregressive generation dynamics, lower-triangular causal masking mechanics, cross-attention memory bridge, the prefill vs. decode runtime duality, KV-cache tensor memory allocation, and production-grade PyTorch implementations.

⚡ The Autoregressive Generative Core: From Teacher Forcing to Real-Time Inference

While the Transformer Encoder builds bidirectional representations by examining all tokens simultaneously, the Transformer Decoder is the engine of open-ended generation. Operating under the probability chain rule, the decoder formulates sequence generation as autoregressive prediction: estimating the categorical distribution $P(x_t \mid x_1, \dots, x_{t-1})$. Modern frontier models (GPT-4, LLaMA-3, Mistral, Qwen) are almost exclusively Decoder-Only architectures, discarding the encoder entirely. To train across full sequences in parallel on GPU tensor cores without temporal information leakage, the decoder implements a strict mathematical constraint: Causal Masked Self-Attention. During training, all positions are updated concurrently via Teacher Forcing; during inference, the model transitions into a token-by-token loop where intermediate key and value activations are cached via KV-Caching, transforming an intractable $O(N^2)$ inference recomputation into an optimized $O(N)$ memory-bound operational regime.

1. Architecture of the Transformer Decoder Layer: Sub-Layer Topography
Examining the complete decoder sub-layer stack: Differentiating classical Encoder-Decoder decoders (with cross-attention) from modern Causal Decoder-Only architectures (LLaMA/GPT style).
ASCII Architecture: Classical Seq2Seq Decoder vs. Modern Causal Decoder-Only
CLASSICAL ENCODER-DECODER DECODER (T5 / Whisper):       MODERN CAUSAL DECODER-ONLY (GPT / LLaMA):
Contains 3 Sub-Layers with Cross-Attention               Contains 2 Sub-Layers (Encoder Completely Removed)

Input Targets: Y_{l-1} ∈ R^{B × S_tgt × d_model}         Input Sequence: X_{l-1} ∈ R^{B × S × d_model}
               │                                                        │
               ├─────────────────────────┐ (Residual 1)                 ├─────────────────────────┐ (Residual 1)
               ▼                         │                              ▼                         │
      ┌─────────────────┐                │                     ┌─────────────────┐                │
      │ LayerNorm / RMS │                │                     │ LayerNorm / RMS │                │
      └────────┬────────┘                │                     └────────┬────────┘                │
               ▼                         │                              ▼                         │
      ┌─────────────────┐                │                     ┌─────────────────┐                │
      │ Causal Masked   │                │                     │ Causal Masked   │                │
      │ Self-Attention  │                │                     │ Self-Attention  │                │
      └────────┬────────┘                │                     └────────┬────────┘                │
               ▼                         │                              ▼                         │
      ┌─────────────────┐                │                     ┌─────────────────┐                │
      │ Element-wise (+)│ ◄──────────────┘                     │ Element-wise (+)│ ◄──────────────┘
      └────────┬────────┘                                      └────────┬────────┘
               │                                                        │
               ├─────────────────────────┐ (Residual 2)                 ├─────────────────────────┐ (Residual 2)
               ▼                         │                              ▼                         │
      ┌─────────────────┐                │                     ┌─────────────────┐                │
      │ LayerNorm / RMS │                │                     │ LayerNorm / RMS │                │
      └────────┬────────┘                │                     └────────┬────────┘                │
               ▼                         │                              ▼                         │
      ┌─────────────────┐                │                     ┌─────────────────┐                │
      │ Cross-Attention │ ◄── Encoder    │                     │ Gated FFN       │                │
      │ (K, V from Enc) │     Memory     │                     │ (SwiGLU / MLP)  │                │
      └────────┬────────┘                │                     └────────┬────────┘                │
               ▼                         │                              ▼                         │
      ┌─────────────────┐                │                     ┌─────────────────┐                │
      │ Element-wise (+)│ ◄──────────────┘                     │ Element-wise (+)│ ◄──────────────┘
      └────────┬────────┘                                      └────────┬────────┘
               │                                                        │
               ├─────────────────────────┐ (Residual 3)                 ▼
               ▼                         │                     Output: X_l ∈ R^{B × S × d_model}
      ┌─────────────────┐                │
      │ Feed-Forward Net│                │
      └────────┬────────┘                │
               ▼                         │
      ┌─────────────────┐                │
      │ Element-wise (+)│ ◄──────────────┘
      └────────┬────────┘
               │
               ▼ Output: Y_l ∈ R^{B × S_tgt × d_model}
The Decoder-Only Paradigm Shift: Pre-training an encoder-decoder model forces a strict division between understanding inputs and generating outputs. In contrast, Decoder-Only models unify both understanding and generation: the prompt and the completion inhabit the same sequence space. By applying a causal mask across both, the model seamlessly transitions from ingesting instructions to emitting novel completions with zero architectural context handover.
2. Causal Masking: The Lower-Triangular Mathematical Guarantee
Mathematical derivation of autoregressive masking: Why unmasked attention violates causal consistency, constructing the additive $-\infty$ upper-triangular matrix, and proving zero information leakage from future tokens.
ASCII Analysis: The Causal Mask Tensor Transformation & Softmax Zeroing
THE CAUSAL MASKING DATA TRANSFORMATION PIPELINE:

Sequence Tokens: ["The", "neural", "network", "generates"]  (Sequence Length S = 4)

1. RAW ATTENTION DOT-PRODUCT MATRIX: S = (Q · K^T) / √d_k  ∈ R^{4 × 4}
   Every token position calculates affinity against EVERY other position:
                 "The"    "neural"  "network"  "generates"
   "The"       ┌  3.2       1.8       0.9         2.1     ┐  <-- Token 1 CANNOT see tokens 2, 3, 4!
   "neural"    │  1.5       4.1       2.2         0.4     │  <-- Token 2 CANNOT see tokens 3, 4!
   "network"   │  0.8       2.0       3.9         1.7     │  <-- Token 3 CANNOT see token 4!
   "generates" └  1.1       0.7       2.8         4.5     ┘

─────────────────────────────────────────────────────────────────────────────────────────

2. ADDITIVE CAUSAL MASK MATRIX M_causal ∈ R^{4 × 4}:
   Upper-triangular components (j > i) are strictly set to -inf:
   M_{i, j} = 0 if j <= i else -inf

               ┌    0       -inf      -inf       -inf     ┐
   M_causal =  │    0         0       -inf       -inf     │
               │    0         0         0        -inf     │
               └    0         0         0          0      ┘

─────────────────────────────────────────────────────────────────────────────────────────

3. SUMMATION AND ROW-WISE SOFTMAX:
   S_masked = S + M_causal

   Applying Softmax: A_{i, j} = exp(S_masked_{i, j}) / ∑_m exp(S_masked_{i, m})
   Because exp(-inf) = 0.0, future positions collapse strictly to zero probability mass!

   Final Lower-Triangular Attention Probability Matrix A ∈ R^{4 × 4}:
                 "The"    "neural"  "network"  "generates"
   "The"       ┌  1.00      0.00      0.00       0.00     ┐  (Row sum = 1.0)
   "neural"    │  0.07      0.93      0.00       0.00     │  (Row sum = 1.0)
   "network"   │  0.03      0.12      0.85       0.00     │  (Row sum = 1.0)
   "generates" └  0.02      0.02      0.14       0.82     ┘  (Row sum = 1.0)
Mathematical Guarantee of Causal Masking: Let the output contextual vector at position $i$ be $\mathbf{o}_i$: $$\mathbf{o}_i = \sum_{j=1}^S A_{i, j} \mathbf{v}_j = \sum_{j=1}^i A_{i, j} \mathbf{v}_j + \sum_{j=i+1}^S 0 \cdot \mathbf{v}_j = \sum_{j=1}^i A_{i, j} \mathbf{v}_j$$ Because $A_{i, j} = 0$ for all $j > i$, the representation $\mathbf{o}_i$ is strictly a function of past and current inputs $\mathbf{x}_{\le i}$, mathematically preventing any backward gradient flow or information leakage from future positions $j > i$.
3. Cross-Attention: Bridging Encoder Representations into Decoder Streams
Analysis of the cross-attention sub-layer in sequence-to-sequence models: Routing queries from target tokens while extracting keys and values from source encoder representations.
ASCII Diagram: Cross-Attention Multi-Modal Query-Key-Value Routing
CROSS-ATTENTION INFORMATION ROUTING (Seq2Seq / T5 / Diffusion Models):

Decoder Hidden State H_dec ∈ R^{B × S_tgt × d_model}     Encoder Memory H_enc ∈ R^{B × S_src × d_model}
               │                                                          │
               ▼ Linear W_Q                                               ├────────────────────────┐
   Queries Q ∈ R^{B × S_tgt × d_k}                                        ▼ Linear W_K             ▼ Linear W_V
               │                                              Keys K ∈ R^{B × S_src × d_k}  Values V ∈ R^{B × S_src × d_v}
               │                                                          │                        │
               └───────────────────────────┬──────────────────────────────┘                        │
                                           │ Matrix Multiply: Q · K^T                              │
                                           ▼                                                       │
                           Raw Affinity S ∈ R^{B × S_tgt × S_src}                                  │
                                           │                                                       │
                                           ▼ Scale (1 / √d_k) + Source Padding Mask                │
                                           ▼ Softmax along S_src dimension                         │
                           Cross-Attention Weights A ∈ R^{B × S_tgt × S_src}                       │
                                           │                                                       │
                                           │ Matrix Multiply: A · V                                │
                                           └──────────────────────────────┬────────────────────────┘
                                                                          │
                                                                          ▼
                                                    Contextual Output ∈ R^{B × S_tgt × d_v}
Cross-Attention Characteristics: 1. Dimensional Decoupling: The target sequence length $S_{\text{tgt}}$ and source sequence length $S_{\text{src}}$ are completely independent. A 10-token prompt can cross-attend over a 2,048-token document. 2. Unmasked Key Axis: In cross-attention, NO causal mask is applied across the Key axis! The decoder is permitted to look at all source encoder tokens (both past and future) simultaneously to produce the next output token.
4. The Operational Duality: Training (Parallel) vs. Inference (Sequential)
Deconstructing the dual modes of generative execution: How Teacher Forcing enables parallel $O(1)$ GPU matrix multiplies during training, and why inference is strictly serial $O(N)$ autoregression.
Comparative Matrix: Training Mode vs. Inference Generation Mode
THE DUAL EXECUTION MODES OF TRANSFORMER DECODERS:
┌─────────────────────────┬──────────────────────────────────────┬────────────────────────────────────────┐
│ OPERATIONAL PROPERTY    │ TRAINING PHASE (TEACHER FORCING)     │ INFERENCE GENERATION PHASE             │
├─────────────────────────┼──────────────────────────────────────┼────────────────────────────────────────┤
│ Sequence Feeding        │ Full sequence fed simultaneously     │ One token generated per step           │
│ Compute Parallelism     │ Fully parallel across all S tokens   │ Strictly sequential $O(S)$ serial loop │
│ Attention Matrix        │ Materializes full lower-triangular   │ Computes single row vector at step t   │
│ Target Tokens Input     │ Ground-truth target tokens y^*_{0..S}│ Model's own previous predicted tokens  │
│ Hardware Bottleneck     │ Compute-Bound (FLOPs bound)          │ Memory Bandwidth Bound (VRAM I/O)      │
│ Wall-Clock Complexity   │ $O(1)$ GPU forward kernel dispatches │ $O(N)$ sequential GPU kernel launches  │
│ KV-Cache Utilization    │ Banned (Redundant; full seq in VRAM) │ Mandatory (Eliminates $O(N^2)$ waste)  │
└─────────────────────────┴──────────────────────────────────────┴────────────────────────────────────────┘
The Autoregressive Loop Mechanics: At inference time: 1. Step 1: Input $[x_0]$, emit logits for $x_1$. Sample $x_1 \sim P(x \mid x_0)$. 2. Step 2: Input $[x_0, x_1]$, emit logits for $x_2$. Sample $x_2 \sim P(x \mid x_0, x_1)$. 3. Step $t$: Input $[x_0, \dots, x_{t-1}]$, emit logits for $x_t$. Notice that at step $t$, the tokens $x_0, \dots, x_{t-2}$ are re-processed from scratch if no caching mechanism is implemented!
5. The KV-Cache Mechanism: Slicing Inference from $O(N^2)$ to $O(N)$
Analyzing KV-Caching: Why previous token Query projections are discarded, saving historical Key and Value representations in GPU memory, and mathematical proof of compute acceleration.
ASCII Diagram: Naive Recomputation vs. KV-Cache State Ingestion
WITHOUT KV-CACHE (NAIVE INFERENCE - O(N^2) COMPUTATION):
At step t = 4, the model re-projects and re-calculates all keys and values for tokens 1, 2, 3:

Step 1: Compute Q_1, K_1, V_1 ──► Emit Token 2
Step 2: Compute Q_{1..2}, K_{1..2}, V_{1..2} (Re-computes token 1!) ──► Emit Token 3
Step 3: Compute Q_{1..3}, K_{1..3}, V_{1..3} (Re-computes tokens 1, 2!) ──► Emit Token 4
Total FLOPs scale as: ∑_{t=1}^N t = N(N + 1) / 2  ➔ O(N^2) redundant matrix multiplications!

─────────────────────────────────────────────────────────────────────────────────────────

WITH KV-CACHE (OPTIMAL INFERENCE - O(N) COMPUTATION):
Keys and Values for historical tokens 1..3 are PERMANENTLY STORED in GPU memory.
Only the SINGLE NEW TOKEN (Token 4) is projected!

Step 4 Input: Only token x_4! (Sequence Length S = 1)
1. Project NEW Query:   q_4 ∈ R^{1 × d_k}
2. Project NEW Key:     k_4 ∈ R^{1 × d_k}
3. Project NEW Value:   v_4 ∈ R^{1 × d_v}

4. UPDATE KV-CACHE IN GPU VRAM:
   K_cached = Concat( K_cached_{1..3},  k_4 ) ∈ R^{4 × d_k}
   V_cached = Concat( V_cached_{1..3},  v_4 ) ∈ R^{4 × d_v}

5. COMPUTE SINGLE-ROW ATTENTION VECTOR:
   Scores = ( q_4 · K_cached^T ) / √d_k  ∈ R^{1 × 4}
   Weights = Softmax( Scores )            ∈ R^{1 × 4}
   Context = Weights · V_cached          ∈ R^{1 × d_v}
Total FLOPs per step: Strictly O(1) linear projections! Total generation FLOPs: O(N).
KV-Cache Memory Footprint Formula: While KV-caching slashes compute time from quadratic to linear, it introduces a massive VRAM capacity footprint. For a model with $L$ layers, hidden dimension $d$, batch size $B$, and sequence length $S$ in FP16 (2 bytes per float): $$\text{Memory}_{\text{KV-Cache}} = 2 \times 2 \times B \times S \times L \times d_{\text{model}} \quad \text{Bytes}$$ For LLaMA-3-70B ($L=80, d=8192$) with $B=16$ users and context $S=8192$: $$\text{Memory} = 4 \times 16 \times 8192 \times 80 \times 8192 \approx 343.6 \text{ Gigabytes of VRAM!}$$ This memory bottleneck catalyzed Grouped-Query Attention (GQA) and vLLM PagedAttention algorithms.
6. Complete Production PyTorch Implementation: Decoder Layer & KV-Cache Engine
Complete PyTorch reference module: Causal masked self-attention, cross-attention, SwiGLU FFN, KV-cache tensor allocation, and verifying equivalence between single-step cached rollouts and full-sequence teacher forcing.
Python / PyTorch (transformer/decoder_kv_cache.py: Production Decoder with Dynamic Cache)
import math
import torch
import torch.nn as nn
import torch.nn.functional as F

# ==============================================================================
# 1. CAUSAL SELF-ATTENTION WITH DYNAMIC KV-CACHE
# ==============================================================================
class CausalSelfAttentionWithCache(nn.Module):
    def __init__(self, d_model: int, num_heads: int):
        super().__init__()
        assert d_model % num_heads == 0
        self.d_model = d_model
        self.num_heads = num_heads
        self.head_dim = d_model // num_heads
        self.scale = 1.0 / math.sqrt(self.head_dim)

        self.q_proj = nn.Linear(d_model, d_model, bias=False)
        self.k_proj = nn.Linear(d_model, d_model, bias=False)
        self.v_proj = nn.Linear(d_model, d_model, bias=False)
        self.out_proj = nn.Linear(d_model, d_model, bias=False)

    def forward(self, x: torch.Tensor, kv_cache: dict = None, use_cache: bool = False):
        """
        x: Input tensor of shape (Batch, SeqLen, d_model)
        kv_cache: Dictionary storing historical {'k': tensor, 'v': tensor}
        use_cache: Boolean flag activating incremental single-token decoding
        """
        b, s, _ = x.shape

        # 1. Project current token representations
        q = self.q_proj(x).view(b, s, self.num_heads, self.head_dim).transpose(1, 2)
        k = self.k_proj(x).view(b, s, self.num_heads, self.head_dim).transpose(1, 2)
        v = self.v_proj(x).view(b, s, self.num_heads, self.head_dim).transpose(1, 2)

        # 2. Ingest / Concatenate historical Key and Value representations from KV-Cache
        if kv_cache is not None and use_cache:
            if 'k' in kv_cache and kv_cache['k'] is not None:
                # Concatenate along sequence dimension (dim=2)
                k = torch.cat([kv_cache['k'], k], dim=2)
                v = torch.cat([kv_cache['v'], v], dim=2)
            kv_cache['k'] = k
            kv_cache['v'] = v

        total_seq_len = k.size(2)

        # 3. Compute Attention Affinity Scores: (B, H, S_q, S_k)
        scores = torch.matmul(q, k.transpose(-2, -1)) * self.scale

        # 4. Apply Causal Mask ONLY if evaluating multi-token sequences (s > 1)
        # In single-token cached decoding (s = 1), new token can attend to ALL historical tokens!
        if s > 1:
            causal_mask = torch.tril(torch.ones(s, total_seq_len, device=x.device, dtype=torch.bool))
            scores = scores.masked_fill(~causal_mask.unsqueeze(0).unsqueeze(0), float('-inf'))

        # 5. Softmax normalization & Value aggregation
        attn_weights = F.softmax(scores, dim=-1)
        context = torch.matmul(attn_weights, v) # (B, H, S_q, head_dim)

        # 6. Recombine heads and project out
        context = context.transpose(1, 2).contiguous().view(b, s, self.d_model)
        output = self.out_proj(context)

        return output, kv_cache

# ==============================================================================
# 2. COMPLETE TRANSFORMER DECODER LAYER (CAUSAL DECODER-ONLY)
# ==============================================================================
class TransformerDecoderLayer(nn.Module):
    def __init__(self, d_model: int, num_heads: int, d_ff: int):
        super().__init__()
        self.norm1 = nn.LayerNorm(d_model)
        self.attn  = CausalSelfAttentionWithCache(d_model, num_heads)
        self.norm2 = nn.LayerNorm(d_model)
        
        # SwiGLU Gated Feed-Forward Network
        self.w_gate = nn.Linear(d_model, d_ff, bias=False)
        self.w_up   = nn.Linear(d_model, d_ff, bias=False)
        self.w_down = nn.Linear(d_ff, d_model, bias=False)

    def forward(self, x: torch.Tensor, kv_cache: dict = None, use_cache: bool = False):
        # Sub-layer 1: Causal Self-Attention + Residual
        norm_x = self.norm1(x)
        attn_out, kv_cache = self.attn(norm_x, kv_cache=kv_cache, use_cache=use_cache)
        x = x + attn_out

        # Sub-layer 2: SwiGLU FFN + Residual
        norm_x = self.norm2(x)
        ffn_out = self.w_down(F.silu(self.w_gate(norm_x)) * self.w_up(norm_x))
        x = x + ffn_out

        return x, kv_cache

# ------------------------------------------------------------------------------
# Runtime Execution & Numerical Equivalence Verification Suite
# ------------------------------------------------------------------------------
if __name__ == "__main__":
    torch.manual_seed(42)
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    B, S, D, H, D_ff = 1, 5, 64, 4, 128

    print("[*] Initializing Transformer Decoder KV-Cache Verification Engine...")
    decoder_layer = TransformerDecoderLayer(d_model=D, num_heads=H, d_ff=D_ff).to(device)
    decoder_layer.eval()

    # Synthetic Input Sequence of 5 tokens
    input_tokens = torch.randn(B, S, D, device=device)

    # 1. PARALLEL FORWARD PASS (TRAINING MODE - NO CACHE, FULL CAUSAL MASK)
    with torch.no_grad():
        full_out, _ = decoder_layer(input_tokens, use_cache=False)
    print(f"[✔] Parallel Full Sequence Pass Computed. Shape: {full_out.shape}")

    # 2. INCREMENTAL CACHED GENERATION (INFERENCE MODE - STEP-BY-STEP)
    print("[*] Running Incremental KV-Cached Inference Loop...")
    cache_store = {}
    cached_outputs = []

    with torch.no_grad():
        for t in range(S):
            # Feed exactly ONE token at a time: shape (1, 1, D)
            current_token = input_tokens[:, t:t+1, :]
            out_step, cache_store = decoder_layer(current_token, kv_cache=cache_store, use_cache=True)
            cached_outputs.append(out_step)

    # Concatenate all single-step outputs along sequence dimension
    incremental_out = torch.cat(cached_outputs, dim=1)
    print(f"[✔] Incremental Cached Pass Concluded. Shape: {incremental_out.shape}")

    # 3. MATHEMATICAL EQUIVALENCE CHECK
    # Check that parallel training pass matches step-by-step cached inference exactly!
    max_error = torch.max(torch.abs(full_out - incremental_out)).item()
    print(f"[✔] Maximum Absolute Discrepancy: {max_error:.2e}")
    assert max_error < 1e-5, "Mathematical Equivalence Failure between Caching and Parallel Pass!"
    print("[✔] MATHEMATICAL EQUIVALENCE VERIFIED: KV-Cache produces bit-exact representations.")
Execution Console Output: [*] Initializing Transformer Decoder KV-Cache Verification Engine... [✔] Parallel Full Sequence Pass Computed. Shape: torch.Size([1, 5, 64]) [*] Running Incremental KV-Cached Inference Loop... [✔] Incremental Cached Pass Concluded. Shape: torch.Size([1, 5, 64]) [✔] Maximum Absolute Discrepancy: 4.77e-07 [✔] MATHEMATICAL EQUIVALENCE VERIFIED: KV-Cache produces bit-exact representations.
7. Token Generation Sampling: Temperature, Top-K & Top-P (Nucleus)
Transforming raw decoder output logits into natural language: Mathematical formulations of Softmax Temperature scaling, Top-K truncation, Top-P (Nucleus) cumulative thresholding, and Min-P sampling.
ASCII Analysis: Sampling Mechanics across Temperature and Probability Nucleus
PROBABILITY LOGIT SAMPLING TRANSFORMATIONS:

Decoder Output Logits: z ∈ R^{|V|} over Vocabulary V

1. TEMPERATURE SCALING: P(x_i) = exp(z_i / τ) / ∑_j exp(z_j / τ)
   • τ = 1.0 : Standard unchanged model calibration.
   • τ ➔ 0.0 : Greedy Argmax (Deterministic, repetitive, zero creativity).
   • τ > 1.0 : Flattened distribution (Higher entropy, creative, risks hallucinations).

─────────────────────────────────────────────────────────────────────────────────────────

2. TOP-K TRUNCATION (Fan et al., 2018):
   Sort logits descendingly. Retain ONLY the top K candidates; set remaining logits to -inf.
   Example (K = 3):
   Raw Candidates : ["cat" (0.50), "dog" (0.30), "rabbit" (0.15), "car" (0.04), "tree" (0.01)]
   Top-K (K=3)    : ["cat", "dog", "rabbit"]  (Discard "car", "tree")
   Flaw: Fixed K cannot adapt to flat vs. peaked confidence distributions.

─────────────────────────────────────────────────────────────────────────────────────────

3. TOP-P / NUCLEUS SAMPLING (Holtzman et al., 2019):
   Dynamically selects the smallest subset of tokens whose cumulative probability exceeds P:

                     ∑_{i ∈ V^(p)} P(x_i) >= P   (typically P = 0.90)

   • Peaked Distribution (High Confidence):
     Candidate 1 has P = 0.92 ➔ Nucleus contains ONLY 1 word!
   • Flat Distribution (High Uncertainty):
     50 words have P = 0.02 each ➔ Nucleus expands dynamically to contain 45 words!
   Enforces context-dependent dynamic truncation, eliminating bizarre long-tail tokens.
Repetition Penalty Formulation: To prevent degenerative looping common in greedy sampling, models apply a repetition penalty $\theta$ (Keskar et al., 2019): $$z_i \leftarrow \begin{cases} z_i / \theta & \text{if } z_i > 0 \\ z_i \cdot \theta & \text{if } z_i \le 0 \end{cases} \quad \text{for all tokens } i \text{ present in generated context}$$ Where setting $\theta \in [1.1, 1.3]$ penalizes already-emitted tokens by diminishing their output logits, preventing infinite loops.
8. Common Antipatterns & Computational Traps in Transformer Decoders
Diagnostic playbook covering critical decoder bugs: Off-by-one label shift errors in Teacher Forcing, causal mask leaking through future positions, and KV-cache dimension desynchronization.
Troubleshooting Playbook: Decoder Anomalies & Engineering Remediations
TRANSFORMER DECODER ANOMALIES & REMEDIATIONS:

1. THE OFF-BY-ONE LABEL SHIFT TRAP IN TEACHER FORCING:
   • Symptom: Training loss drops to 0.0 in 2 steps, but the model produces complete gibberish at inference!
   • Theoretical Cause: Feeding the targets as their own labels without shifting:
     loss = criterion(logits, targets)  # FATAL: Model learns identity mapping: x_t predicts x_t!
   • Correct Mathematical Slicing:
     inputs  = tokens[:, :-1]   # Feed tokens 0 to S-1
     targets = tokens[:, 1:]    # Predict tokens 1 to S
     loss = criterion(model(inputs), targets)

2. UNINTENTIONAL CAUSAL MASK INVERSION:
   • Symptom: Decoder tokens attend ONLY to future tokens; generation degenerates immediately.
   • Cause: Using torch.triu() instead of torch.tril() when building boolean attention masks!
     torch.triu selects the UPPER triangle (future tokens), allowing the model to cheat during training.
   • Correct Code:
     causal_mask = torch.tril(torch.ones(S, S, dtype=torch.bool))

3. KV-CACHE MEMORY FRAGMENTATION IN PYTHON LISTS:
   • Symptom: GPU Out of Memory after serving 100 requests due to VRAM memory fragmentation.
   • Cause: Appending tensors dynamically using 'torch.cat()' inside the decoding loop!
     Re-allocating a new tensor every single token step causes thousands of tiny memory fragments.
   • Remediation: Pre-allocate static tensor buffers for max_seq_len ahead of time:
     k_cache = torch.empty(B, H, max_seq_len, head_dim, device=device)
     And maintain a write pointer index: k_cache[:, :, step:step+1, :] = new_k

4. APPLYING CAUSAL MASK DURING SINGLE-TOKEN INFERENCE:
   • Symptom: Tokens generated during cached inference have zero context awareness (attend only to self).
   • Cause: Applying an S×S causal mask when input length S = 1!
     When S = 1, torch.tril(ones(1, total_len)) sets all historical keys to -inf!
   • Remediation: In single-token decoding (s = 1), DO NOT apply causal masking.
Diagnostic Verification Tool: To verify that causal masking contains zero future information leakage, compute the Jacobian of the output with respect to future inputs: $$\frac{\partial \mathbf{o}_i}{\partial \mathbf{x}_j} = \mathbf{0} \quad \text{for all } j > i$$ If any gradient backpropagates from output position $i$ to input position $j > i$, causal masking is breached.
9. Technical Specification & Examination Reference Matrix
Comprehensive reference matrix summarizing Transformer Decoder components, mathematical formulations, runtime complexities, and academic examination definitions for B.Tech students.
Transformer Decoder Architecture Technical Specification Matrix
TRANSFORMER DECODER ARCHITECTURE SPECIFICATION MATRIX:
┌─────────────────────────┬──────────────────────────────┬──────────────────┬───────────────────────────────┐
│ DECODER COMPONENT       │ MATHEMATICAL FORMULATION     │ RUNTIME PHASE    │ PRIMARY OPERATIONAL ROLE      │
├─────────────────────────┼──────────────────────────────┼──────────────────┼───────────────────────────────┤
│ Target Shifted Inputs   │ X_input = tokens[:, :-1]     │ Training (TF)    │ Autoregressive input prefix   │
│ Target Shifted Labels   │ Y_target = tokens[:, 1:]     │ Training (TF)    │ Ground-truth supervision      │
│ Additive Causal Mask    │ M_ij = 0 (j <= i) else -inf  │ Training / Prefil│ Enforces temporal causality   │
│ Cross-Attention Q       │ Q = H_dec · W_Q              │ Seq2Seq Decode   │ Interrogates encoder memory   │
│ Cross-Attention K, V    │ K = H_enc · W_K, V = H_enc·WV│ Seq2Seq Decode   │ External factual payload      │
│ KV-Cache Storage        │ K_cache = [K_old; K_new]     │ Inference (Gen)  │ Eliminates O(N^2) recompute   │
│ Temperature Scaling     │ Softmax(logits / τ)          │ Inference (Gen)  │ Calibrates sampling entropy   │
│ Nucleus Sampling (Top-P)│ Smallest set where ∑ P >= p  │ Inference (Gen)  │ Dynamic tail truncation       │
│ Prefill Phase           │ Process prompt tokens in par.│ Inference Start  │ Compute-bound initial cache   │
│ Decode Phase            │ Step-by-step token generation│ Inference Rollout│ Memory-bandwidth bound loop   │
├─────────────────────────┴──────────────────────────────┴──────────────────┴───────────────────────────────┤
│ TOP 5 TRANSFORMER DECODER AXIOMS FOR AI ENGINEERS:                                                        │
│ 1. Teacher Forcing Decouples Steps: During training, feeding shifted ground-truth labels allows all       │
│    sequence tokens to be processed in a single parallel forward pass with zero serial looping.            │
│ 2. Causal Masking Preserves Causality: Additive -inf upper-triangular masking ensures token i cannot      │
│    peek at future tokens j > i during parallel training.                                                  │
│ 3. KV-Cache Slashes Compute: In autoregressive generation, caching historical Keys and Values transforms  │
│    quadratic O(N^2) inference time into linear O(N) operations.                                           │
│ 4. Single-Token Attention is Unmasked: In cached inference where S_q = 1, the new query attends to ALL    │
│    historical cached keys without a causal mask.                                                          │
│ 5. Shift Labels by One: Always slice inputs as tokens[:, :-1] and targets as tokens[:, 1:] to prevent     │
│    the model from learning an identity mapping.                                                           │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Academic & Professional Standard: The Transformer Decoder represents the computational foundation of modern generative foundation models. Mastering the mathematical mechanics of lower-triangular causal masking, the transition between parallel Teacher Forcing and serial inference, KV-cache tensor allocation, and logit sampling dynamics provides the core engineering expertise required to build, optimize, and serve large-scale generative AI platforms.