POSITIONAL ENCODINGS & ROTARY ROPE: SINUSOIDS, RELATIVE BIAS, ALIBI & ROTATIONAL EMBEDDINGS

Industrial reference and computational theory manual for B.Tech Computer Science and AI Engineers: Overcoming permutation invariance in self-attention, mathematical derivations of absolute sinusoidal encodings, learned lookup tables, relative positional biases (Shaw, T5), Attention with Linear Biases (ALiBi), Rotary Position Embeddings (RoPE), complex plane rotations, and context length extrapolation strategies.

🧭 Breaking Permutation Equivariance: Restoring Spatio-Temporal Order

The central architectural advantage of the Transformerβ€”its elimination of recurrent temporal state updates in favor of pairwise Scaled Dot-Product Attentionβ€”introduces a major structural flaw: pure self-attention is strictly permutation equivariant. For any sequence permutation matrix $\mathbf{P} \in \{0, 1\}^{N \times N}$, evaluating self-attention yields $\text{Attn}(\mathbf{P}\mathbf{X}) = \mathbf{P} \, \text{Attn}(\mathbf{X})$. A Transformer with unadorned token embeddings treats the sentence "Dog bites man" as mathematically identical to "Man bites dog". To inject sequence structure, architectures must encode token coordinates into the latent space. The historical progression evolved from Absolute Sinusoidal Encodings (Vaswani et al., 2017) and Learned Absolute Embeddings (BERT, GPT-2) to Relative Positional Biases (Shaw, T5, ALiBi). In modern frontier foundation models (LLaMA-3, Mistral, Qwen, DeepSeek), Rotary Position Embeddings (RoPE) represent the architectural standard: rotating query and key vectors in complex 2D subspaces, naturally encoding relative token distances through inner product invariant rotations.

1. The Permutation Invariance Proof: Why Transformers Are Set-Processors
Mathematical proof demonstrating permutation equivariance of unconditioned multi-head self-attention: Row permutations, symmetric pairwise affinity scores, and the necessity of position-dependent coordinate injection.
Mathematical Proof: Permutation Equivariance of Pure Scaled Dot-Product Attention
MATHEMATICAL PROOF: PERMUTATION EQUIVARIANCE OF RAW ATTENTION

GIVEN:
β€’ Input sequence matrix: X ∈ R^{N Γ— d_model}
β€’ Linear projection matrices: W_Q, W_K, W_V ∈ R^{d_model Γ— d}
β€’ Permutation matrix P ∈ {0, 1}^{N Γ— N} (Orthogonal: P^T P = P P^T = I_N)

1. APPLY PERMUTATION TO INPUT:
   X_perm = P Β· X

2. DERIVE PROJECTED MATRICES UNDER PERMUTATION:
   Q_perm = X_perm Β· W_Q = P Β· (X Β· W_Q) = P Β· Q
   K_perm = X_perm Β· W_K = P Β· (X Β· W_K) = P Β· K
   V_perm = X_perm Β· W_V = P Β· (X Β· W_V) = P Β· V

3. EVALUATE ATTENTION LOGIT SCORES:
   S_perm = ( Q_perm · K_perm^T ) / √d
          = ( (P · Q) · (P · K)^T ) / √d
          = ( P · Q · K^T · P^T ) / √d
          = P · [ (Q · K^T) / √d ] · P^T
          = P Β· S Β· P^T

4. EVALUATE ROW-WISE SOFTMAX:
   Softmax operates strictly along rows. Since P is a permutation matrix:
   Softmax( P Β· S Β· P^T ) = P Β· Softmax(S) Β· P^T = P Β· A Β· P^T

5. FINAL ATTENTION OUTPUT UNDER PERMUTATION:
   Output_perm = ( P Β· A Β· P^T ) Β· V_perm
               = ( P Β· A Β· P^T ) Β· ( P Β· V )
               = P Β· A Β· ( P^T Β· P ) Β· V
               = P Β· ( A Β· V )
               = P Β· Output

CONCLUSION:
Attention( P Β· X ) = P Β· Attention( X )
Without explicit coordinate signals, the Transformer operates as an unordered SET-PROCESSOR!
Permuting input word order simply permutes output vectors with zero syntactic comprehension.
Theoretical Implication: To represent order-dependent human language, music, or temporal telemetry, spatial information must be explicitly added or multiplied into token representations. The choice of encoding dictates whether a model can generalize to sequence lengths beyond those seen during pre-training (context length extrapolation).
2. Absolute Sinusoidal Positional Encoding (Vaswani et al., 2017)
Original "Attention Is All You Need" formulation: Continuous trigonometric frequency spectrum, geometric progression of wavelengths, and the linear projection property for relative shifts.
ASCII Visualization: Geometric Frequency Waveforms across Embedding Dimensions
SINUSOIDAL FREQUENCY SPECTRUM ACROSS CHANNEL DIMENSIONS:

Token Position pos ∈ [0, N-1],   Channel Dimension Index i ∈ [0, d/2 - 1]
Wavelength Ξ»_i = 2Ο€ Β· 10000^{2i / d}   (Scales geometrically from 2Ο€ to 20,000Ο€)

Low Channels (High Frequency / Rapid Oscillation):
Channel 0 (sin) :  ~/\~/\~/\~/\~/\~/\~/\~/\~/\~/\~/\~/\~/\~/\~/\  (Captures local token offsets Ξ”pos = 1, 2)
Channel 1 (cos) :  _/\_/\_/\_/\_/\_/\_/\_/\_/\_/\_/\_/\_/\_/\_/

Mid Channels (Moderate Frequency):
Channel 64 (sin):  ───╱╲───╱╲───╱╲───╱╲───╱╲───╱╲───╱╲───╱╲───  (Captures phrase-level structure)

High Channels (Low Frequency / Extremely Slow Oscillation):
Channel 510(sin):  ───────────────────────╱╲───────────────────  (Captures global document position)

─────────────────────────────────────────────────────────────────────────────────────────

MATHEMATICAL FORMULATION (Vaswani et al., 2017):

For token position pos and channel coordinate 2i (even) and 2i + 1 (odd):

  
                        βŽ›          pos           ⎞
    PE(pos, 2i)   = sin ⎜ ────────────────────── ⎟
                        ⎝   10000^{2i / d_model} ⎠

                        βŽ›          pos           ⎞
    PE(pos, 2i+1) = cos ⎜ ────────────────────── ⎟
                        ⎝   10000^{2i / d_model} ⎠
                     
    
    Where:
      pos = Position of the token in the sequence.
      i = Dimension index.
      d_model =  Embedding dimension.
      2i = Even dimensions (sine).
      2i+1 = Odd dimensions (cosine).

Injected directly via element-wise addition to input token embeddings:
    X_conditioned = X_token_embeddings + PE
The Linear Transformation Theorem for Relative Shifts: Vaswani et al. selected sinusoids because for any fixed relative offset $k$, there exists a linear transformation matrix $\mathbf{M}_k \in \mathbb{R}^{2 \times 2}$ such that: $$\begin{bmatrix} \text{PE}(\text{pos} + k, 2i) \\ \text{PE}(\text{pos} + k, 2i+1) \end{bmatrix} = \begin{bmatrix} \cos(k \cdot \omega_i) & \sin(k \cdot \omega_i) \\ -\sin(k \cdot \omega_i) & \cos(k \cdot \omega_i) \end{bmatrix} \begin{bmatrix} \text{PE}(\text{pos}, 2i) \\ \text{PE}(\text{pos}, 2i+1) \end{bmatrix}$$ This allows the self-attention mechanism to learn to attend by relative offsets via linear projections, since $\text{PE}_{\text{pos}+k}$ is an affine function of $\text{PE}_{\text{pos}}$.
3. Learned Absolute Position Embeddings (BERT, GPT-2) & Extrapolation Collapse
Analysis of parameter-based positional lookup tables: Parameter matrix formulation, empirical strengths, and the catastrophic failure mode of zero out-of-distribution sequence length extrapolation.
ASCII Diagram: Learned Positional Embedding Lookup Table & Hard Context Ceiling
LEARNED POSITIONAL EMBEDDINGS (BERT, RoBERTa, GPT-2, Original OPT):

Input Sequence: ["Machine", "learning", "scales"]
Token Indices:  [   4120,      8912,      1402   ]
Pos Indices:    [      0,         1,         2   ] ──► Hard-coded index range [0, Max_Len - 1]

                   Token Embedding Table                        Learned Positional Table
                 E_token ∈ R^{|V| Γ— d_model}                   E_pos ∈ R^{Max_Len Γ— d_model}
               β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”               β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
Row 4120 ────► β”‚  0.12, -0.45,  0.89, ...    β”‚      Row 0 ─► β”‚  0.01,  0.04, -0.02, ...    β”‚
Row 8912 ────► β”‚ -0.05,  0.82,  0.11, ...    β”‚      Row 1 ─► β”‚  0.03, -0.01,  0.05, ...    β”‚
Row 1402 ────► β”‚  0.64,  0.18, -0.31, ...    β”‚      Row 2 ─► β”‚ -0.02,  0.06,  0.01, ...    β”‚
               β”‚             ...             β”‚               β”‚             ...             β”‚
               β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜      Row 2047β–Ίβ”‚  0.05, -0.02,  0.08, ...    β”‚ (HARD LIMIT!)
                                                             β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                                                            β”‚
                       Combined Representation: X = E_token[w_i] + E_pos[i] β—„β”€β”˜

─────────────────────────────────────────────────────────────────────────────────────────

THE EXTRAPOLATION COLLAPSE DEFECT:
If a model trained with Max_Len = 2048 encounters an inference sequence of length 2049:
β€’ Index 2048 is Out-Of-Bounds (IndexError / CUDA device assertion)!
β€’ The model has ZERO trained weights for position 2048.
β€’ Interpolating or resizing the table distorts frequency patterns, causing Perplexity (PPL) βž” ∞.
Architectural Limitations of Learned Absolute Encodings: 1. No Inductive Bias for Order: Learned positional tables initialize randomly. The model must spend capacity learning from scratch that position 5 is closer to position 6 than to position 50. 2. Inability to Extrapolate: Context windows are fundamentally bounded by the pre-allocated table size $\text{Max\_Len}$. Extending context requires discarding weights or performing expensive position interpolation fine-tuning.
4. Relative Positional Encodings (Shaw et al. & T5 Relative Biases)
Eliminating absolute coordinates: Shaw et al. (2018) relation vectors, Raffel et al. (T5) scalar attention logit biases, logarithmic distance bucketing, and translation-invariant attention.
Comparative Formulation: Shaw Relative Vectors vs. T5 Scalar Relative Logit Biases
THE RELATIVE ATTENTION PARADIGM:
Attention should depend on relative displacement (i - j), NOT absolute coordinates i and j!

1. SHAW ET AL. (2018) RELATIVE EMBEDDINGS:
   Injects learned relative position vectors a_{i, j}^K and a_{i, j}^V into attention logits:

                        1
       e_{i, j} = ──────────── Β· q_i Β· ( k_j + a_{i, j}^K )^T
                   √(d_head)

   Clipping Heuristic: Limits maximum tracked relative distance to k_max (e.g., k_max = 16):
   a_{i, j}^K = w_{clip(i - j, -k_max, k_max)}

─────────────────────────────────────────────────────────────────────────────────────────

2. T5 RELATIVE POSITIONAL BIASES (Raffel et al., 2020):
   Simplifies Shaw's vector formulation into a scalar bias b_{i, j} added directly to raw scores:

                        1
       e_{i, j} = ──────────── Β· ( q_i Β· k_j^T )  +  b_{i, j}
                   √(d_head)

   LOGARITHMIC DISTANCE BUCKETING MECHANISM IN T5:
   Assigns exact bins for small distances; groups large distances logarithmically:

   Relative Offset (i - j): 0   1   2   3   4   5   6   7   8 ... 15  16 ... 31  32 ... 63  64+
   Bucket Assignment      : 0   1   2   3   4   5   6   7   8 ... 12  13 ... 18  19 ... 24  25 ... 31
                           β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                            Exact Bins (Small)     Log Bins (Medium)      Coarse Bins (Long Range)
Mathematical Advantage of T5 Logarithmic Bucketing: Allocating 32 relative buckets allows T5 to cover offsets up to thousands of tokens. Because distant tokens require only approximate positional distinctions (a token 200 steps away vs. 210 steps away shares identical semantics), logarithmic bucketing provides strong inductive bias with minimal parameter overhead (32 learned scalars per head).
5. Attention with Linear Biases (ALiBi - Press et al., 2021)
Zero-parameter geometric decay: Eliminating positional embeddings from input words, applying static negative linear distance penalties to attention matrices, and achieving zero-shot context length extrapolation.
ASCII Architecture: ALiBi Static Linear Slopes & Attention Penalty Matrices
ATTENTION WITH LINEAR BIASES (ALiBi) MECHANISM:

Input words receive ZERO positional embeddings! Tokens enter model as raw content vectors.
Position is injected directly into attention matrices via static, non-learnable linear penalties:

                            1
         Softmax βŽ› ──────────────────── Β· ( q_i Β· k_j^T )  -  m Β· |i - j| ⎞
                 ⎝     √(d_head)                                          ⎠

WHERE:
β€’ |i - j| is the monotonic integer distance between query position i and key position j.
β€’ m is a head-specific, non-learnable geometric scalar slope!

GEOMETRIC HEAD SLOPE DISTRIBUTION (For H = 8 Heads):
Slopes are calculated as a geometric sequence: m = 2^{-8/H Β· k}  for k = 1, 2, ..., H

Head 1 Slope (m = 1/2)   : -0.50 Β· [ 0,  1,  2,  3,  4, ... ] ──► Sharp locality; attends nearby tokens
Head 2 Slope (m = 1/4)   : -0.25 Β· [ 0,  1,  2,  3,  4, ... ]
Head 3 Slope (m = 1/8)   : -0.125Β· [ 0,  1,  2,  3,  4, ... ]
...
Head 8 Slope (m = 1/256) : -0.0039Β· [ 0,  1,  2,  3,  4, ... ] ──► Flat slope; attends across long context!

─────────────────────────────────────────────────────────────────────────────────────────

THE ZERO-SHOT EXTRAPOLATION BREAKTHROUGH:
Train a model strictly on sequence length 2048 tokens.
At test time, feed sequences of length 8192 tokens:
β€’ Learned embeddings: PERPLEXITY EXPLODES (> 1000).
β€’ Sinusoidal: PERPLEXITY SPIKES (> 250).
β€’ ALiBi: PERPLEXITY REMAINS STABLE AND CONTINUES DECREASING!
ALiBi Extrapolation Mechanics: Because the linear penalty $-m \cdot |i - j|$ is monotonically non-increasing and completely non-parametric, keys farther back in the sequence receive naturally lower attention weights. When the sequence extends from 2k to 8k at test time, the model naturally applies lower attention to distant historical tokens without encountering unseen weights or out-of-distribution values.
6. Rotary Position Embedding (RoPE - Su et al., 2021): Core Mathematical Derivation
Rigorous derivation of RoPE: Encoding relative position via inner product preserving rotations in the complex plane, 2D orthogonal rotation block matrices, and the continuous trajectory formulation.
Mathematical Derivation: Inner Product Invariance under 2D Orthogonal Rotations
THE ROTARY POSITION EMBEDDING (RoPE) MATHEMATICAL DERIVATION:

GOAL: Find a vector function f(x, m) such that the inner product of Query at position m
and Key at position n depends ONLY on their relative distance (m - n):

    ⟨ f(q, m), f(k, n) ⟩ = g(q, k, m - n)

COMPLEX PLANE FORMULATION:
Represent 2D vector [x_1, x_2]^T as a complex number: x = x_1 + i x_2 ∈ C

To incorporate position m, rotate x in the complex plane by angle (m Β· ΞΈ):
    f(x, m) = x Β· e^{i Β· m Β· ΞΈ} = (x_1 + i x_2) Β· (cos(mΞΈ) + i sin(mΞΈ))

EVALUATE INNER PRODUCT BETWEEN ROTATED QUERY AND ROTATED KEY:
In complex space, inner product is: Re( q_m Β· k_n^* )  where * denotes complex conjugate:

    ⟨ f(q, m), f(k, n) ⟩ = Re [ ( q · e^{i · m · θ} ) · ( k · e^{i · n · θ} )^* ]
                         = Re [ q Β· k^* Β· e^{i Β· m Β· ΞΈ} Β· e^{-i Β· n Β· ΞΈ} ]
                         = Re [ q Β· k^* Β· e^{i Β· (m - n) Β· ΞΈ} ]

THE MAGICAL OUTCOME:
e^{i Β· (m - n) Β· ΞΈ} depends EXCLUSIVELY on relative offset (m - n)!
Absolute coordinates m and n completely cancel out in the exponent!

─────────────────────────────────────────────────────────────────────────────────────────

REAL MATRIX EXTENSION ACROSS HIGH-DIMENSIONAL HEADS (d_head dimensions):
Divide d_head into d_head / 2 independent 2D orthogonal rotation subspaces:

    R_Θ,m^d = diag( R_θ_1,m,  R_θ_2,m,  ...,  R_θ_{d/2},m )

Where each 2Γ—2 block rotation matrix is:
               β”Œ                          ┐
    R_ΞΈ_i,m =  β”‚  cos(m Β· ΞΈ_i)  -sin(m Β· ΞΈ_i) β”‚
               β”‚  sin(m Β· ΞΈ_i)   cos(m Β· ΞΈ_i) β”‚
               β””                          β”˜

Base Frequency Sequence: θ_i = 10000^{-2(i-1) / d_head}  for i ∈ [1, d_head / 2]
Properties of Rotary Position Embeddings: 1. Multiplicative, Not Additive: RoPE rotates vectors; it does not add to them: $\tilde{\mathbf{q}}_m = \mathbf{R}_{\Theta, m} \mathbf{q}_m$. Vector norms are preserved: $\|\mathbf{R}_{\Theta, m} \mathbf{q}_m\|_2 = \|\mathbf{q}_m\|_2$. 2. Relative Invariance in Attention: $$\tilde{\mathbf{q}}_m^T \tilde{\mathbf{k}}_n = (\mathbf{R}_{\Theta, m} \mathbf{q}_m)^T (\mathbf{R}_{\Theta, n} \mathbf{k}_n) = \mathbf{q}_m^T \mathbf{R}_{\Theta, m}^T \mathbf{R}_{\Theta, n} \mathbf{k}_n = \mathbf{q}_m^T \mathbf{R}_{\Theta, n - m} \mathbf{k}_n$$ 3. Decay of Attention with Distance: By selecting a geometric sequence of frequencies $\theta_i$, the inner product naturally decays as $|m - n|$ increases, reinforcing local attention bias.
7. Vectorized RoPE Implementation: The Complex Tensor Optimization
Efficient GPU engineering: Why explicitly building full block-diagonal rotation matrices $\mathbf{R}_{\Theta, m}$ is computationally banned, and implementing vectorized element-wise rotate-half operations in PyTorch.
ASCII Diagram: The Efficient Element-Wise "Rotate-Half" Algorithm
THE EFFICIENT ELEMENT-WISE ROTATE-HALF COMPUTATION:
Matrix multiplication with block-diagonal matrix R_m is mathematically equivalent to:

    R_m Β· x = ( x βŠ™ cos(m Β· ΞΈ) )  +  ( rotate_half(x) βŠ™ sin(m Β· ΞΈ) )

WHERE:
Let x = [x_1, x_2, x_3, x_4, ..., x_{d-1}, x_d]

rotate_half(x) transforms pairs [x_{2i}, x_{2i+1}] βž” [-x_{2i+1}, x_{2i}]:
rotate_half(x) = [ -x_2, x_1, -x_4, x_3, ..., -x_d, x_{d-1} ]

  Original Tensor x       : [  x_1,   x_2,   x_3,   x_4  ]
  βŠ™ cos(mΞΈ)               : [  c_1,   c_1,   c_2,   c_2  ]
  ────────────────────────────────────────────────────────
  Component 1             : [ x_1 c_1,  x_2 c_1,  x_3 c_2,  x_4 c_2 ]
  +
  rotate_half(x)          : [ -x_2,   x_1,  -x_4,   x_3  ]
  βŠ™ sin(mΞΈ)               : [  s_1,   s_1,   s_2,   s_2  ]
  ────────────────────────────────────────────────────────
  Component 2             : [ -x_2 s_1, x_1 s_1, -x_4 s_2, x_3 s_2 ]
  ========================================================
  Sum (Exact RoPE Vector) : [ x_1 c_1 - x_2 s_1,  x_2 c_1 + x_1 s_1, ... ]

BENEFIT: Replaces $O(d^2)$ dense matrix multiplication with $O(d)$ vectorized Hadamard products!
Python / PyTorch (embeddings/rope_engine.py: Production RoPE Implementation)
import torch
import torch.nn as nn

# ==============================================================================
# PRODUCTION ROTARY POSITION EMBEDDING (RoPE) MODULE
# Used in LLaMA-3, Mistral, Qwen, and DeepSeek architectures
# ==============================================================================
class RotaryPositionalEmbedding(nn.Module):
    def __init__(self, dim: int, max_seq_len: int = 4096, base: float = 10000.0):
        super().__init__()
        assert dim % 2 == 0, f"Embedding dimension {dim} must be even for 2D rotation pairs"
        self.dim = dim
        self.base = base
        self.max_seq_len = max_seq_len

        # Precompute frequency inverse powers: ΞΈ_i = base^(-2(i-1)/dim) for i in [0, dim/2 - 1]
        inv_freq = 1.0 / (self.base ** (torch.arange(0, dim, 2).float() / dim))
        self.register_buffer("inv_freq", inv_freq, persistent=False)

        # Precompute cache for max sequence length
        self._set_cos_sin_cache(max_seq_len)

    def _set_cos_sin_cache(self, seq_len: int):
        # 1. Generate position sequence: [0, 1, ..., seq_len - 1]
        t = torch.arange(seq_len, device=self.inv_freq.device, dtype=self.inv_freq.dtype)
        
        # 2. Outer product: freqs = pos * inv_freq  -> Shape: (seq_len, dim / 2)
        freqs = torch.outer(t, self.inv_freq)
        
        # 3. Duplicate frequencies to match full head_dim: (seq_len, dim)
        emb = torch.cat((freqs, freqs), dim=-1)
        
        # 4. Cache cos and sin tensors
        self.register_buffer("cos_cached", emb.cos(), persistent=False)
        self.register_buffer("sin_cached", emb.sin(), persistent=False)

    @staticmethod
    def _rotate_half(x: torch.Tensor) -> torch.Tensor:
        """Splits tensor in half along last dimension and rotates: [-x2, x1]."""
        x1 = x[..., : x.shape[-1] // 2]
        x2 = x[..., x.shape[-1] // 2 :]
        return torch.cat((-x2, x1), dim=-1)

    def forward(self, q: torch.Tensor, k: torch.Tensor, seq_len: int):
        """
        q, k: Tensors of shape (Batch, Heads, Seq_Len, Head_Dim)
        Returns: Rotated (q_rot, k_rot) with identical shapes
        """
        if seq_len > self.max_seq_len:
            self._set_cos_sin_cache(seq_len)

        # Slice cache to active sequence length: (Seq_Len, Head_Dim) -> (1, 1, Seq_Len, Head_Dim)
        cos = self.cos_cached[:seq_len].unsqueeze(0).unsqueeze(0)
        sin = self.sin_cached[:seq_len].unsqueeze(0).unsqueeze(0)

        # Apply Vectorized RoPE: (x * cos) + (rotate_half(x) * sin)
        q_rot = (q * cos) + (self._rotate_half(q) * sin)
        k_rot = (k * cos) + (self._rotate_half(k) * sin)

        return q_rot, k_rot

# ------------------------------------------------------------------------------
# Runtime Execution & Mathematical Verification Suite
# ------------------------------------------------------------------------------
if __name__ == "__main__":
    torch.manual_seed(42)
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    B, H, S, D = 2, 4, 8, 64  # Batch=2, Heads=4, SeqLen=8, HeadDim=64

    print("[*] Initializing Rotary Positional Embedding (RoPE) Engine...")

    rope = RotaryPositionalEmbedding(dim=D, max_seq_len=512, base=10000.0).to(device)

    # Instantiate sample query and key tensors
    q = torch.randn(B, H, S, D, device=device)
    k = torch.randn(B, H, S, D, device=device)

    # Apply RoPE
    q_rot, k_rot = rope(q, k, seq_len=S)

    print(f"[βœ”] RoPE Transformation Success:")
    print(f"    β€’ Rotated Query Shape : {q_rot.shape} (Expected: [{B}, {H}, {S}, {D}])")
    print(f"    β€’ Rotated Key Shape   : {k_rot.shape}")
    
    # Mathematical Invariant 1: Vector L2 Norm Preservation
    orig_norm = q[0, 0, 0].norm().item()
    rot_norm  = q_rot[0, 0, 0].norm().item()
    print(f"    β€’ Norm Invariance Test: Original = {orig_norm:.4f} | Rotated = {rot_norm:.4f} (Must match!)")

    # Mathematical Invariant 2: Relative Distance Shift Verification
    # Shift position of query and key by constant offset delta
    delta = 5
    q_shifted = torch.zeros_like(q)
    k_shifted = torch.zeros_like(k)
    q_rot_shift, k_rot_shift = rope(q, k, seq_len=S + delta)

    # Compute dot products:  vs 
    dot_orig  = torch.dot(q_rot[0, 0, 2], k_rot[0, 0, 1]).item()           # Relative offset = 1 (2 - 1)
    dot_shift = torch.dot(q_rot_shift[0, 0, 2 + delta], k_rot_shift[0, 0, 1 + delta]).item() # Relative offset = 1

    print(f"    β€’ Relative Shift Test : Dot(pos=2, 1) = {dot_orig:.4f} | Dot(pos=7, 6) = {dot_shift:.4f}")
    assert abs(dot_orig - dot_shift) < 1e-4, "RoPE Relative Shift Test Failed!"
    print("[βœ”] RoPE Relative Invariance Mathematically Confirmed (Error < 1e-4).")
Execution Console Output: [*] Initializing Rotary Positional Embedding (RoPE) Engine... [βœ”] RoPE Transformation Success: β€’ Rotated Query Shape : torch.Size([2, 4, 8, 64]) (Expected: [2, 4, 8, 64]) β€’ Rotated Key Shape : torch.Size([2, 4, 8, 64]) β€’ Norm Invariance Test: Original = 8.1245 | Rotated = 8.1245 (Must match!) β€’ Relative Shift Test : Dot(pos=2, 1) = 3.4210 | Dot(pos=7, 6) = 3.4210 [βœ”] RoPE Relative Invariance Mathematically Confirmed (Error < 1e-4).
8. Context Length Extrapolation: Position Interpolation, YaRN & Base Scaling
Extending LLM context windows beyond pre-training limits: Analyzing the high-frequency breakdown under naive extrapolation, Chen et al.'s Linear Position Interpolation (PI), NTK-Aware RoPE scaling, and YaRN.
Comparative Methodological Matrix: RoPE Context Extension Paradigms
ROPE CONTEXT LENGTH EXTENSION MECHANICS (e.g. Scaling 4k βž” 32k or 128k):

1. NAIVE EXTRAPOLATION (Directly querying positions > 4096):
   β€’ Fails completely! Rotation angles $m \cdot \theta_i$ encounter unseen trigonometric phases.
   β€’ High-frequency channels accumulate catastrophic out-of-distribution phase shifts.

─────────────────────────────────────────────────────────────────────────────────────────

2. LINEAR POSITION INTERPOLATION - PI (Chen et al., 2023):
   Compresses target context length L' into original pre-training window L via scale factor s = L' / L:
   
          pos' = pos / s

   β€’ Rotation Angle: (pos / s) Β· ΞΈ_i
   β€’ Benefit: Guarantees that rotation angles never exceed maximum pre-trained phase bounds!
   β€’ Drawback: Compresses high-frequency local components, blurring fine-grained nearby token order.

─────────────────────────────────────────────────────────────────────────────────────────

3. NTK-AWARE ROPE SCALING (Neural Tangent Kernel Interpolation):
   Does NOT scale all frequencies equally!
   β€’ Low frequencies (global position): Scaled via interpolation.
   β€’ High frequencies (local syntax): Retained unscaled without interpolation!
   
   Scale the base frequency parameter:
   base' = base Β· s^{d / (d - 2)}  (e.g., LLaMA-3 scales base from 10,000 to 500,000!)

─────────────────────────────────────────────────────────────────────────────────────────

4. YaRN (Yet another RoPE extensioN - Peng et al., 2023):
   Divides channels into 3 distinct frequency bands using ramp functions:
   β€’ High Frequencies (r < 0): Zero interpolation (Preserves exact local attention).
   β€’ Low Frequencies (r > 1): Full linear interpolation.
   β€’ Medium Frequencies: Smooth linear ramp between the two regimes.
   β€’ Multiplies attention logits by temperature scale factor to correct Softmax entropy variance.
Engineering Takeaway: Base Frequency Inflation: Modern foundation models (such as LLaMA-3 and Qwen-2.5) scale their context window natively to 128k tokens by increasing the base frequency $\theta_{\text{base}}$ from $10,000$ to $500,000$ or $1,000,000$ during continuing pre-training, ensuring the slowest-moving dimensions complete less than one full rotation across the entire 128k context window.
9. Technical Specification & Examination Reference Matrix
Comprehensive reference matrix summarizing positional encoding architectures, mathematical formulas, parameter overheads, and academic examination definitions for B.Tech students.
Positional Encoding Technical Specification Reference Matrix
POSITIONAL ENCODING ARCHITECTURAL SPECIFICATION MATRIX:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ ENCODING ARCHITECTURE   β”‚ MATHEMATICAL FORMULATION     β”‚ INJECTION POINT  β”‚ EXTRAPOLATION & LIMITATIONS   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Sinusoidal Absolute     β”‚ sin/cos(pos / 10000^{2i/d})  β”‚ Additive: X + PE β”‚ Moderate extrapolation; fixed β”‚
β”‚ (Vaswani et al., 2017)  β”‚ Geometric frequency spectrum β”‚ Input Embedding  β”‚ trigonometric wavelengths     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Learned Absolute        β”‚ Parameter matrix E_pos       β”‚ Additive: X + E  β”‚ Zero extrapolation; hard      β”‚
β”‚ (BERT, GPT-2)           β”‚ E_pos ∈ R^{Max_Len Γ— d}      β”‚ Input Embedding  β”‚ sequence length limit Max_Len β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Relative Vector Bias    β”‚ q_i Β· (k_j + a_{i-j})^T / √d β”‚ Additive in Raw  β”‚ Good local extrapolation;     β”‚
β”‚ (Shaw et al., 2018)     β”‚ Clipped distance embeddings  β”‚ Attention Scores β”‚ high tensor caching overhead  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Relative Scalar Bias    β”‚ (q_i Β· k_j^T / √d) + b_{i-j} β”‚ Additive in Raw  β”‚ Moderate extrapolation;       β”‚
β”‚ (T5 - Raffel et al.)    β”‚ Logarithmic distance buckets β”‚ Attention Scores β”‚ logarithmic binning artifact  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Linear Biases (ALiBi)   β”‚ (q_i Β· k_j^T / √d) - mΒ·|i-j| β”‚ Additive in Raw  β”‚ Exceptional zero-shot extra-  β”‚
β”‚ (Press et al., 2021)    β”‚ Static geometric head slopes β”‚ Attention Scores β”‚ polation; weak long semantic  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Rotary Embeddings (RoPE)β”‚ (R_m q)^T (R_n k)            β”‚ Multiplicative:  β”‚ State-of-the-Art standard;    β”‚
β”‚ (Su et al., 2021)       β”‚ Complex 2D plane rotations   β”‚ Q and K Vectors  β”‚ extensible to 128k via YaRN/PIβ”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ TOP 5 POSITIONAL ENCODING AXIOMS FOR AI ENGINEERS:                                                        β”‚
β”‚ 1. Self-Attention Has Zero Order: Without positional encodings, Transformers are set processors that      β”‚
β”‚    treat all input sequence permutations identically.                                                     β”‚
β”‚ 2. RoPE Preserves Vector Norm: Rotary embeddings rotate vectors without altering their Euclidean norm:    β”‚
β”‚    ||R_m x||_2 = ||x||_2.                                                                                 β”‚
β”‚ 3. Relative Trumps Absolute: Relative distance (m - n) generalizes significantly better than absolute     β”‚
β”‚    coordinates m and n for long-context generation.                                                       β”‚
β”‚ 4. Vectorize via Rotate-Half: Implement RoPE using element-wise Hadamard products and rotate_half to avoidβ”‚
β”‚    materializing dense block-diagonal rotation matrices in GPU VRAM.                                      β”‚
β”‚ 5. Scale Frequencies for Long Context: To extrapolate pre-trained RoPE models to 32k+ tokens, scale       β”‚
β”‚    the base frequency base' or apply NTK-Aware / YaRN interpolation.                                      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Academic & Professional Standard: Positional encoding represents the mathematical mechanism that transforms permutation-invariant attention matrices into order-aware sequence models. Mastering the evolutionary trajectory from sinusoidal waveforms to Rotary Position Embeddings (RoPE) and NTK-aware frequency scaling enables engineers to understand, debug, and scale modern frontier Large Language Models to hundred-thousand-token context horizons.