π§ Breaking Permutation Equivariance: Restoring Spatio-Temporal Order
The central architectural advantage of the Transformerβits elimination of recurrent temporal state updates in favor of pairwise Scaled Dot-Product Attentionβintroduces a major structural flaw: pure self-attention is strictly permutation equivariant. For any sequence permutation matrix $\mathbf{P} \in \{0, 1\}^{N \times N}$, evaluating self-attention yields $\text{Attn}(\mathbf{P}\mathbf{X}) = \mathbf{P} \, \text{Attn}(\mathbf{X})$. A Transformer with unadorned token embeddings treats the sentence "Dog bites man" as mathematically identical to "Man bites dog". To inject sequence structure, architectures must encode token coordinates into the latent space. The historical progression evolved from Absolute Sinusoidal Encodings (Vaswani et al., 2017) and Learned Absolute Embeddings (BERT, GPT-2) to Relative Positional Biases (Shaw, T5, ALiBi). In modern frontier foundation models (LLaMA-3, Mistral, Qwen, DeepSeek), Rotary Position Embeddings (RoPE) represent the architectural standard: rotating query and key vectors in complex 2D subspaces, naturally encoding relative token distances through inner product invariant rotations.
MATHEMATICAL PROOF: PERMUTATION EQUIVARIANCE OF RAW ATTENTION
GIVEN:
β’ Input sequence matrix: X β R^{N Γ d_model}
β’ Linear projection matrices: W_Q, W_K, W_V β R^{d_model Γ d}
β’ Permutation matrix P β {0, 1}^{N Γ N} (Orthogonal: P^T P = P P^T = I_N)
1. APPLY PERMUTATION TO INPUT:
X_perm = P Β· X
2. DERIVE PROJECTED MATRICES UNDER PERMUTATION:
Q_perm = X_perm Β· W_Q = P Β· (X Β· W_Q) = P Β· Q
K_perm = X_perm Β· W_K = P Β· (X Β· W_K) = P Β· K
V_perm = X_perm Β· W_V = P Β· (X Β· W_V) = P Β· V
3. EVALUATE ATTENTION LOGIT SCORES:
S_perm = ( Q_perm Β· K_perm^T ) / βd
= ( (P Β· Q) Β· (P Β· K)^T ) / βd
= ( P Β· Q Β· K^T Β· P^T ) / βd
= P Β· [ (Q Β· K^T) / βd ] Β· P^T
= P Β· S Β· P^T
4. EVALUATE ROW-WISE SOFTMAX:
Softmax operates strictly along rows. Since P is a permutation matrix:
Softmax( P Β· S Β· P^T ) = P Β· Softmax(S) Β· P^T = P Β· A Β· P^T
5. FINAL ATTENTION OUTPUT UNDER PERMUTATION:
Output_perm = ( P Β· A Β· P^T ) Β· V_perm
= ( P Β· A Β· P^T ) Β· ( P Β· V )
= P Β· A Β· ( P^T Β· P ) Β· V
= P Β· ( A Β· V )
= P Β· Output
CONCLUSION:
Attention( P Β· X ) = P Β· Attention( X )
Without explicit coordinate signals, the Transformer operates as an unordered SET-PROCESSOR!
Permuting input word order simply permutes output vectors with zero syntactic comprehension.
SINUSOIDAL FREQUENCY SPECTRUM ACROSS CHANNEL DIMENSIONS:
Token Position pos β [0, N-1], Channel Dimension Index i β [0, d/2 - 1]
Wavelength Ξ»_i = 2Ο Β· 10000^{2i / d} (Scales geometrically from 2Ο to 20,000Ο)
Low Channels (High Frequency / Rapid Oscillation):
Channel 0 (sin) : ~/\~/\~/\~/\~/\~/\~/\~/\~/\~/\~/\~/\~/\~/\~/\ (Captures local token offsets Ξpos = 1, 2)
Channel 1 (cos) : _/\_/\_/\_/\_/\_/\_/\_/\_/\_/\_/\_/\_/\_/\_/
Mid Channels (Moderate Frequency):
Channel 64 (sin): ββββ±β²ββββ±β²ββββ±β²ββββ±β²ββββ±β²ββββ±β²ββββ±β²ββββ±β²βββ (Captures phrase-level structure)
High Channels (Low Frequency / Extremely Slow Oscillation):
Channel 510(sin): ββββββββββββββββββββββββ±β²βββββββββββββββββββ (Captures global document position)
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
MATHEMATICAL FORMULATION (Vaswani et al., 2017):
For token position pos and channel coordinate 2i (even) and 2i + 1 (odd):
β pos β
PE(pos, 2i) = sin β ββββββββββββββββββββββ β
β 10000^{2i / d_model} β
β pos β
PE(pos, 2i+1) = cos β ββββββββββββββββββββββ β
β 10000^{2i / d_model} β
Where:
pos = Position of the token in the sequence.
i = Dimension index.
d_model = Embedding dimension.
2i = Even dimensions (sine).
2i+1 = Odd dimensions (cosine).
Injected directly via element-wise addition to input token embeddings:
X_conditioned = X_token_embeddings + PE
LEARNED POSITIONAL EMBEDDINGS (BERT, RoBERTa, GPT-2, Original OPT):
Input Sequence: ["Machine", "learning", "scales"]
Token Indices: [ 4120, 8912, 1402 ]
Pos Indices: [ 0, 1, 2 ] βββΊ Hard-coded index range [0, Max_Len - 1]
Token Embedding Table Learned Positional Table
E_token β R^{|V| Γ d_model} E_pos β R^{Max_Len Γ d_model}
βββββββββββββββββββββββββββββββ βββββββββββββββββββββββββββββββ
Row 4120 βββββΊ β 0.12, -0.45, 0.89, ... β Row 0 ββΊ β 0.01, 0.04, -0.02, ... β
Row 8912 βββββΊ β -0.05, 0.82, 0.11, ... β Row 1 ββΊ β 0.03, -0.01, 0.05, ... β
Row 1402 βββββΊ β 0.64, 0.18, -0.31, ... β Row 2 ββΊ β -0.02, 0.06, 0.01, ... β
β ... β β ... β
βββββββββββββββββββββββββββββββ Row 2047βΊβ 0.05, -0.02, 0.08, ... β (HARD LIMIT!)
βββββββββββββββββββββββββββββββ
β
Combined Representation: X = E_token[w_i] + E_pos[i] βββ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
THE EXTRAPOLATION COLLAPSE DEFECT:
If a model trained with Max_Len = 2048 encounters an inference sequence of length 2049:
β’ Index 2048 is Out-Of-Bounds (IndexError / CUDA device assertion)!
β’ The model has ZERO trained weights for position 2048.
β’ Interpolating or resizing the table distorts frequency patterns, causing Perplexity (PPL) β β.
THE RELATIVE ATTENTION PARADIGM:
Attention should depend on relative displacement (i - j), NOT absolute coordinates i and j!
1. SHAW ET AL. (2018) RELATIVE EMBEDDINGS:
Injects learned relative position vectors a_{i, j}^K and a_{i, j}^V into attention logits:
1
e_{i, j} = ββββββββββββ Β· q_i Β· ( k_j + a_{i, j}^K )^T
β(d_head)
Clipping Heuristic: Limits maximum tracked relative distance to k_max (e.g., k_max = 16):
a_{i, j}^K = w_{clip(i - j, -k_max, k_max)}
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
2. T5 RELATIVE POSITIONAL BIASES (Raffel et al., 2020):
Simplifies Shaw's vector formulation into a scalar bias b_{i, j} added directly to raw scores:
1
e_{i, j} = ββββββββββββ Β· ( q_i Β· k_j^T ) + b_{i, j}
β(d_head)
LOGARITHMIC DISTANCE BUCKETING MECHANISM IN T5:
Assigns exact bins for small distances; groups large distances logarithmically:
Relative Offset (i - j): 0 1 2 3 4 5 6 7 8 ... 15 16 ... 31 32 ... 63 64+
Bucket Assignment : 0 1 2 3 4 5 6 7 8 ... 12 13 ... 18 19 ... 24 25 ... 31
ββββββββββ¬βββββββββ βββββββββββββ¬βββββββββ βββββββββββββ¬ββββββββββ
Exact Bins (Small) Log Bins (Medium) Coarse Bins (Long Range)
ATTENTION WITH LINEAR BIASES (ALiBi) MECHANISM:
Input words receive ZERO positional embeddings! Tokens enter model as raw content vectors.
Position is injected directly into attention matrices via static, non-learnable linear penalties:
1
Softmax β ββββββββββββββββββββ Β· ( q_i Β· k_j^T ) - m Β· |i - j| β
β β(d_head) β
WHERE:
β’ |i - j| is the monotonic integer distance between query position i and key position j.
β’ m is a head-specific, non-learnable geometric scalar slope!
GEOMETRIC HEAD SLOPE DISTRIBUTION (For H = 8 Heads):
Slopes are calculated as a geometric sequence: m = 2^{-8/H Β· k} for k = 1, 2, ..., H
Head 1 Slope (m = 1/2) : -0.50 Β· [ 0, 1, 2, 3, 4, ... ] βββΊ Sharp locality; attends nearby tokens
Head 2 Slope (m = 1/4) : -0.25 Β· [ 0, 1, 2, 3, 4, ... ]
Head 3 Slope (m = 1/8) : -0.125Β· [ 0, 1, 2, 3, 4, ... ]
...
Head 8 Slope (m = 1/256) : -0.0039Β· [ 0, 1, 2, 3, 4, ... ] βββΊ Flat slope; attends across long context!
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
THE ZERO-SHOT EXTRAPOLATION BREAKTHROUGH:
Train a model strictly on sequence length 2048 tokens.
At test time, feed sequences of length 8192 tokens:
β’ Learned embeddings: PERPLEXITY EXPLODES (> 1000).
β’ Sinusoidal: PERPLEXITY SPIKES (> 250).
β’ ALiBi: PERPLEXITY REMAINS STABLE AND CONTINUES DECREASING!
THE ROTARY POSITION EMBEDDING (RoPE) MATHEMATICAL DERIVATION:
GOAL: Find a vector function f(x, m) such that the inner product of Query at position m
and Key at position n depends ONLY on their relative distance (m - n):
β¨ f(q, m), f(k, n) β© = g(q, k, m - n)
COMPLEX PLANE FORMULATION:
Represent 2D vector [x_1, x_2]^T as a complex number: x = x_1 + i x_2 β C
To incorporate position m, rotate x in the complex plane by angle (m Β· ΞΈ):
f(x, m) = x Β· e^{i Β· m Β· ΞΈ} = (x_1 + i x_2) Β· (cos(mΞΈ) + i sin(mΞΈ))
EVALUATE INNER PRODUCT BETWEEN ROTATED QUERY AND ROTATED KEY:
In complex space, inner product is: Re( q_m Β· k_n^* ) where * denotes complex conjugate:
β¨ f(q, m), f(k, n) β© = Re [ ( q Β· e^{i Β· m Β· ΞΈ} ) Β· ( k Β· e^{i Β· n Β· ΞΈ} )^* ]
= Re [ q Β· k^* Β· e^{i Β· m Β· ΞΈ} Β· e^{-i Β· n Β· ΞΈ} ]
= Re [ q Β· k^* Β· e^{i Β· (m - n) Β· ΞΈ} ]
THE MAGICAL OUTCOME:
e^{i Β· (m - n) Β· ΞΈ} depends EXCLUSIVELY on relative offset (m - n)!
Absolute coordinates m and n completely cancel out in the exponent!
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
REAL MATRIX EXTENSION ACROSS HIGH-DIMENSIONAL HEADS (d_head dimensions):
Divide d_head into d_head / 2 independent 2D orthogonal rotation subspaces:
R_Ξ,m^d = diag( R_ΞΈ_1,m, R_ΞΈ_2,m, ..., R_ΞΈ_{d/2},m )
Where each 2Γ2 block rotation matrix is:
β β
R_ΞΈ_i,m = β cos(m Β· ΞΈ_i) -sin(m Β· ΞΈ_i) β
β sin(m Β· ΞΈ_i) cos(m Β· ΞΈ_i) β
β β
Base Frequency Sequence: ΞΈ_i = 10000^{-2(i-1) / d_head} for i β [1, d_head / 2]
THE EFFICIENT ELEMENT-WISE ROTATE-HALF COMPUTATION:
Matrix multiplication with block-diagonal matrix R_m is mathematically equivalent to:
R_m Β· x = ( x β cos(m Β· ΞΈ) ) + ( rotate_half(x) β sin(m Β· ΞΈ) )
WHERE:
Let x = [x_1, x_2, x_3, x_4, ..., x_{d-1}, x_d]
rotate_half(x) transforms pairs [x_{2i}, x_{2i+1}] β [-x_{2i+1}, x_{2i}]:
rotate_half(x) = [ -x_2, x_1, -x_4, x_3, ..., -x_d, x_{d-1} ]
Original Tensor x : [ x_1, x_2, x_3, x_4 ]
β cos(mΞΈ) : [ c_1, c_1, c_2, c_2 ]
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Component 1 : [ x_1 c_1, x_2 c_1, x_3 c_2, x_4 c_2 ]
+
rotate_half(x) : [ -x_2, x_1, -x_4, x_3 ]
β sin(mΞΈ) : [ s_1, s_1, s_2, s_2 ]
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Component 2 : [ -x_2 s_1, x_1 s_1, -x_4 s_2, x_3 s_2 ]
========================================================
Sum (Exact RoPE Vector) : [ x_1 c_1 - x_2 s_1, x_2 c_1 + x_1 s_1, ... ]
BENEFIT: Replaces $O(d^2)$ dense matrix multiplication with $O(d)$ vectorized Hadamard products!
import torch
import torch.nn as nn
# ==============================================================================
# PRODUCTION ROTARY POSITION EMBEDDING (RoPE) MODULE
# Used in LLaMA-3, Mistral, Qwen, and DeepSeek architectures
# ==============================================================================
class RotaryPositionalEmbedding(nn.Module):
def __init__(self, dim: int, max_seq_len: int = 4096, base: float = 10000.0):
super().__init__()
assert dim % 2 == 0, f"Embedding dimension {dim} must be even for 2D rotation pairs"
self.dim = dim
self.base = base
self.max_seq_len = max_seq_len
# Precompute frequency inverse powers: ΞΈ_i = base^(-2(i-1)/dim) for i in [0, dim/2 - 1]
inv_freq = 1.0 / (self.base ** (torch.arange(0, dim, 2).float() / dim))
self.register_buffer("inv_freq", inv_freq, persistent=False)
# Precompute cache for max sequence length
self._set_cos_sin_cache(max_seq_len)
def _set_cos_sin_cache(self, seq_len: int):
# 1. Generate position sequence: [0, 1, ..., seq_len - 1]
t = torch.arange(seq_len, device=self.inv_freq.device, dtype=self.inv_freq.dtype)
# 2. Outer product: freqs = pos * inv_freq -> Shape: (seq_len, dim / 2)
freqs = torch.outer(t, self.inv_freq)
# 3. Duplicate frequencies to match full head_dim: (seq_len, dim)
emb = torch.cat((freqs, freqs), dim=-1)
# 4. Cache cos and sin tensors
self.register_buffer("cos_cached", emb.cos(), persistent=False)
self.register_buffer("sin_cached", emb.sin(), persistent=False)
@staticmethod
def _rotate_half(x: torch.Tensor) -> torch.Tensor:
"""Splits tensor in half along last dimension and rotates: [-x2, x1]."""
x1 = x[..., : x.shape[-1] // 2]
x2 = x[..., x.shape[-1] // 2 :]
return torch.cat((-x2, x1), dim=-1)
def forward(self, q: torch.Tensor, k: torch.Tensor, seq_len: int):
"""
q, k: Tensors of shape (Batch, Heads, Seq_Len, Head_Dim)
Returns: Rotated (q_rot, k_rot) with identical shapes
"""
if seq_len > self.max_seq_len:
self._set_cos_sin_cache(seq_len)
# Slice cache to active sequence length: (Seq_Len, Head_Dim) -> (1, 1, Seq_Len, Head_Dim)
cos = self.cos_cached[:seq_len].unsqueeze(0).unsqueeze(0)
sin = self.sin_cached[:seq_len].unsqueeze(0).unsqueeze(0)
# Apply Vectorized RoPE: (x * cos) + (rotate_half(x) * sin)
q_rot = (q * cos) + (self._rotate_half(q) * sin)
k_rot = (k * cos) + (self._rotate_half(k) * sin)
return q_rot, k_rot
# ------------------------------------------------------------------------------
# Runtime Execution & Mathematical Verification Suite
# ------------------------------------------------------------------------------
if __name__ == "__main__":
torch.manual_seed(42)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
B, H, S, D = 2, 4, 8, 64 # Batch=2, Heads=4, SeqLen=8, HeadDim=64
print("[*] Initializing Rotary Positional Embedding (RoPE) Engine...")
rope = RotaryPositionalEmbedding(dim=D, max_seq_len=512, base=10000.0).to(device)
# Instantiate sample query and key tensors
q = torch.randn(B, H, S, D, device=device)
k = torch.randn(B, H, S, D, device=device)
# Apply RoPE
q_rot, k_rot = rope(q, k, seq_len=S)
print(f"[β] RoPE Transformation Success:")
print(f" β’ Rotated Query Shape : {q_rot.shape} (Expected: [{B}, {H}, {S}, {D}])")
print(f" β’ Rotated Key Shape : {k_rot.shape}")
# Mathematical Invariant 1: Vector L2 Norm Preservation
orig_norm = q[0, 0, 0].norm().item()
rot_norm = q_rot[0, 0, 0].norm().item()
print(f" β’ Norm Invariance Test: Original = {orig_norm:.4f} | Rotated = {rot_norm:.4f} (Must match!)")
# Mathematical Invariant 2: Relative Distance Shift Verification
# Shift position of query and key by constant offset delta
delta = 5
q_shifted = torch.zeros_like(q)
k_shifted = torch.zeros_like(k)
q_rot_shift, k_rot_shift = rope(q, k, seq_len=S + delta)
# Compute dot products: vs
dot_orig = torch.dot(q_rot[0, 0, 2], k_rot[0, 0, 1]).item() # Relative offset = 1 (2 - 1)
dot_shift = torch.dot(q_rot_shift[0, 0, 2 + delta], k_rot_shift[0, 0, 1 + delta]).item() # Relative offset = 1
print(f" β’ Relative Shift Test : Dot(pos=2, 1) = {dot_orig:.4f} | Dot(pos=7, 6) = {dot_shift:.4f}")
assert abs(dot_orig - dot_shift) < 1e-4, "RoPE Relative Shift Test Failed!"
print("[β] RoPE Relative Invariance Mathematically Confirmed (Error < 1e-4).")
ROPE CONTEXT LENGTH EXTENSION MECHANICS (e.g. Scaling 4k β 32k or 128k):
1. NAIVE EXTRAPOLATION (Directly querying positions > 4096):
β’ Fails completely! Rotation angles $m \cdot \theta_i$ encounter unseen trigonometric phases.
β’ High-frequency channels accumulate catastrophic out-of-distribution phase shifts.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
2. LINEAR POSITION INTERPOLATION - PI (Chen et al., 2023):
Compresses target context length L' into original pre-training window L via scale factor s = L' / L:
pos' = pos / s
β’ Rotation Angle: (pos / s) Β· ΞΈ_i
β’ Benefit: Guarantees that rotation angles never exceed maximum pre-trained phase bounds!
β’ Drawback: Compresses high-frequency local components, blurring fine-grained nearby token order.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
3. NTK-AWARE ROPE SCALING (Neural Tangent Kernel Interpolation):
Does NOT scale all frequencies equally!
β’ Low frequencies (global position): Scaled via interpolation.
β’ High frequencies (local syntax): Retained unscaled without interpolation!
Scale the base frequency parameter:
base' = base Β· s^{d / (d - 2)} (e.g., LLaMA-3 scales base from 10,000 to 500,000!)
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
4. YaRN (Yet another RoPE extensioN - Peng et al., 2023):
Divides channels into 3 distinct frequency bands using ramp functions:
β’ High Frequencies (r < 0): Zero interpolation (Preserves exact local attention).
β’ Low Frequencies (r > 1): Full linear interpolation.
β’ Medium Frequencies: Smooth linear ramp between the two regimes.
β’ Multiplies attention logits by temperature scale factor to correct Softmax entropy variance.
POSITIONAL ENCODING ARCHITECTURAL SPECIFICATION MATRIX:
βββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ¬βββββββββββββββββββ¬ββββββββββββββββββββββββββββββββ
β ENCODING ARCHITECTURE β MATHEMATICAL FORMULATION β INJECTION POINT β EXTRAPOLATION & LIMITATIONS β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β Sinusoidal Absolute β sin/cos(pos / 10000^{2i/d}) β Additive: X + PE β Moderate extrapolation; fixed β
β (Vaswani et al., 2017) β Geometric frequency spectrum β Input Embedding β trigonometric wavelengths β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β Learned Absolute β Parameter matrix E_pos β Additive: X + E β Zero extrapolation; hard β
β (BERT, GPT-2) β E_pos β R^{Max_Len Γ d} β Input Embedding β sequence length limit Max_Len β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β Relative Vector Bias β q_i Β· (k_j + a_{i-j})^T / βd β Additive in Raw β Good local extrapolation; β
β (Shaw et al., 2018) β Clipped distance embeddings β Attention Scores β high tensor caching overhead β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β Relative Scalar Bias β (q_i Β· k_j^T / βd) + b_{i-j} β Additive in Raw β Moderate extrapolation; β
β (T5 - Raffel et al.) β Logarithmic distance buckets β Attention Scores β logarithmic binning artifact β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β Linear Biases (ALiBi) β (q_i Β· k_j^T / βd) - mΒ·|i-j| β Additive in Raw β Exceptional zero-shot extra- β
β (Press et al., 2021) β Static geometric head slopes β Attention Scores β polation; weak long semantic β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β Rotary Embeddings (RoPE)β (R_m q)^T (R_n k) β Multiplicative: β State-of-the-Art standard; β
β (Su et al., 2021) β Complex 2D plane rotations β Q and K Vectors β extensible to 128k via YaRN/PIβ
βββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ΄βββββββββββββββββββ΄ββββββββββββββββββββββββββββββββ€
β TOP 5 POSITIONAL ENCODING AXIOMS FOR AI ENGINEERS: β
β 1. Self-Attention Has Zero Order: Without positional encodings, Transformers are set processors that β
β treat all input sequence permutations identically. β
β 2. RoPE Preserves Vector Norm: Rotary embeddings rotate vectors without altering their Euclidean norm: β
β ||R_m x||_2 = ||x||_2. β
β 3. Relative Trumps Absolute: Relative distance (m - n) generalizes significantly better than absolute β
β coordinates m and n for long-context generation. β
β 4. Vectorize via Rotate-Half: Implement RoPE using element-wise Hadamard products and rotate_half to avoidβ
β materializing dense block-diagonal rotation matrices in GPU VRAM. β
β 5. Scale Frequencies for Long Context: To extrapolate pre-trained RoPE models to 32k+ tokens, scale β
β the base frequency base' or apply NTK-Aware / YaRN interpolation. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ