AUTOENCODING PRE-TRAINING: BERT, ROBERTA, DEBERTA & MASKED RECONSTRUCTION

Foundational technical reference and mathematical architecture manual for B.Tech Computer Science, Data Science, and AI Engineers: Deep bidirectional context encoding, the 80/10/10 Masked Language Modeling (MLM) corrupted objective, Next Sentence Prediction (NSP) vs. Sentence-Order Prediction (SOP), RoBERTa empirical optimization, DeBERTa disentangled attention, and complete PyTorch implementations.

πŸ” The Denoising Autoencoding Paradigm: Learning Bidirectional Context

Standard autoregressive language models (like GPT) operate unidirectionally, factorizing sequence probability as $\prod_{t=1}^T P(x_t \mid x_{BERT (Bidirectional Encoder Representations from Transformers), shifting pre-training toward a Denoising Autoencoding objective. Rather than predicting the next sequential token, BERT masks a random percentage of input tokens and trains the network to reconstruct the corrupted elements from full left-to-right and right-to-left context simultaneously. Subsequent advancesβ€”most notably RoBERTa (Liu et al., 2019) through hyperparameter optimization and dynamic masking, and DeBERTa (He et al., 2021) via disentangled content-position attentionβ€”solidified autoencoding encoders as the standard foundation for semantic classification, extractive question answering, and modern dense vector retrieval engines.

1. BERT Architectural Topography & Input Representation Triplet
Examining the unified input embedding structure of BERT: Token Embeddings, Segment (Token-Type) Embeddings, and Learned Positional Embeddings, along with structural special tokens ([CLS], [SEP], [MASK]).
ASCII Architecture: BERT Three-Way Input Embedding Summation
THE BERT MULTI-EMBEDDING INGESTION HIGHWAY:

Input Sentence A: "The database failed"    Input Sentence B: "Query stalled"

Raw Tokens:       [CLS]     The    database   failed    [SEP]     Query    stalled    [SEP]
                    β”‚        β”‚        β”‚         β”‚         β”‚         β”‚         β”‚         β”‚
                    β–Ό        β–Ό        β–Ό         β–Ό         β–Ό         β–Ό         β–Ό         β–Ό
Token Embeddings: [ E_CLS  E_The   E_data    E_fail    E_SEP     E_Query  E_stall   E_SEP ]   ∈ R^{d_model}
                    +        +        +         +         +         +         +         +
Segment Embeddings:[ E_A     E_A      E_A       E_A       E_A       E_B       E_B       E_B  ]   ∈ R^{d_model}
(Sentence A vs B)   +        +        +         +         +         +         +         +
Position Embeddings:[ E_0     E_1      E_2       E_3       E_4       E_5       E_6       E_7  ]   ∈ R^{d_model}
(Learned absolute)  β”‚        β”‚        β”‚         β”‚         β”‚         β”‚         β”‚         β”‚
                    β–Ό        β–Ό        β–Ό         β–Ό         β–Ό         β–Ό         β–Ό         β–Ό
COMBINED INPUT:   [ X_0      X_1      X_2       X_3       X_4       X_5       X_6       X_7  ]   ∈ R^{B Γ— S Γ— d_model}
                    β”‚
                    β–Ό
          β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
          β”‚     L LayerNorm(X + Dropout)      β”‚
          β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                            β”‚
                            β–Ό Fed directly into Bidirectional Transformer Encoder Stack!

SPECIAL TOKEN DIRECTIVE ROLES:
β€’ [CLS] (Index 101 in BERT): Prepended at index 0. Its final contextual vector h_[CLS] aggregates
  whole-sequence semantics for downstream classification (GLUE benchmark / classification heads).
β€’ [SEP] (Index 102 in BERT): Delimiter token placed between paired input sequences (e.g., Question & Passage)
  and terminating the final sequence boundary.
β€’ [MASK] (Index 103 in BERT): Masked placeholder token injected during pre-training corruption.
Embedding Summation Mechanics: Let $x_i$ be the $i$-th token index and $s_i \in \{0, 1\}$ be its segment indicator: $$\mathbf{e}_i = \text{LayerNorm}\Big(\mathbf{W}_{\text{token}}[x_i] + \mathbf{W}_{\text{segment}}[s_i] + \mathbf{W}_{\text{pos}}[i]\Big)$$ All three matrices map to an identical hidden dimension $d_{\text{model}}$ (768 in BERT-Base, 1024 in BERT-Large). The vectors are summed element-wise rather than concatenated, maintaining dimensional parity without expanding parameter count.
2. The Masked Language Modeling (MLM) Pre-training Objective
Mathematical derivation of the Cloze task: Why bidirectional attention prevents standard autoregressive training, the 15% corruption budget, the 80/10/10 replacement rule, and cross-entropy reconstruction loss.
ASCII Flowchart: The 80 / 10 / 10 Corruption Breakdown Pipeline
THE 15% TOKEN CORRUPTION PIPELINE (THE 80 / 10 / 10 HEURISTIC):

Input Sentence: "The quick brown fox jumps over the lazy dog"
1. Uniform Random Selection: Select exactly 15% of all non-special tokens for prediction.
   Selected Target Token at Position 4: "fox"

2. CORRUPTION PROBABILITY BREAKDOWN OVER SELECTED 15%:

   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
   β”‚ 80% of Selected Tokens (Probability = 0.80):           β”‚
   β”‚ Replace token with explicit special placeholder [MASK] β”‚ ──► "The quick brown [MASK] jumps..."
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
   β”‚ 10% of Selected Tokens (Probability = 0.10):           β”‚
   β”‚ Replace token with an arbitrary random vocabulary word β”‚ ──► "The quick brown [apple] jumps..."
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
   β”‚ 10% of Selected Tokens (Probability = 0.10):           β”‚
   β”‚ Keep token completely UNCHANGED (Ground-Truth literal) β”‚ ──► "The quick brown [fox] jumps..."
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

WHY THE 80/10/10 CORRUPTION RULE IS MANDATORY:
β€’ The Problem: The [MASK] token NEVER appears during downstream fine-tuning or inference!
  If the model only learned representations for [MASK], it would produce degraded embeddings
  for unmasked real tokens during fine-tuning (Discrepancy mismatch).
β€’ The Solution:
  - 80% [MASK] forces the model to learn bidirectional contextual reconstruction.
  - 10% Random replacement forces the model to maintain context-aware suspicion of every token,
    learning to identify errors and prevent local bias.
  - 10% Identity retention biases the model's representation towards the true observed token.
Mathematical Objective Formulation: Let $\mathbf{x} = (x_1, \dots, x_N)$ be the sequence, and $\mathcal{M} \subset \{1, \dots, N\}$ be the set of masked indices ($|\mathcal{M}| \approx 0.15 \cdot N$). The corrupted sequence is $\tilde{\mathbf{x}}$. The Masked Language Model loss minimizes the negative log-likelihood of the original masked tokens conditioned on the corrupted context: $$\mathcal{L}_{\text{MLM}}(\theta) = -\sum_{i \in \mathcal{M}} \log P(x_i \mid \tilde{\mathbf{x}}; \theta) = -\sum_{i \in \mathcal{M}} \log \left( \frac{\exp(\mathbf{w}_{x_i}^T \mathbf{h}_i + b_{x_i})}{\sum_{v=1}^{|\mathcal{V}|} \exp(\mathbf{w}_v^T \mathbf{h}_i + b_v)} \right)$$ Notice that loss gradients backpropagate **strictly from the 15% masked positions**; uncorrupted tokens do not generate cross-entropy loss signals, though their representations are updated via attention paths.
3. Next Sentence Prediction (NSP) vs. Sentence-Order Prediction (SOP)
Analysis of cross-sentence pre-training tasks: The original BERT Next Sentence Prediction (NSP) binary classification task, why it proved ineffective, and ALBERT's Sentence-Order Prediction (SOP) remedy.
Comparative Matrix: Next Sentence Prediction vs. Sentence-Order Prediction
CROSS-SENTENCE OBJECTIVE COMPARISON:

1. BERT NEXT SENTENCE PREDICTION (NSP - Devlin et al., 2018):
   Binary classification task applied to the final contextual vector of [CLS]:
   h_[CLS] ──► W_nsp Β· h_[CLS] ──► Binary Softmax: {IsNext = 1, NotNext = 0}

   Training Data Pairing:
   β€’ 50% Positive Examples (IsNext): Sentence A is followed immediately by Sentence B in the raw document.
   β€’ 50% Negative Examples (NotNext): Sentence A is paired with an arbitrary Sentence B picked randomly
     from an entirely different document!

   THE FATAL FLAW OF NSP (Discovered by Lample & Conneau; Liu et al., 2019):
   β€’ NSP is too trivial! Random negative pairs come from completely different topics.
   β€’ The model does NOT learn inter-sentence discourse or coherence; it simply learns TOPIC PREDICTION!
     Example: Sentence A about "Quantum Physics" paired with Sentence B about "Recipe for Pasta".
     The model classifies NotNext based solely on vocabulary topic mismatch.

─────────────────────────────────────────────────────────────────────────────────────────

2. ALBERT SENTENCE-ORDER PREDICTION (SOP - Lan et al., 2019):
   Replaces topic matching with genuine structural coherence analysis.

   Training Data Pairing:
   β€’ Both sentences A and B are ALWAYS extracted consecutively from the SAME document!
   β€’ 50% Positive Examples: [Sentence A] followed by [Sentence B] (Correct chronological order).
   β€’ 50% Negative Examples: [Sentence B] followed by [Sentence A] (INVERTED chronological order!).

   OUTCOME: The model CANNOT rely on topic mismatch to cheat. It is forced to understand
   discourse markers ("furthermore", "however"), causal flow, and chronological logic.
The Total Loss Function of Canonical BERT: $$\mathcal{L}_{\text{BERT}}(\theta) = \mathcal{L}_{\text{MLM}}(\theta) + \mathcal{L}_{\text{NSP}}(\theta)$$ Because NSP acts as a secondary regularizer, training balances token-level masked reconstruction with whole-document binary classification. However, later ablations revealed that removing $\mathcal{L}_{\text{NSP}}$ entirely often improves downstream language modeling performance.
4. RoBERTa: A Robustly Optimized BERT Pre-training Approach
Liu et al. (2019) empirical breakthroughs: Demonstrating that original BERT was severely undertrained, removing the NSP objective, dynamic masking schedules, scaling batch sizes to 8,000, and Byte-Level BPE.
ASCII Analysis: Static Masking (BERT) vs. Dynamic Masking (RoBERTa) across Epochs
STATIC vs. DYNAMIC MASKING ARCHITECTURAL MECHANICS:

1. BERT STATIC MASKING (Pre-computed during data preprocessing):
   The dataset is duplicated 10 times; 10 distinct random mask patterns are generated offline.
   Epoch 1:  "The [MASK] brown fox jumps over the lazy dog"
   Epoch 2:  "The quick brown [MASK] jumps over the lazy dog"
   ...
   Epoch 11: "The [MASK] brown fox jumps over the lazy dog"  <-- REPEATS IDENTICAL MASK!
   Outcome: Across 40 training epochs, the model sees the exact same masked tokens multiple times,
   leading to subtle overfitting on specific static mask positions.

─────────────────────────────────────────────────────────────────────────────────────────

2. RoBERTa DYNAMIC MASKING (Generated on-the-fly on GPU/CPU worker streams):
   Masking pattern is generated dynamically every single time a sequence is fed into a mini-batch:
   Pass 1: "The [MASK] brown fox jumps over the lazy dog"
   Pass 2: "The quick [MASK] fox jumps over the lazy dog"
   Pass 3: "The quick brown fox [MASK] over the lazy dog"
   Pass 4: "The quick brown fox jumps over [MASK] lazy dog"
   Pass N: Every single pass encounters a fresh, novel stochastic masking pattern!

─────────────────────────────────────────────────────────────────────────────────────────

ROBERTA'S FOUR GOLDEN ARCHITECTURAL MODIFICATIONS:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ MODIFICATION            β”‚ ORIGINAL BERT CONFIG         β”‚ ROBERTA OPTIMIZED CONFIG                 β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Training Objective      β”‚ MLM + Next Sentence Pred.    β”‚ MLM ONLY (NSP completely removed!)       β”‚
β”‚ Masking Strategy        β”‚ Static offline masking       β”‚ Dynamic runtime masking                  β”‚
β”‚ Mini-Batch Size         β”‚ 256 sequences                β”‚ 8,192 sequences (Massive parallelism!)   β”‚
β”‚ Learning Rate Peak      β”‚ 1e-4                         β”‚ 6e-4 (Scaled with batch size)            β”‚
β”‚ Training Corpus Scale   β”‚ 16 GB (Books + English Wiki) β”‚ 160 GB (Books + CC-News + OpenWeb + CC)  β”‚
β”‚ Tokenizer Vocabulary    β”‚ 30,000 WordPiece tokens      β”‚ 50,000 Byte-Level BPE (No OOV tokens!)   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Empirical Takeaway from RoBERTa: Liu et al. showed that BERT was significantly undertrained. Without changing the core Transformer Encoder layer architecture, simply removing the NSP objective, training with dynamic masking over larger batches ($8k$ sequences), using larger corpora ($160\text{ GB}$), and training for longer steps matched or exceeded all downstream models released after BERT.
5. DeBERTa: Disentangled Attention & Enhanced Masked Decoder
He et al. (2021) state-of-the-art encoder advancements: Disentangled content-position attention matrices, two-vector representation ($c_i$ and $p_i$), and the Enhanced Masked Decoder (EMD).
Mathematical Decomposition: Disentangled Attention 4-Component Matrix Sum
DEBERTA DISENTANGLED ATTENTION MECHANICS:

Every token at position i is represented by TWO separate, decoupled vectors:
β€’ Content Vector:   c_i ∈ R^d  (Encodes semantic meaning: "dog", "database")
β€’ Relative Pos:     p_{i|j} ∈ R^d  (Encodes relative distance: i - j)

THE ATTENTION SCORE DECOMPOSITION:
Standard attention sums content and position prior to projection: (c_i + p_i) Β· (c_j + p_j)^T.
Expanding this yields 4 cross-terms:

    Score(i, j) = ( c_i Β· W_q + p_{i|j} Β· W_q,r ) Β· ( c_j Β· W_k + p_{j|i} Β· W_k,r )^T

DeBERTa explicitly computes attention across these disentangled matrices:

    Score(i, j) =   c_i Β· W_{q,c} Β· W_{k,c}^T Β· c_j^T        (1. Content-to-Content: What is word j given word i?)
                  + c_i Β· W_{q,c} Β· W_{k,r}^T Β· p_{j|i}^T    (2. Content-to-Position: Where does word i expect arguments?)
                  + p_{i|j} Β· W_{q,r} Β· W_{k,c}^T Β· c_j^T    (3. Position-to-Content: What word appears at offset Ξ΄?)
                  + p_{i|j} Β· W_{q,r} Β· W_{k,r}^T Β· p_{j|i}^T(4. Position-to-Position: Redundant; DROPPED!)

DeBERTa drops term (4) because relative position alone provides zero semantic information!

─────────────────────────────────────────────────────────────────────────────────────────

THE ENHANCED MASKED DECODER (EMD):
In standard BERT, absolute position is added at layer 0.
DeBERTa keeps all intermediate layers RELATIVE.
It injects ABSOLUTE positional coordinates strictly at the final prediction head right before
predicting masked tokens, allowing syntactic parsing to stay relative while sentence reconstruction
maintains structural awareness.
Performance Milestone: By decoupling content and position representations into separate projection spaces, DeBERTa (and DeBERTa-v3 with ELECTRA-style replaced token detection) consistently outperforms BERT and RoBERTa on the SuperGLUE benchmark, reaching human parity on complex natural language understanding tasks with fewer total parameters.
6. Complete PyTorch Implementation: BERT Architecture with Dynamic MLM Loss
Complete PyTorch reference implementation: Three-way embedding module, 12-layer bidirectional encoder stack, MLM prediction head, dynamic 80/10/10 masking collator, and forward loss computation.
Python / PyTorch (models/bert_from_scratch.py: Complete BERT Architecture & MLM Loss Engine)
import math
import random
import torch
import torch.nn as nn
import torch.nn.functional as F

# ==============================================================================
# 1. THREE-WAY BERT EMBEDDING MODULE
# Sums: Token Embeddings + Segment Embeddings + Learned Position Embeddings
# ==============================================================================
class BERTEmbeddings(nn.Module):
    def __init__(self, vocab_size: int, d_model: int = 768, max_len: int = 512, 
                 type_vocab_size: int = 2, dropout: float = 0.1):
        super().__init__()
        self.word_embeddings = nn.Embedding(vocab_size, d_model, padding_idx=0)
        self.position_embeddings = nn.Embedding(max_len, d_model)
        self.token_type_embeddings = nn.Embedding(type_vocab_size, d_model)

        self.layer_norm = nn.LayerNorm(d_model, eps=1e-12)
        self.dropout = nn.Dropout(dropout)

    def forward(self, input_ids: torch.Tensor, token_type_ids: torch.Tensor = None) -> torch.Tensor:
        seq_len = input_ids.size(1)
        device = input_ids.device

        # Default token_type_ids to all zeros if not provided
        if token_type_ids is None:
            token_type_ids = torch.zeros_like(input_ids)

        # Generate position indices: [0, 1, 2, ..., seq_len - 1]
        position_ids = torch.arange(seq_len, dtype=torch.long, device=device).unsqueeze(0)

        # Compute and sum embeddings
        words_emb = self.word_embeddings(input_ids)
        position_emb = self.position_embeddings(position_ids)
        token_type_emb = self.token_type_embeddings(token_type_ids)

        embeddings = words_emb + position_emb + token_type_emb
        return self.dropout(self.layer_norm(embeddings))

# ==============================================================================
# 2. MASKED LANGUAGE MODELING (MLM) PREDICTION HEAD
# Projects hidden states: Dense -> GELU -> LayerNorm -> Projection to Vocab Logits
# ==============================================================================
class BERTMLMHead(nn.Module):
    def __init__(self, d_model: int, vocab_size: int):
        super().__init__()
        self.dense = nn.Linear(d_model, d_model)
        self.layer_norm = nn.LayerNorm(d_model, eps=1e-12)
        self.decoder = nn.Linear(d_model, vocab_size, bias=False)
        self.bias = nn.Parameter(torch.zeros(vocab_size))
        self.decoder.bias = self.bias

    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
        # Transform representations
        x = self.dense(hidden_states)
        x = F.gelu(x)
        x = self.layer_norm(x)
        # Project to vocabulary logits: (B, S, vocab_size)
        return self.decoder(x)

# ==============================================================================
# 3. COMPLETE BIDIRECTIONAL BERT MODEL STACK
# ==============================================================================
class BERTModel(nn.Module):
    def __init__(self, vocab_size: int = 30522, d_model: int = 768, num_layers: int = 12, 
                 num_heads: int = 12, d_ff: int = 3072, dropout: float = 0.1):
        super().__init__()
        self.embeddings = BERTEmbeddings(vocab_size=vocab_size, d_model=d_model, dropout=dropout)
        
        # Bidirectional Transformer Encoder Stack
        encoder_layer = nn.TransformerEncoderLayer(
            d_model=d_model, 
            nhead=num_heads, 
            dim_feedforward=d_ff, 
            dropout=dropout, 
            activation="gelu", 
            batch_first=True,
            norm_first=False # Post-LN matching canonical BERT
        )
        self.encoder = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)
        
        # MLM Head
        self.mlm_head = BERTMLMHead(d_model, vocab_size)

        # Weight tying: share input token embeddings with output projection head
        self.mlm_head.decoder.weight = self.embeddings.word_embeddings.weight

    def forward(self, input_ids: torch.Tensor, token_type_ids: torch.Tensor = None, 
                attention_mask: torch.Tensor = None) -> torch.Tensor:
        """
        attention_mask: Boolean tensor (B, S) where True indicates valid, False indicates PAD.
        """
        x = self.embeddings(input_ids, token_type_ids)
        
        # Invert mask for PyTorch Transformer (True indicates positions to ignore)
        src_key_padding_mask = (attention_mask == 0) if attention_mask is not None else None
        
        # Bidirectional representation extraction
        hidden_states = self.encoder(x, src_key_padding_mask=src_key_padding_mask)
        
        # Compute vocabulary prediction logits
        logits = self.mlm_head(hidden_states)
        return logits, hidden_states

# ==============================================================================
# 4. RUNTIME 80 / 10 / 10 MASKING COLLATOR
# ==============================================================================
def apply_80_10_10_masking(input_ids: torch.Tensor, mask_token_id: int, 
                           vocab_size: int, pad_token_id: int = 0) -> tuple:
    """
    Applies the exact 15% MLM corruption rule:
    - 80% [MASK]
    - 10% Random Token
    - 10% Unchanged
    Returns: (corrupted_input_ids, mlm_labels)
    """
    labels = input_ids.clone()
    corrupted_inputs = input_ids.clone()
    
    # Create 15% probability mask (ignoring padding tokens)
    prob_matrix = torch.full(input_ids.shape, 0.15, device=input_ids.device)
    prob_matrix.masked_fill_(input_ids == pad_token_id, 0.0)
    
    masked_indices = torch.bernoulli(prob_matrix).bool()
    # Labels: Set non-masked positions to -100 so PyTorch CrossEntropyLoss ignores them!
    labels[~masked_indices] = -100

    # Of the 15% selected:
    # 80% -> [MASK]
    indices_replaced = torch.bernoulli(torch.full(input_ids.shape, 0.8, device=input_ids.device)).bool() & masked_indices
    corrupted_inputs[indices_replaced] = mask_token_id

    # 10% -> Random Word (10% of total masked is 50% of remaining 20%)
    indices_random = torch.bernoulli(torch.full(input_ids.shape, 0.5, device=input_ids.device)).bool() & masked_indices & ~indices_replaced
    random_words = torch.randint(low=4, high=vocab_size, size=input_ids.shape, dtype=torch.long, device=input_ids.device)
    corrupted_inputs[indices_random] = random_words[indices_random]

    # Remaining 10% are kept as original tokens (already present in corrupted_inputs)
    return corrupted_inputs, labels

# ------------------------------------------------------------------------------
# Runtime Execution & Verification Suite
# ------------------------------------------------------------------------------
if __name__ == "__main__":
    torch.manual_seed(42)
    device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
    
    B, S = 2, 16
    VOCAB_SIZE = 1000
    D_MODEL = 256
    LAYERS = 4
    HEADS = 8
    MASK_ID = 103
    PAD_ID = 0

    print("[*] Initializing BERT Architecture with MLM Pre-training Engine...")
    model = BERTModel(vocab_size=VOCAB_SIZE, d_model=D_MODEL, num_layers=LAYERS, num_heads=HEADS).to(device)

    # Synthetic Input Tokens (Batch=2, SeqLen=16)
    raw_tokens = torch.randint(5, VOCAB_SIZE, (B, S), device=device)
    raw_tokens[:, 14:] = PAD_ID # Inject padding tokens at end

    # 1. Apply Dynamic 80/10/10 MLM Corruption
    corrupted_tokens, mlm_targets = apply_80_10_10_masking(raw_tokens, mask_token_id=MASK_ID, vocab_size=VOCAB_SIZE, pad_token_id=PAD_ID)
    
    print(f"[βœ”] Dynamic Masking Applied:")
    print(f"    β€’ Original Tokens Batch 0 : {raw_tokens[0].tolist()}")
    print(f"    β€’ Corrupted Tokens Batch 0: {corrupted_tokens[0].tolist()}")
    print(f"    β€’ Target Labels (Ignored=-100): {mlm_targets[0].tolist()}")

    # 2. Forward Pass: Compute Bidirectional Logits
    attention_mask = (corrupted_tokens != PAD_ID).long()
    logits, _ = model(corrupted_tokens, attention_mask=attention_mask)
    print(f"[βœ”] Forward Pass Completed. Prediction Logits Shape: {logits.shape} (Expected: [{B}, {S}, {VOCAB_SIZE}])")

    # 3. Compute Masked Cross-Entropy Loss
    loss_fn = nn.CrossEntropyLoss(ignore_index=-100)
    loss = loss_fn(logits.view(-1, VOCAB_SIZE), mlm_targets.view(-1))
    print(f"[βœ”] Masked Language Model Loss Computed: {loss.item():.4f}")

    # 4. Backward Pass Verification
    loss.backward()
    print(f"[βœ”] Backpropagation Verified. Word Embedding Grad Norm: {model.embeddings.word_embeddings.weight.grad.norm().item():.4f}")
Execution Console Output: [*] Initializing BERT Architecture with MLM Pre-training Engine... [βœ”] Dynamic Masking Applied: β€’ Original Tokens Batch 0 : [627, 442, 706, 908, 888, 110, 484, 883, 114, 521, 574, 959, 219, 580, 0, 0] β€’ Corrupted Tokens Batch 0: [627, 442, 706, 908, 103, 110, 484, 883, 114, 521, 574, 762, 219, 580, 0, 0] β€’ Target Labels (Ignored=-100): [-100, -100, -100, -100, 888, -100, -100, -100, -100, -100, -100, 959, -100, -100, -100, -100] [βœ”] Forward Pass Completed. Prediction Logits Shape: torch.Size([2, 16, 1000]) (Expected: [2, 16, 1000]) [βœ”] Masked Language Model Loss Computed: 6.9421 [βœ”] Backpropagation Verified. Word Embedding Grad Norm: 1.8412
7. Downstream Adaptation: Fine-Tuning BERT for Classification & Extraction
Transfer learning mechanics: How pre-trained autoencoding weights are adapted to downstream tasks via classification pooling heads, span extraction for QA (SQuAD), and token-level labeling (NER).
ASCII Diagram: Adapting Pre-trained BERT to Multiple Downstream Topologies
TRANSFER LEARNING TOPOLOGIES FOR PRE-TRAINED BERT:

(A) SEQUENCE CLASSIFICATION (Sentiment / GLUE Benchmark):
Extract vector at index 0 (the [CLS] token) and apply linear classification head:
  h_[CLS] ∈ R^{d_model} ──► [ Dense Layer (d βž” d) + Tanh ] ──► [ Linear (d βž” C) ] ──► Softmax Logits

─────────────────────────────────────────────────────────────────────────────────────────

(B) TOKEN-LEVEL SEQUENCE LABELING (Named Entity Recognition - NER):
Process EVERY token's output vector h_1 ... h_S through a shared token classification projection:
  h_1 ──► [ Linear (d βž” Num_Entities) ] ──► Label: B-PER (Person)
  h_2 ──► [ Linear (d βž” Num_Entities) ] ──► Label: I-PER
  h_3 ──► [ Linear (d βž” Num_Entities) ] ──► Label: O (Outside)

─────────────────────────────────────────────────────────────────────────────────────────

(C) EXTRACTIVE QUESTION ANSWERING (SQuAD Benchmark):
Input: "[CLS] Question tokens [SEP] Passage context tokens [SEP]"
Predict the exact answer boundary within the passage:
β€’ Start Logit: s_start(i) = w_start^T Β· h_i
β€’ End Logit:   s_end(j)   = w_end^T Β· h_j
Select span (i, j) that maximizes s_start(i) + s_end(j) subject to j >= i!
Fine-Tuning Parameter Updates: During fine-tuning, all parameters across the Transformer encoder layers are updated end-to-end alongside the newly added task-specific output head. Because the pre-trained weights already encode syntax, grammar, and factual associations, fine-tuning converges in a fraction of pre-training timeβ€”typically 2 to 4 epochs over small task-specific datasets using small learning rates ($2\times 10^{-5}$ to $5\times 10^{-5}$).
8. Common Antipatterns & Computational Traps in Autoencoders
Diagnostic playbook covering critical autoencoding bugs: Setting target mask cross-entropy reduction indices incorrectly, leaking test tokens during fine-tuning, and catastrophic forgetting during full parameter adaptation.
Troubleshooting Playbook: Autoencoding Defects & Engineering Remediations
AUTOENCODING PRE-TRAINING ANOMALIES & REMEDIATIONS:

1. COMPUTING MLM LOSS OVER UNCORRUPTED TOKENS (85% LEAKAGE BUG):
   β€’ Symptom: Training loss stays low, but the model learns nothing; downstream representations are useless.
   β€’ Cause: Calling nn.CrossEntropyLoss() directly on raw targets without masking out the 85% unselected tokens!
     The model trivially memorizes identity pass-throughs because 85% of input tokens match target tokens!
   β€’ Remediation: In PyTorch, set labels at all non-masked positions to -100:
     labels[~masked_indices] = -100
     And ensure: nn.CrossEntropyLoss(ignore_index=-100).

2. PASSING CAUSAL MASKS TO BERT ENCODERS:
   β€’ Symptom: BERT behaves like a poor unidirectional language model; GLUE scores drop by 30%.
   β€’ Cause: Using a causal lower-triangular mask instead of an unmasked bidirectional attention matrix.
   β€’ Remediation: Encoders must remain unmasked across all non-padding sequence tokens.

3. FINE-TUNING OVERFITTING & CATASTROPHIC FORGETTING:
   β€’ Symptom: Performance on validation datasets drops to near-chance after 3 epochs of fine-tuning.
   β€’ Cause: Using a standard learning rate (e.g. 1e-3). Pre-trained representations are destroyed!
   β€’ Remediation:
     - Use small learning rates: lr ∈ [1e-5, 5e-5].
     - Apply linear warmup for the first 10% of fine-tuning steps, followed by linear decay.
     - Implement Layer-wise Discriminative Learning Rate Decay (LLRD): Lower layers receive smaller
       learning rates than the top layers.

4. ACCIDENTAL LOSS COMPUTATION OVER PADDING TOKENS:
   β€’ Symptom: Gradient norms spike; loss plateaus at a high value.
   β€’ Cause: Failing to exclude pad_token_id from the initial 15% selection mask.
   β€’ Remediation: Mask out pad tokens before calling torch.bernoulli().
Diagnostic Tool: To measure whether an autoencoding encoder is retaining bidirectional contextual representations, inspect the cosine similarity between the representations of a polysemous word across different contexts (e.g., "apple" in "apple fruit recipe" vs. "apple stock market"): $$\text{Contextual Distance} = 1 - \cos(\mathbf{h}_{\text{apple}, 1}, \mathbf{h}_{\text{apple}, 2})$$ If the distance is near zero, the model has collapsed into static, context-independent token representations. In a healthy bidirectional encoder, this distance is significantly greater than zero ($> 0.4$).
9. Technical Specification & Examination Reference Matrix
Comprehensive reference matrix summarizing autoencoding architectures, pre-training objectives, operational roles, and academic examination definitions for B.Tech students.
Autoencoding Architecture Technical Specification Matrix
AUTOENCODING ARCHITECTURAL SPECIFICATION MATRIX:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ MODEL ARCHITECTURE      β”‚ PRIMARY PRE-TRAINING OBJECTIVEβ”‚ ATTENTION MASK   β”‚ CRITICAL INNOVATION           β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ BERT                    β”‚ MLM (80/10/10 rule)           β”‚ Bidirectional    β”‚ First deep bidirectional      β”‚
β”‚ (Devlin et al., 2018)   β”‚ + Next Sentence Pred. (NSP)   β”‚ (Unmasked)       β”‚ denoising representation modelβ”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ RoBERTa                 β”‚ Dynamic MLM Only              β”‚ Bidirectional    β”‚ Drops NSP; 8k batch size;     β”‚
β”‚ (Liu et al., 2019)      β”‚ (NSP completely removed)      β”‚ (Unmasked)       β”‚ 10x larger dataset; Byte-BPE  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ ALBERT                  β”‚ Dynamic MLM                   β”‚ Bidirectional    β”‚ Factorized embedding params;  β”‚
β”‚ (Lan et al., 2019)      β”‚ + Sentence-Order Pred. (SOP)  β”‚ (Cross-layer par)β”‚ cross-layer weight sharing    β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ ELECTRA                 β”‚ Replaced Token Detection      β”‚ Bidirectional    β”‚ Replaces MLM with generator/  β”‚
β”‚ (Clark et al., 2020)    β”‚ (Binary Discriminator loss)   β”‚ (Unmasked)       β”‚ discriminator binary detectionβ”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ DeBERTa                 β”‚ Disentangled Attention        β”‚ Disentangled     β”‚ Decouples content and relativeβ”‚
β”‚ (He et al., 2021)       β”‚ (Content & Rel-Position sep)  β”‚ (Unmasked)       β”‚ position; Enhanced Masked Dec.β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ TOP 5 AUTOENCODING AXIOMS FOR AI ENGINEERS:                                                                β”‚
β”‚ 1. Bidirectionality Demands Masking: You cannot train bidirectional models autoregressively; you must      β”‚
β”‚    corrupt tokens with Masked Language Modeling (MLM) to avoid trivial identity leaks.                     β”‚
β”‚ 2. The 80/10/10 Rule Mitigates Discrepancy: Replacing 80% with [MASK], 10% with random tokens, and 10%     β”‚
β”‚    unchanged bridges the gap between pre-training noise and clean inference inputs.                        β”‚
β”‚ 3. NSP is Ineffective: Topic-level negative sampling makes NSP trivial; remove it (RoBERTa) or replace     β”‚
β”‚    it with Sentence-Order Prediction (ALBERT).                                                             β”‚
β”‚ 4. Decouple Content from Position: Disentangled attention (DeBERTa) yields higher parameter efficiency by  β”‚
β”‚    evaluating content-to-content and content-to-position matrices independently.                           β”‚
β”‚ 5. Autoencoders Excel at Understanding: Use causal decoder models (GPT) for open-ended text generation;    β”‚
β”‚    use bidirectional autoencoders (BERT, RoBERTa, DeBERTa) for classification, extraction, and retrieval.  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Academic & Professional Standard: Autoencoding architectures represent the foundation of natural language understanding and semantic representation. Mastering bidirectional Masked Language Modeling, the empirical optimizations of RoBERTa, and DeBERTa's disentangled attention formulations equips engineers with the technical background needed to design search systems, embedding models, and classification pipelines across modern AI architectures.