π The Text-to-Text Unification: Bridging Understanding and Generation
Prior to 2019, Natural Language Processing suffered from architectural partitioning: encoder-only models (BERT, RoBERTa) excelled at understanding, classification, and extraction tasks through bidirectional attention, but could not synthesize text; conversely, decoder-only models (GPT series) generated coherent prose autoregressively, but were crippled on conditional transformation tasks due to unidirectional causal constraints. Sequence-to-Sequence (Seq2Seq) Pre-training bridged this divide. Formalized by Raffel et al. in the Text-to-Text Transfer Transformer (T5) and Lewis et al. in BART (Bidirectional and Auto-Regressive Transformers), this paradigm casts every NLP problemβtranslation, question answering, document summarization, grammatical error correction, and multi-class classificationβinto a unified text-to-text format. By pairing a bidirectional encoder (which ingests unconstrained corrupted source context) with an autoregressive decoder (which reconstructs missing spans conditioned on cross-attention), Seq2Seq pre-training establishes a mathematically robust framework for high-fidelity conditional text generation.
THE T5 UNIVERSAL TEXT-TO-TEXT FORMULATION (Raffel et al., 2019):
Every task ingests a string prefixed by a task directive and emits a target string.
Zero task-specific heads! Identical Cross-Entropy loss! Identical Autoregressive Decoding!
TASK 1: MACHINE TRANSLATION (English β German)
Input String : "translate English to German: That is good."
Model Output : "Das ist gut."
TASK 2: ABSTRACTIVE DOCUMENT SUMMARIZATION
Input String : "summarize: The multi-year infrastructure overhaul concluded ahead of schedule..."
Model Output : "Infrastructure upgrades finished early."
TASK 3: BINARY SENTIMENT CLASSIFICATION (GLUE Benchmark)
Input String : "sst2 sentence: The film provided exceptional cinematography and depth."
Model Output : "positive" (Direct string emission; no softmax classification head!)
TASK 4: LINGUISTIC ACCEPTABILITY (CoLA Benchmark)
Input String : "cola sentence: The course is jumped well."
Model Output : "unacceptable"
TASK 5: EXTRACTIVE QUESTION ANSWERING (SQuAD)
Input String : "question: Where is the Eiffel Tower? context: Located in Paris, France..."
Model Output : "Paris, France"
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
ARCHITECTURAL ADVANTAGE:
β’ Eliminates custom classification projection matrices (W_class β R^{d Γ C}).
β’ Multi-task pre-training and fine-tuning share the exact same parameter space and gradient updates.
β’ The model leverages syntactic transfer learning across summarization, classification, and translation!
T5 SPAN CORRUPTION PRE-TRAINING PIPELINE:
Original Unlabeled Sentence:
"Thank you for inviting me to your birthday party on Sunday afternoon"
1. SAMPLE CORRUPTION SPANS (15% corruption budget, mean span length ΞΌ = 3):
Span 1: "for inviting me"
Span 2: "birthday"
Span 3: "Sunday afternoon"
2. REPLACE ENTIRE SPANS WITH UNIQUE SENTINEL TOKENS IN ENCODER INPUT:
Encoder Input Sequence x:
"Thank you <extra_id_0> to your <extra_id_1> party on <extra_id_2>"
3. TARGET OUTPUT DECODER SEQUENCE CONTAINS ONLY CORRUPTED SPANS + SENTINELS:
Decoder Target Sequence y:
"<extra_id_0> for inviting me <extra_id_1> birthday <extra_id_2> Sunday afternoon <extra_id_3>"
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
WHY SPAN CORRUPTION OUTPERFORMS BERT'S SINGLE-TOKEN MLM:
1. TARGET SEQUENCE COMPRESSION (COMPUTATIONAL EFFICIENCY):
β’ In BERT or full reconstruction, the decoder outputs all N tokens: O(N) decoding steps.
β’ In T5 Span Corruption, target length is compressed strictly to 15% of N!
For N = 512, target length is only ~77 tokens!
Slashes decoder training time and FLOP consumption by > 70%!
2. FORCES LONGER-RANGE REASONING:
β’ Single-token masking allows models to rely on trivial local n-gram syntax ("for [MASK] me").
β’ Multi-token spans (ΞΌ = 3 tokens) force the encoder to capture multi-word semantic chunks.
<extra_id_k>, which are dynamically mapped to reserved token IDs in the vocabulary.BART DENOISING NOISE TRANSFORMATIONS:
BART applies arbitrary, non-linear corruption to document X; the decoder must reconstruct the ORIGINAL clean document X.
βββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββββββ
β CORRUPTION NOISE TYPE β CORRUPTED ENCODER INPUT ~X β OBJECTIVE & LEARNING SIGNAL β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββ€
β Token Masking β "The [MASK] brown fox [MASK] over..." β BERT-style MLM reconstruction; forces β
β β β local contextual token prediction. β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββ€
β Token Deletion β "The brown jumps over the dog" β Forces model to detect MISSING positions β
β β (Tokens "quick" & "fox" completely gone) β and insert tokens without [MASK] clues! β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββ€
β Text Infilling β "The [MASK] jumps over [MASK] dog" β Poisson-sampled span (0 to k tokens). β
β β (Single [MASK] replaces 3 tokens!) β Forces model to predict span LENGTH. β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββ€
β Sentence Permutation β "[Sentence C]. [Sentence A]. [Sent B]" β Shuffles full sentences; forces model to β
β β β learn global discourse & narrative flow. β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββββββ€
β Document Rotation β "fox jumps over. The quick brown" β Selects random token as start; forces β
β β β identification of true document opening. β
βββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββββββ
THE THREE ATTENTION CONNECTIVITY REGIMES:
Sequence: Tokens 1, 2, 3 (Input Prefix / Prompt) | Tokens 4, 5, 6 (Target Completion)
1. FULLY BIDIRECTIONAL (BERT): 2. STRICTLY CAUSAL (GPT): 3. PREFIX LANGUAGE MODEL (Prefix LM):
All tokens attend to all tokens. Tokens attend ONLY to past <= i. Prefix tokens attend bidirectionally;
No generation capability! Autoregressive generation only. Target tokens attend causally!
1 2 3 4 5 6 1 2 3 4 5 6 1 2 3 4 5 6
1 [β β β β β β ] 1 [β Β· Β· Β· Β· Β·] 1 [β β β Β· Β· Β·] βββ Bidirectional
2 [β β β β β β ] 2 [β β Β· Β· Β· Β·] 2 [β β β Β· Β· Β·] Prefix Block
3 [β β β β β β ] 3 [β β β Β· Β· Β·] 3 [β β β Β· Β· Β·] βββ (Tokens 1..3)
4 [β β β β β β ] 4 [β β β β Β· Β·] 4 [β β β β Β· Β·] βββ Causal
5 [β β β β β β ] 5 [β β β β β Β·] 5 [β β β β β Β·] Autoregressive
6 [β β β β β β ] 6 [β β β β β β ] 6 [β β β β β β ] Target (4..6)
ADVANTAGE OF PREFIX LM (Used in PaLM, GLM, UL2):
Allows decoder-only models to process input prompts with bidirectional attention (like an encoder),
while emitting generated tokens autoregressively (like a decoder) in a single unified architecture!
THE C4 (COLOSSAL CLEAN CRAWLED CORPUS) HEURISTIC CLEANING PIPELINE (Raffel et al.):
Raw Common Crawl Web Dump (~20+ Terabytes of uncurated HTML/text)
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STEP 1: TERMINAL PUNCTUATION FILTER β
β Discard any line that does NOT end with a valid terminalβ
β punctuation mark (period, exclamation, question mark). β βββΊ Drops navigation menus, sidebars, headers.
ββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STEP 2: MINIMUM LENGTH & WORD COUNT THRESHOLDS β
β Discard any document with fewer than 5 sentences or β
β containing fewer than 3 words per line on average. β βββΊ Drops error pages, 404 logs, search snippets.
ββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STEP 3: LORDS OF VULGARITY & BAD WORDS REMOVAL β
β Scan documents against the 'List of Dirty, Naughty, β
β Obscene or Otherwise Bad Words'; drop matching pages. β βββΊ Mitigates offensive / toxic pre-training text.
ββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STEP 4: CODE & PROGRAMMING SYNTAX REMOVAL β
β Discard any document containing placeholder symbols β
β such as '{' or '}' (curly braces) or 'lorem ipsum'. β βββΊ Prunes JavaScript snippets & boilerplate.
ββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ
β
βΌ
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β STEP 5: EXACT THREE-SENTENCE DEDUPLICATION β
β Hash all 3-sentence spans across the entire dataset; β
β delete any document sharing duplicate hashes. β βββΊ Eliminates repeated licensing terms/TOS.
ββββββββββββββββββββββββββββββ¬ββββββββββββββββββββββββββββ
β
βΌ
Final Cleaned C4 Dataset: ~750 Gigabytes of clean, continuous English text!
import math
import random
import torch
import torch.nn as nn
import torch.nn.functional as F
# ==============================================================================
# 1. T5 DYNAMIC SPAN CORRUPTION COLLATOR
# Corrupts contiguous token spans and replaces them with sentinels
# ==============================================================================
class T5SpanCorruptionCollator:
def __init__(self, vocab_size: int, sentinel_start_id: int,
noise_density: float = 0.15, mean_noise_span_length: float = 3.0):
self.vocab_size = vocab_size
self.sentinel_start_id = sentinel_start_id
self.noise_density = noise_density
self.mean_noise_span_length = mean_noise_span_length
def __call__(self, input_ids: torch.Tensor) -> tuple:
"""
input_ids: LongTensor of shape (Batch_Size, Seq_Len)
Returns:
encoder_input_ids : Corrupted sequence with tokens
decoder_target_ids: Target sequence containing only corrupted spans + sentinels
"""
batch_size, seq_len = input_ids.shape
device = input_ids.device
encoder_inputs = []
decoder_targets = []
for b in range(batch_size):
tokens = input_ids[b].tolist()
num_tokens_to_mask = int(math.ceil(seq_len * self.noise_density))
# Determine span lengths and starting points
masked_indices = set()
while len(masked_indices) < num_tokens_to_mask:
span_len = max(1, int(random.expovariate(1.0 / self.mean_noise_span_length)))
start_idx = random.randint(0, seq_len - 1)
for i in range(start_idx, min(start_idx + span_len, seq_len)):
masked_indices.add(i)
# Build encoder inputs and decoder targets using sentinel tokens
enc_in = []
dec_tgt = []
sentinel_idx = 0
in_masked_span = False
for i, token in enumerate(tokens):
if i in masked_indices:
if not in_masked_span:
# Enter new corrupted span: Emit unique sentinel
sentinel_token = self.sentinel_start_id - sentinel_idx
enc_in.append(sentinel_token)
dec_tgt.append(sentinel_token)
sentinel_idx += 1
in_masked_span = True
# Append masked token to target output
dec_tgt.append(token)
else:
in_masked_span = False
enc_in.append(token)
# Append terminal sentinel to target
dec_tgt.append(self.sentinel_start_id - sentinel_idx)
encoder_inputs.append(torch.tensor(enc_in, dtype=torch.long, device=device))
decoder_targets.append(torch.tensor(dec_tgt, dtype=torch.long, device=device))
# Pad sequences to max length in current batch
enc_pad = nn.utils.rnn.pad_sequence(encoder_inputs, batch_first=True, padding_value=0)
dec_pad = nn.utils.rnn.pad_sequence(decoder_targets, batch_first=True, padding_value=0)
return enc_pad, dec_pad
# ==============================================================================
# 2. SEQUENCE-TO-SEQUENCE TRANSFORMER MODULE (T5 ARCHITECTURAL BLUEPRINT)
# ==============================================================================
class Seq2SeqTransformer(nn.Module):
def __init__(self, vocab_size: int, d_model: int = 256, nhead: int = 8,
num_encoder_layers: int = 3, num_decoder_layers: int = 3,
dim_feedforward: int = 1024, dropout: float = 0.1):
super().__init__()
self.d_model = d_model
self.embedding = nn.Embedding(vocab_size, d_model, padding_idx=0)
# Canonical Transformer core
self.transformer = nn.Transformer(
d_model=d_model,
nhead=nhead,
num_encoder_layers=num_encoder_layers,
num_decoder_layers=num_decoder_layers,
dim_feedforward=dim_feedforward,
dropout=dropout,
batch_first=True,
norm_first=True # Pre-LN architectural stability
)
# Output language modeling projection head
self.lm_head = nn.Linear(d_model, vocab_size, bias=False)
# Weight tying: Share embedding weights with projection head
self.lm_head.weight = self.embedding.weight
def forward(self, src: torch.Tensor, tgt_input: torch.Tensor,
src_pad_mask: torch.Tensor = None, tgt_pad_mask: torch.Tensor = None):
"""
src : (Batch_Size, Src_Len)
tgt_input : (Batch_Size, Tgt_Len)
src_pad_mask : (Batch_Size, Src_Len)
tgt_pad_mask : (Batch_Size, Tgt_Len)
"""
# Embed and scale representations
src_emb = self.embedding(src) * math.sqrt(self.d_model)
tgt_emb = self.embedding(tgt_input) * math.sqrt(self.d_model)
# Build causal mask for decoder
tgt_len = tgt_input.size(1)
causal_mask = nn.Transformer.generate_square_subsequent_mask(tgt_len, device=src.device)
# Execute Transformer Encoder-Decoder dataflow
out = self.transformer(
src=src_emb,
tgt=tgt_emb,
tgt_mask=causal_mask,
src_key_padding_mask=src_pad_mask,
tgt_key_padding_mask=tgt_pad_mask
)
# Project hidden states to vocabulary logits: (Batch, Tgt_Len, vocab_size)
return self.lm_head(out)
# ------------------------------------------------------------------------------
# Runtime Execution & Verification Suite
# ------------------------------------------------------------------------------
if __name__ == "__main__":
torch.manual_seed(42)
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
VOCAB_SIZE = 1000
SENTINEL_START = 999 # Tokens 999, 998, 997 act as , ...
B, S = 2, 20
print("[*] Initializing T5 Span Corruption & Seq2Seq Pre-training Engine...")
collator = T5SpanCorruptionCollator(
vocab_size=VOCAB_SIZE,
sentinel_start_id=SENTINEL_START,
noise_density=0.15,
mean_noise_span_length=3.0
)
# Synthetic Input Tokens (Batch=2, SeqLen=20)
raw_text_tokens = torch.randint(10, VOCAB_SIZE - 20, (B, S), device=device)
# 1. Apply Dynamic Span Corruption
enc_inputs, dec_targets = collator(raw_text_tokens)
print(f"[β] Span Corruption Collator Success:")
print(f" β’ Original Tokens Batch 0 : {raw_text_tokens[0].tolist()[:10]}...")
print(f" β’ Encoder Inputs Batch 0 : {enc_inputs[0].tolist()}")
print(f" β’ Decoder Targets Batch 0 : {dec_targets[0].tolist()}")
# 2. Prepare Shifted Targets for Teacher Forcing
# Decoder inputs: targets shifted right by one token
dec_input = dec_targets[:, :-1]
dec_label = dec_targets[:, 1:]
# 3. Instantiate Seq2Seq Model and Execute Forward Pass
model = Seq2SeqTransformer(vocab_size=VOCAB_SIZE, d_model=128, nhead=4).to(device)
src_pad_mask = (enc_inputs == 0)
tgt_pad_mask = (dec_input == 0)
logits = model(enc_inputs, dec_input, src_pad_mask=src_pad_mask, tgt_pad_mask=tgt_pad_mask)
print(f"[β] Forward Pass Success. Logits Shape: {logits.shape} (Expected: [{B}, {dec_input.size(1)}, {VOCAB_SIZE}])")
# 4. Compute Cross-Entropy Loss over Targets
loss_fn = nn.CrossEntropyLoss(ignore_index=0) # Ignore padding tokens
loss = loss_fn(logits.view(-1, VOCAB_SIZE), dec_label.contiguous().view(-1))
print(f"[β] Span Infilling Cross-Entropy Loss Computed: {loss.item():.4f}")
# 5. Backward Gradient Verification
loss.backward()
print(f"[β] Backpropagation Verified. Shared Embedding Grad Norm: {model.embedding.weight.grad.norm().item():.4f}")
THE UL2 MIXTURE-OF-DENOISERS (MoD) FRAMEWORK:
Instead of committing to a single pre-training objective, UL2 dynamically mixes THREE distinct
denoising modes within the SAME model using mode prefix tokens:
1. R-DENOISER (Regular Span Corruption - T5 Style):
β’ Mode Token: "[R-DENOISE]"
β’ Configuration: Noise density = 15%, Span length ΞΌ = 3 tokens.
β’ Goal: Acquires local grammatical knowledge and fine-grained subword infilling.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
2. S-DENOISER (Sequential / Prefix Language Modeling - GPT Style):
β’ Mode Token: "[S-DENOISE]"
β’ Configuration: Splits sequence into Prefix (e.g. 75%) and Target (e.g. 25%).
β’ Goal: Develops fluent, open-ended autoregressive generation and in-context learning.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
3. X-DENOISER (Extreme Denoising - Long Span Infilling):
β’ Mode Token: "[X-DENOISE]"
β’ Configuration: Extreme noise density = 50%, Long span length ΞΌ = 12 tokens.
β’ Goal: Forces the model to synthesize massive missing document sections without local clues!
OUTCOME IN DOWNSTREAM EVALUATIONS:
Models pre-trained with UL2 (e.g. PaLM-2, Flan-UL2) match or exceed GPT models on generative benchmarks
while matching or exceeding RoBERTa/DeBERTa on classification and retrieval benchmarks!
SEQ2SEQ PRE-TRAINING ANOMALIES & REMEDIATIONS:
1. THE SENTINEL TOKEN ID COLLISION DISASTER:
β’ Symptom: Training diverges; the model generates standard vocabulary words when it should output
sentinel boundary markers <extra_id_k>.
β’ Root Cause: Allocating sentinel token IDs inside the range of normal text vocabulary:
sentinel_id = random.randint(0, vocab_size) # FATAL: Collides with common words like "the"!
β’ Remediation: Sentinel tokens must occupy dedicated, reserved IDs outside the primary vocabulary:
Reserved IDs: [vocab_size - 100, vocab_size - 1].
2. TARGET TEACHER-FORCING SHIFT MISALIGNMENT:
β’ Symptom: The model achieves 0.01 loss on training batches, but generates garbage sentinels during inference!
β’ Root Cause: Passing the identical target sequence as both decoder input and target label:
loss = criterion(logits, dec_targets) # Model learns identity mapping: predicts token t given token t!
β’ Remediation: Shift target inputs and labels explicitly:
dec_input = dec_targets[:, :-1] # Feed tokens 0 to M-1
dec_label = dec_targets[:, 1:] # Supervise tokens 1 to M
3. CASCADING OVER-GENERATION IN INFERENCE (MISSING EOS TERMINATION):
β’ Symptom: Model summarizes text correctly, but continues emitting repetitive sentences until max_len.
β’ Cause: The training targets lacked an explicit End-Of-Sequence token (<EOS> / </s>).
β’ Remediation: Always append terminal sentinels or <EOS> to every training target sequence:
dec_tgt.append(eos_token_id).
4. CROSS-ATTENTION RECOMPUTATION BOTTLENECK IN GENERATION:
β’ Symptom: Generating a 100-token summary takes 15 seconds; GPU compute utilization sits at 5%.
β’ Cause: Re-running the entire encoder stack on every single decoder autoregressive step!
β’ Remediation: Execute encoder ONCE during prefill; cache and reuse encoder memory (H_enc) across
all subsequent decoder steps.
SEQUENCE-TO-SEQUENCE PRE-TRAINING SPECIFICATION MATRIX:
βββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ¬βββββββββββββββββββ¬ββββββββββββββββββββββββββββββββ
β MODEL ARCHITECTURE βPRIMARY PRE-TRAINING OBJECTIVEβ TARGET FORMAT β REPRESENTATIVE DATASET β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β T5 β Span Corruption (ΞΌ = 3) β Compressed Spans β Colossal Clean Crawled β
β (Raffel et al., 2019) β Sentinel Infilling β + Sentinels Only β Corpus (C4 - 750 GB) β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β BART β 5 Denoising Transformations β Full Clean β 160 GB Corpus β
β (Lewis et al., 2019) β Infilling, Deletion, Permute β Text Sequence β (Books, Wikipedia, News) β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β mT5 β Multilingual Span Corruption β Compressed Spans β mC4 (101 Languages, β
β (Xue et al., 2020) β 250k Vocabulary Size β + Sentinels Only β Petabyte Scale Web Text) β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β Prefix LM β Bidirectional Prefix + β Sequential β Massive Web Corpora β
β (PaLM, GLM, UL2) β Causal Autoregressive Target β Completion Text β (Multi-Task Prompt Mixtures) β
βββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββΌββββββββββββββββββββββββββββββββ€
β UL2 β Mixture-of-Denoisers (MoD): β Variable by Mode β Cleaned Common Crawl β
β (Tay et al., 2022) β R-Denoiser, S-Denoiser, X-Denβ [R], [S], [X] β (Mixture-of-Experts Scaling) β
βββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ΄βββββββββββββββββββ΄ββββββββββββββββββββββββββββββββ€
β TOP 5 SEQ2SEQ PRE-TRAINING AXIOMS FOR AI ENGINEERS: β
β 1. Text-to-Text Unifies Everything: Framing all NLP tasks as text-to-text string mappings eliminates β
β custom output heads, allowing multi-task transfer learning across classification, QA, and translation. β
β 2. Span Corruption Saves FLOPs: Masking multi-token spans and predicting ONLY the missing tokens (T5) β
β slashes decoder training compute by over 70% compared to full document reconstruction. β
β 3. Sentinels Prevent Length Ambiguity: Replacing variable-length spans with unique indexed sentinels β
β (<extra_id_k>) gives the decoder explicit anchor coordinates for where infilling occurs. β
β 4. Precompute Encoder States: In Seq2Seq generation, run the encoder ONCE; caching encoder memory avoids β
β redundant O(N) evaluations during the autoregressive decoding loop. β
β 5. Clean Data Dominates Architecture: The C4 heuristic cleaning pipeline demonstrated that filtering β
β boilerplate, JavaScript syntax, and duplicate text is as critical as scaling model parameters. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ