π¬ From Discrete Symbols to High-Dimensional Continuous Semantics
Natural language cannot be directly ingested by neural network architectures; it exists as discrete, symbolic, non-continuous strings. The evolution of modern Natural Language Processing (NLP) centers on constructing dense, continuous vector spaces where geometric proximity reflects semantic meaning. Early techniques relied on one-hot sparse encodings where word orthogonalities failed to capture synonyms. Static embeddings (Word2Vec, GloVe) grounded representation learning in the Distributional Hypothesisβwords sharing similar context windows inhabit adjacent vector neighborhoods via inner product maximizing objectives. However, static representations assign a single static vector to polysemous words (e.g., "bank" of a river vs. financial "bank"). Contextual Transformer embeddings resolve this by dynamically computing token representations conditioned on surrounding tokens via self-attention. Today, these vectors form the operational foundation of Retrieval-Augmented Generation (RAG) knowledge stores and scalable Parameter-Efficient Fine-Tuning (PEFT / LoRA) adaptation for foundation Large Language Models (LLMs).
THE NATURAL LANGUAGE REPRESENTATION SPECTRUM:
1. SYMBOLIC DISCRETE LEVEL:
* One-Hot Encoding: Vocabulary V (e.g. 50,000 words). Dimension D = 50,000.
- Vectors: "cat" = [1, 0, 0, ...], "feline" = [0, 1, 0, ...]
- Dot Product: cos(cat, feline) = 0.0 (Zero semantic relationship!).
* Byte-Pair Encoding (BPE / WordPiece):
- Iteratively merges most frequent byte/character pairs.
- Fixes Out-Of-Vocabulary (OOV) failure by decomposing unknown words into subwords.
2. STATIC DENSE EMBEDDINGS (Word2Vec / GloVe):
* Projections: x_w β β^d (d typically 100 to 300 dimensions).
* Skip-Gram Objective: Maximize P(context | center) using negative sampling:
L = log Ο(v'_c Β· v_w) + β_{k=1}^K E_{w_k ~ P_n} [ log Ο(-v'_k Β· v_w) ]
* Bottleneck: Polysemy blindness ("bank" has 1 fixed vector regardless of context).
3. CONTEXTUAL DENSE EMBEDDINGS & FOUNDATION LLMS:
* Transformer Encoders (BERT, RoBERTa, Modern BGE / Voyage / E5):
- Dynamic vector conditioned on full bidirectional cross-attention:
e_w = Transformer_Encoder(tokens)[position_w]
- "river bank" != "bank account" in vector space.
* Pooling Strategies:
- [CLS] Token Pooling: First classification token representation.
- Mean Pooling: Element-wise mean of all token contextual vectors across sequence.
* Parameter-Efficient Fine-Tuning (PEFT / LoRA):
- Freezes base model weights W_0 β β^{d Γ k}, adds low-rank decomposition matrices:
W = W_0 + ΞW = W_0 + (B Β· A) where B β β^{d Γ r}, A β β^{r Γ k}, r << min(d, k)
from collections import Counter, defaultdict
import pandas as pd
# Micro text corpus representing typical user customer-support queries
corpus = [
"low lower lowest",
"newer newest wide wider",
"low wide lower newest"
]
# 1. Initialize words as split characters ending with an end-of-word marker ''
def initialize_vocab(text_list):
vocab = Counter()
for sentence in text_list:
for word in sentence.strip().split():
# Separate each character by space and append special delimiter
tokenized_word = " ".join(list(word)) + " "
vocab[tokenized_word] += 1
return vocab
# 2. Count adjacent bigram frequencies across all active vocabulary words
def get_stats(vocab):
pairs = defaultdict(int)
for word, freq in vocab.items():
symbols = word.split()
for i in range(len(symbols) - 1):
pairs[symbols[i], symbols[i + 1]] += freq
return pairs
# 3. Merge the most frequent consecutive pair across all words
def merge_vocab(pair, v_in):
v_out = {}
bigram = " ".join(pair)
replacement = "".join(pair)
for word in v_in:
# Replace occurrences of 'char_a char_b' with 'char_a+char_b'
w_out = word.replace(bigram, replacement)
v_out[w_out] = v_in[word]
return v_out
vocab = initialize_vocab(corpus)
NUM_MERGES = 6
merge_history = []
for step in range(1, NUM_MERGES + 1):
pairs = get_stats(vocab)
if not pairs:
break
best_pair = max(pairs, key=pairs.get)
vocab = merge_vocab(best_pair, vocab)
merge_history.append({
"Iteration": step,
"Best_Merged_Pair": f"{best_pair[0]} + {best_pair[1]}",
"Corpus_Frequency": pairs[best_pair]
})
df_merges = pd.DataFrame(merge_history)
print("=" * 65)
print("BYTE-PAIR ENCODING (BPE) LEARNED VOCABULARY MERGE AUDIT")
print("=" * 65)
print(df_merges.to_string(index=False))
print("-" * 65)
print("Resulting Subword Vocabulary Post-Merge:")
for word_structure, freq in vocab.items():
print(f" {word_structure:25s} (Count: {freq})")
print("-" * 65)
print("OOV Resilience:")
print(" An unseen word like 'lowestwide' decomposes into learned subword units:")
print(" ['low', 'est', 'wide'], eliminating out-of-vocabulary crashes.")
import numpy as np
import pandas as pd
# Training text representing semantic corporate/banking operations
sentences = [
"bank approves loan application",
"customer deposits money in bank",
"bank provides financial investment loan",
"finance company evaluates risk"
]
# 1. Build Token Index mappings
tokens = [w for s in sentences for w in s.split()]
vocab = sorted(list(set(tokens)))
word2idx = {w: i for i, w in enumerate(vocab)}
idx2word = {i: w for i, w in enumerate(vocab)}
VOCAB_SIZE = len(vocab)
EMBED_DIM = 4
K_NEGATIVES = 2 # 2 negative samples drawn per positive pair
np.random.seed(42)
# Center word embedding matrix W and Context embedding matrix W_context
W_center = np.random.randn(VOCAB_SIZE, EMBED_DIM) * 0.1
W_context = np.random.randn(VOCAB_SIZE, EMBED_DIM) * 0.1
def sigmoid(z):
return 1.0 / (1.0 + np.exp(-np.clip(z, -20.0, 20.0)))
# 2. Extract Positive Center-Context Pairs (Window Size = 1)
pairs = []
for s in sentences:
words = s.split()
for idx, w in enumerate(words):
c_idx = word2idx[w]
# Previous word
if idx > 0:
pairs.append((c_idx, word2idx[words[idx - 1]]))
# Next word
if idx < len(words) - 1:
pairs.append((c_idx, word2idx[words[idx + 1]]))
# 3. Train Skip-Gram with Negative Sampling (SGNS) over 150 iterations
LR = 0.05
for epoch in range(150):
for center_w, context_w in pairs:
# A. Positive Pair Logistic Regression: y = 1
v_c = W_center[center_w]
v_ctx = W_context[context_w]
score_pos = np.dot(v_ctx, v_c)
grad_pos = (sigmoid(score_pos) - 1.0) # Derivative of log-sigmoid
# B. Negative Sampling Logistic Regression: y = 0
neg_samples = np.random.choice(VOCAB_SIZE, size=K_NEGATIVES, replace=True)
# Update weights via gradient descent
W_center[center_w] -= LR * (grad_pos * v_ctx)
W_context[context_w] -= LR * (grad_pos * v_c)
for neg_w in neg_samples:
if neg_w == context_w:
continue
v_neg = W_context[neg_w]
score_neg = np.dot(v_neg, v_c)
grad_neg = sigmoid(score_neg)
W_center[center_w] -= LR * (grad_neg * v_neg)
W_context[neg_w] -= LR * (grad_neg * v_c)
# Normalize learned center embeddings to unit length for cosine checks
normalized_W = W_center / (np.linalg.norm(W_center, axis=1, keepdims=True) + 1e-12)
# Verify learned cosine similarity between semantically related words vs unrelated
def cosine_sim(w1, w2):
i1, i2 = word2idx[w1], word2idx[w2]
return np.dot(normalized_W[i1], normalized_W[i2])
sim_bank_loan = cosine_sim("bank", "loan")
sim_bank_risk = cosine_sim("bank", "risk")
sim_money_deposits = cosine_sim("money", "deposits")
print("=" * 65)
print("WORD2VEC SKIP-GRAM WITH NEGATIVE SAMPLING (SGNS) AUDIT")
print("=" * 65)
print(f"Corpus Vocabulary Size (|V|) : {VOCAB_SIZE} unique words")
print(f"Embedding Space Dimensions : {EMBED_DIM} features")
print(f"Total Positive Pairs Mined : {len(pairs)} center-context tuples")
print("-" * 65)
print("Learned Semantic Proximities (Cosine Similarities):")
print(f" Similarity('bank', 'loan') : {sim_bank_loan:+.4f} (Frequent co-occurrence)")
print(f" Similarity('money', 'deposits') : {sim_money_deposits:+.4f} (Frequent co-occurrence)")
print(f" Similarity('bank', 'risk') : {sim_bank_risk:+.4f} (Distant context)")
print("-" * 65)
print("SGNS Optimization Benefit:")
print(f" Instead of evaluating Softmax over all {VOCAB_SIZE} vocabulary words, Negative Sampling")
print(f" evaluates 1 positive + {K_NEGATIVES} negative dot products, slashing compute by O(|V|/K).")
import torch
import torch.nn.functional as F
import pandas as pd
# Simulate Transformer Encoder Output:
# Batch=2 sentences, SequenceLength=5 tokens (including padding), Hidden Dimension D=4
# Sentence 1: "financial market boom" (Length 3 + 2 padding tokens)
# Sentence 2: "interest rates increase sharply" (Length 4 + 1 padding token)
torch.manual_seed(42)
B, S, D = 2, 5, 4
# Synthetic contextual token vectors emitted by last transformer hidden layer
last_hidden_states = torch.randn(B, S, D)
# Attention masks: 1 = real token, 0 = padding token
attention_mask = torch.tensor([
[1, 1, 1, 0, 0], # Sentence 1 (3 active tokens, 2 padding)
[1, 1, 1, 1, 0] # Sentence 2 (4 active tokens, 1 padding)
])
# ==========================================================
# 1. CLS POOLING (Takes first token representation [0])
# ==========================================================
cls_embeddings = last_hidden_states[:, 0, :]
# ==========================================================
# 2. MEAN POOLING WITH ATTENTION MASK EXCLUSION
# Formula: sum(tokens * mask) / sum(mask)
# ==========================================================
def masked_mean_pooling(token_embeddings, attn_mask):
# Expand attention mask to match embedding dimensions: (Batch, SeqLen, D)
input_mask_expanded = attn_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
# Zero-out padding token vectors
sum_embeddings = torch.sum(token_embeddings * input_mask_expanded, dim=1)
# Calculate sum of active tokens, clamped to prevent zero division
sum_mask = torch.clamp(input_mask_expanded.sum(dim=1), min=1e-9)
# Return average vector across valid tokens
return sum_embeddings / sum_mask
mean_embeddings = masked_mean_pooling(last_hidden_states, attention_mask)
# 3. L2 Normalize to Unit Sphere (Cosine Similarity = Dot Product)
cls_normalized = F.normalize(cls_embeddings, p=2, dim=1)
mean_normalized = F.normalize(mean_embeddings, p=2, dim=1)
# Measure Cosine Similarity between Sentence 1 and Sentence 2 under both pooling methods
sim_cls = torch.dot(cls_normalized[0], cls_normalized[1]).item()
sim_mean = torch.dot(mean_normalized[0], mean_normalized[1]).item()
print("=" * 65)
print("CONTEXTUAL TRANSFORMER EMBEDDING POOLING AUDIT")
print("=" * 65)
print(f"Token Representation Tensor : (Batch={B}, SeqLen={S}, Dim={D})")
print("-" * 65)
print("CLS Token Extraction Vector (Sentence 1):")
print(f" {cls_normalized[0].numpy().round(4)}")
print("Masked Mean Pooled Vector (Sentence 1):")
print(f" {mean_normalized[0].numpy().round(4)}")
print("-" * 65)
print(f"Pairwise Inter-Sentence Cosine Similarity (CLS Pooling) : {sim_cls:+.4f}")
print(f"Pairwise Inter-Sentence Cosine Similarity (Mean Pooling): {sim_mean:+.4f}")
print("-" * 65)
print("Engineering Insight: Reimers & Gurevych (Sentence-BERT) proved that")
print("raw [CLS] tokens from un-finetuned encoders yield poor semantic clusters.")
print("Attention-masked Mean Pooling aggregates signals across all content tokens.")
import time
import numpy as np
import pandas as pd
from sklearn.neighbors import NearestNeighbors
# Scenario: Enterprise Knowledge Base with 25,000 document embedding chunks
np.random.seed(42)
N_DOCUMENTS = 25000
DIM = 64
K_RETRIEVAL = 5
# Generate normalized corpus vectors (on unit hypersphere)
corpus_embeddings = np.random.randn(N_DOCUMENTS, DIM).astype(np.float32)
corpus_embeddings /= np.linalg.norm(corpus_embeddings, axis=1, keepdims=True)
# Incoming user query vector
query_vec = np.random.randn(1, DIM).astype(np.float32)
query_vec /= np.linalg.norm(query_vec, axis=1, keepdims=True)
# ==========================================================
# 1. EXACT FLAT BRUTE-FORCE VECTOR SEARCH (Linear Scan)
# ==========================================================
t0 = time.perf_counter()
# Cosine similarity via matrix multiplication against all 25k records
cosine_similarities = np.dot(corpus_embeddings, query_vec.T).flatten()
# Top-K indices via argpartition
exact_top_k = np.argpartition(cosine_similarities, -K_RETRIEVAL)[-K_RETRIEVAL:]
exact_top_k = exact_top_k[np.argsort(cosine_similarities[exact_top_k])[::-1]]
time_exact = (time.perf_counter() - t0) * 1000.0 # ms
# ==========================================================
# 2. APPROXIMATE NEAREST NEIGHBORS (KD/BALL TREE SPATIAL GRAPH)
# ==========================================================
t0 = time.perf_counter()
# BallTree spatial partitioning index
tree_index = NearestNeighbors(n_neighbors=K_RETRIEVAL, algorithm="ball_tree", metric="euclidean")
tree_index.fit(corpus_embeddings)
time_index_build = time.perf_counter() - t0
t0 = time.perf_counter()
_, ann_top_k = tree_index.kneighbors(query_vec)
time_ann_query = (time.perf_counter() - t0) * 1000.0 # ms
# Measure top-K recall fidelity
recall = len(set(exact_top_k).intersection(set(ann_top_k[0]))) / K_RETRIEVAL
df_search = pd.DataFrame([
{
"Indexing_Engine": "Exact Flat Search (Brute-Force)",
"Search_Complexity": f"O(N * D) = {N_DOCUMENTS * DIM:,} ops",
"Query_Latency_ms": round(time_exact, 3),
"Recall_at_K": "100.0%",
"Throughput_Speed": "1.0x (Baseline)"
},
{
"Indexing_Engine": "Spatial Graph / Tree (ANN)",
"Search_Complexity": f"O(D * log N) β {int(DIM * np.log2(N_DOCUMENTS)):,} ops",
"Query_Latency_ms": round(time_ann_query, 3),
"Recall_at_K": f"{recall * 100:.1f}%",
"Throughput_Speed": f"{time_exact / time_ann_query:.1f}x Faster"
}
])
print("=" * 65)
print(f"VECTOR DATABASE SEARCH BENCHMARK ({N_DOCUMENTS:,} CHUNKS, D={DIM})")
print("=" * 65)
print(df_search.to_string(index=False))
print("-" * 65)
print("Top 3 Exact Nearest Document IDs & Scores:")
for rank, doc_id in enumerate(exact_top_k[:3], start=1):
print(f" Rank {rank}: Document_Chunk #{doc_id:05d} βββΊ Cosine Similarity = {cosine_similarities[doc_id]:.4f}")
print("-" * 65)
print("Production RAG Note: Modern vector engines (Qdrant, Pinecone, pgvector)")
print("employ HNSW graph indexing to maintain sub-10ms latencies across millions of vectors.")
import numpy as np
import pandas as pd
# Knowledge corpus: Internal corporate cloud security guidelines
knowledge_base = [
{"doc_id": 101, "text": "PostgreSQL database passwords must be rotated every 90 days via HashiCorp Vault."},
{"doc_id": 102, "text": "Production Kubernetes ingress controllers require TLS 1.3 encryption with strict HSTS headers."},
{"doc_id": 103, "text": "AWS IAM user accounts must enforce Multi-Factor Authentication (MFA) and zero active API access keys."},
{"doc_id": 104, "text": "Developer laptop hard drives must use FileVault or BitLocker full-disk AES-256 encryption."}
]
# Simulated query
user_query = "What is the policy for database password rotation cycles?"
# 1. STAGE 1: BI-ENCODER DENSE RETRIEVAL (MOCKED VECTOR SIMILARITY)
# In production, embeddings are extracted via models like BGE-Large or OpenAI text-embedding-3
def mock_bi_encoder_score(query, doc_text):
# Overlap word score simulating embedding cosine similarity
q_words = set(query.lower().split())
d_words = set(doc_text.lower().split())
intersection = q_words.intersection(d_words)
return len(intersection) / np.sqrt(len(q_words) * len(d_words))
retrieval_candidates = []
for doc in knowledge_base:
score = mock_bi_encoder_score(user_query, doc["text"])
retrieval_candidates.append({
"doc_id": doc["doc_id"],
"text": doc["text"],
"bi_encoder_similarity": score
})
# Extract Top-3 Candidates from Vector Search
retrieval_candidates = sorted(retrieval_candidates, key=lambda x: x["bi_encoder_similarity"], reverse=True)[:3]
# 2. STAGE 2: CROSS-ENCODER RERANKER (DEEP FULL-ATTENTION CROSS MATCH)
# Evaluates query and passage together through all transformer attention layers
def mock_cross_encoder_rerank(query, doc_text):
# Exact keyword priority boosting + semantic alignment
relevance = 0.1
if "password" in doc_text.lower() and "rotation" in doc_text.lower():
relevance = 0.98
elif "password" in doc_text.lower() or "database" in doc_text.lower():
relevance = 0.45
return relevance
for doc in retrieval_candidates:
doc["rerank_cross_score"] = mock_cross_encoder_rerank(user_query, doc["text"])
# Sort by cross-encoder score
final_grounded_chunks = sorted(retrieval_candidates, key=lambda x: x["rerank_cross_score"], reverse=True)
# 3. CONTEXT INJECTION (LLM PROMPT GROUNDING)
retrieved_context = "\n".join([f"- [Doc {c['doc_id']}]: {c['text']}" for c in final_grounded_chunks[:2]])
final_llm_prompt = f"""[SYSTEM]: Answer the question using ONLY the retrieved facts below.
[CONTEXT]:
{retrieved_context}
[USER QUESTION]: {user_query}
[GENERATION DIRECTIVE]: Cite specific document IDs."""
df_rag = pd.DataFrame(retrieval_candidates)[["doc_id", "bi_encoder_similarity", "rerank_cross_score", "text"]]
print("=" * 65)
print("TWO-STAGE RAG (RETRIEVAL + RERANKING) PIPELINE AUDIT")
print("=" * 65)
print(df_rag.round(4).to_string(index=False))
print("-" * 65)
print("Grounded LLM Prompt Assembly (Context Injected):")
print(final_llm_prompt)
print("-" * 65)
print("Hallucination Defense: The Reranker promotes Doc 101 to 0.98 relevance,")
print("ensuring the LLM receives verified context, preventing hallucinated timeframes.")
import torch
import torch.nn as nn
import pandas as pd
# Scenario: Fine-tuning a Transformer Attention Projection (d_model = 1024)
IN_FEATURES = 1024
OUT_FEATURES = 1024
LORA_RANK = 8 # Low rank r = 8 (Huge compression!)
LORA_ALPHA = 16.0 # Scaling factor alpha
# ==========================================================
# CUSTOM LORA LINEAR LAYER IMPLEMENTATION
# Forward Equation: h = W_0 * x + (alpha / r) * (B * A) * x
# ==========================================================
class LoRALinear(nn.Module):
def __init__(self, in_features, out_features, r=8, lora_alpha=16.0):
super(LoRALinear, self).__init__()
# 1. Pretrained Base Weight Matrix W_0 (FROZEN)
self.base_layer = nn.Linear(in_features, out_features, bias=False)
self.base_layer.weight.requires_grad = False # Freeze pre-trained weights!
self.r = r
self.lora_alpha = lora_alpha
self.scaling = lora_alpha / r
# 2. Trainable Low-Rank Matrices:
# Matrix A initialized from Gaussian N(0, 1/r)
self.lora_A = nn.Parameter(torch.randn(r, in_features) * (1.0 / np.sqrt(r)))
# Matrix B initialized to exact zeros so ΞW = B @ A = 0 at the start of training!
self.lora_B = nn.Parameter(torch.zeros(out_features, r))
def forward(self, x):
# Base forward pass (No grad required through W_0)
base_out = self.base_layer(x)
# Low-rank adapter branch: x @ A^T @ B^T * scaling
lora_out = (x @ self.lora_A.T @ self.lora_B.T) * self.scaling
return base_out + lora_out
torch.manual_seed(42)
lora_layer = LoRALinear(IN_FEATURES, OUT_FEATURES, r=LORA_RANK, lora_alpha=LORA_ALPHA)
# Count Parameters
base_params = lora_layer.base_layer.weight.numel()
lora_params = lora_layer.lora_A.numel() + lora_layer.lora_B.numel()
trainable_params = sum(p.numel() for p in lora_layer.parameters() if p.requires_grad)
# Verify zero-initialization identity: At step 0, LoRA output must match base output exactly
test_x = torch.randn(2, IN_FEATURES)
base_only_out = lora_layer.base_layer(test_x)
combined_out = lora_layer(test_x)
initial_parity = torch.allclose(base_only_out, combined_out, atol=1e-7)
df_lora = pd.DataFrame([
{
"Component": "Pretrained Base Layer (W_0)",
"Dimensions": f"({OUT_FEATURES}, {IN_FEATURES})",
"Parameters": f"{base_params:,}",
"Requires_Grad": False
},
{
"Component": "LoRA Adapter A",
"Dimensions": f"({LORA_RANK}, {IN_FEATURES})",
"Parameters": f"{lora_layer.lora_A.numel():,}",
"Requires_Grad": True
},
{
"Component": "LoRA Adapter B",
"Dimensions": f"({OUT_FEATURES}, {LORA_RANK})",
"Parameters": f"{lora_layer.lora_B.numel():,}",
"Requires_Grad": True
}
])
print("=" * 65)
print("LOW-RANK ADAPTATION (LoRA) PARAMETER FOOTPRINT AUDIT")
print("=" * 65)
print(df_lora.to_string(index=False))
print("-" * 65)
print(f"Total Base Model Weights (Frozen) : {base_params:,} parameters")
print(f"Total Trainable LoRA Weights (r={LORA_RANK}) : {trainable_params:,} parameters")
print(f"VRAM Gradient Memory Reduction : {(1.0 - trainable_params / base_params) * 100:.2f}% fewer gradients!")
print(f"Zero Initialization Parity Check : {initial_parity} (Output unchanged at step 0)")
print("-" * 65)
print("Production Serving: In deployment, weights can be merged algebraically:")
print("W_deployed = W_0 + (alpha/r) * (B @ A) -> Zero additional inference latency!")
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.pipeline import Pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, roc_auc_score
# Customer sentiment feedback logs across digital banking apps
feedback_data = [
"Mobile app transfers are completely seamless and instant",
"Customer service resolved my dispute very quickly great experience",
"The new update is extremely slow and crashes constantly",
"Fraud department blocked my card without warning terrible service",
"Easy to navigate clean UI and intuitive account summary",
"App keeps disconnecting during payment authorization very frustrating",
"Helpful support agents resolved my login problem within minutes",
"Horrible banking app transaction failed and account debited twice",
"Biometric face login works flawlessly very convenient feature",
"Unresponsive support ticketing system waste of valuable time"
]
sentiments = [1, 1, 0, 0, 1, 0, 1, 0, 1, 0] # 1 = Positive, 0 = Negative
df_sentiment = pd.DataFrame({"text": feedback_data, "target": sentiments})
# 1. STRICT ISOLATION: Split raw text strings BEFORE tokenization/vectorization
X_train, X_test, y_train, y_test = train_test_split(
df_sentiment["text"],
df_sentiment["target"],
test_size=0.30,
stratify=df_sentiment["target"],
random_state=42
)
# 2. Build Leakage-Free Scikit-Learn Pipeline
# TfidfVectorizer computes vocabulary and Inverse Document Frequency (IDF)
# strictly from training folds during fit()
nlp_pipeline = Pipeline([
("vectorizer", TfidfVectorizer(ngram_range=(1, 2), stop_words="english")),
("classifier", LogisticRegression(random_state=42))
])
# Fit pipeline on training split
nlp_pipeline.fit(X_train, y_train)
# Evaluate on held-out test split
y_pred_test = nlp_pipeline.predict(X_test)
y_pred_probs = nlp_pipeline.predict_proba(X_test)[:, 1]
# Vocabulary diagnostics
vocab_size_train = len(nlp_pipeline.named_steps["vectorizer"].vocabulary_)
test_tokens = set([t for s in X_test for t in s.lower().split()])
train_tokens = set(nlp_pipeline.named_steps["vectorizer"].vocabulary_.keys())
oov_tokens = test_tokens - train_tokens
print("=" * 65)
print("LEAKAGE-FREE NLP PIPELINE & OOV GENERALIZATION AUDIT")
print("=" * 65)
print(f"Training Documents Partition : {len(X_train)} samples")
print(f"Testing Documents Partition : {len(X_test)} samples")
print("-" * 65)
print(f"Learned Training Vocabulary : {vocab_size_train} distinct n-grams")
print(f"Unseen Out-Of-Vocabulary Words: {len(oov_tokens)} tokens {list(oov_tokens)[:4]}")
print(f"Held-Out Test Set Accuracy : {nlp_pipeline.score(X_test, y_test)*100:.2f}%")
print(f"Held-Out Test ROC-AUC : {roc_auc_score(y_test, y_pred_probs):.4f}")
print("-" * 65)
print("Integrity Check: Inverse Document Frequency (IDF) weights were fitted strictly")
print("on training records. Test set vocabulary distributions remained unobserved.")
import torch
import torch.nn.functional as F
import numpy as np
import pandas as pd
# Simulated next-token prediction logits across 8 candidate vocabulary tokens
vocab_tokens = ["algorithm", "model", "network", "system", "banana", "galaxy", "pizza", "shoe"]
logits = torch.tensor([5.2, 4.8, 3.9, 3.1, 0.4, -0.8, -1.2, -2.5], dtype=torch.float32)
# ==========================================================
# 1. GREEDY DECODING (Argmax)
# ==========================================================
greedy_idx = torch.argmax(logits).item()
greedy_token = vocab_tokens[greedy_idx]
# ==========================================================
# 2. TOP-K FILTERING (k = 3)
# Keeps only the top 3 highest logits; masks the rest to -inf
# ==========================================================
def top_k_sampling(raw_logits, k=3):
indices_to_remove = raw_logits < torch.topk(raw_logits, k)[0][..., -1, None]
filtered_logits = raw_logits.clone()
filtered_logits[indices_to_remove] = float("-inf")
return F.softmax(filtered_logits, dim=-1)
top_k_probs = top_k_sampling(logits, k=3).numpy()
# ==========================================================
# 3. TOP-P (NUCLEUS) SAMPLING (p = 0.85)
# Keeps the smallest set of tokens whose cumulative probability exceeds p
# ==========================================================
def top_p_sampling(raw_logits, p=0.85):
sorted_logits, sorted_indices = torch.sort(raw_logits, descending=True)
cumulative_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
# Remove tokens with cumulative probability above the threshold p
sorted_indices_to_remove = cumulative_probs > p
# Shift indices right to keep the first token that crossed the threshold
sorted_indices_to_remove[..., 1:] = sorted_indices_to_remove[..., :-1].clone()
sorted_indices_to_remove[..., 0] = 0
# Scatter back to original logit order
indices_to_remove = sorted_indices_to_remove.scatter(0, sorted_indices, sorted_indices_to_remove)
filtered_logits = raw_logits.clone()
filtered_logits[indices_to_remove] = float("-inf")
return F.softmax(filtered_logits, dim=-1)
top_p_probs = top_p_sampling(logits, p=0.85).numpy()
raw_probs = F.softmax(logits, dim=-1).numpy()
df_decoding = pd.DataFrame({
"Candidate_Token": vocab_tokens,
"Raw_Logit": logits.numpy(),
"Base_Probability": np.round(raw_probs, 4),
"Top_K_Prob (k=3)": np.round(top_k_probs, 4),
"Top_P_Prob (p=0.85)": np.round(top_p_probs, 4)
})
print("=" * 65)
print("LLM AUTOREGRESSIVE DECODING STRATEGY COMPARISON")
print("=" * 65)
print(df_decoding.to_string(index=False))
print("-" * 65)
print(f"Greedy Decoding Selection : ['{greedy_token}'] (Deterministic argmax)")
print("Top-k (k=3) Tail Behavior : Tail tokens ('banana', 'galaxy') strictly zeroed out.")
print("Top-p (Nucleus) Adaptability : Dynamically included 3 semantically aligned tokens,")
print(" re-allocating probability mass over the 85% confidence pool.")
NLP & LLM ARCHITECTURAL PARADIGMS COMPARISON:
ββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββ
β Paradigm / Component β Mathematical Formulation β Key Hyperparameters β Primary Production Role β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β Byte-Pair Encoding (BPE) β Iterative frequency merge β vocab_size β Universal tokenization for LLMs β
β β of consecutive byte pairs β special_tokens β (GPT, LLaMA); eliminates OOV β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β Word2Vec (Skip-Gram SGNS) β Maximize log-sigmoid dot β window_size, β Static word associations, low- β
β β products with k negatives β negative_samples (k) β compute recommender embeddings β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β Contextual Embeddings β Multi-layer self-attention β pooling='mean', β Semantic search, RAG retrieval, β
β (Sentence-BERT / BGE / E5) β Mean pooling over tokens β normalize=True β document clustering & dedupe β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β HNSW Vector Index β Multi-layer geometric skip- β M (connections), β High-throughput vector search β
β (Hierarchical Small World) β list graph: O(log N) search β efConstruction, ef β databases (Qdrant, Pinecone) β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β Two-Stage RAG Pipeline β Dense Bi-Encoder Retrieval + β top_k_retrieval, β Enterprise knowledge grounding, β
β β Cross-Encoder Joint Rerank β top_k_rerank β hallucination prevention in LLMs β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β Low-Rank Adaptation (LoRA) β W = W_0 + (Ξ±/r) * (B @ A) β rank (r = 8-64), β Parameter-efficient fine-tuning β
β β B initialized 0, A Gaussian β lora_alpha (2*r) β (PEFT) on consumer GPU VRAM β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββ€
β Top-p (Nucleus) Decoding β Cumulative prob cutoff: β temperature (T), β Natural autoregressive text β
β β β_{i β V^(p)} P(x_i) β₯ p β top_p (0.8 - 0.95) β generation; prevents loops β
ββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββ€
β NLP & LLM PRODUCTION BEST PRACTICES: β
β 1. In RAG pipelines, ALWAYS normalize embedding vectors (L2 norm) so Cosine Similarity reduces to fast dot products. β
β 2. Pair Bi-Encoder vector retrieval with a Cross-Encoder Reranker to maximize context precision before LLM prompting. β
β 3. For LLM fine-tuning, use LoRA (r=8 or 16) targeting attention projections (q_proj, v_proj) to save 90%+ VRAM. β
β 4. Fit text tokenizers and TF-IDF models strictly on training splits to prevent vocabulary and IDF leakage. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ