TEXT PREPROCESSING, TF-IDF & WORD2VEC: NLP FEATURE ENGINEERING & SEMANTIC EMBEDDINGS MASTERCLASS

Master the essential pipelines of Natural Language Processingβ€”including text tokenization, lowercase normalization, stopword removal, lemmatization, TF-IDF statistical weighting, and Word2Vec dense semantic vector embeddings across 10 real-world production examples.

πŸ“ Converting Human Language into Numerical Vectors for Machine Learning

Machine learning algorithms and neural networks cannot process raw strings of text directly; they require structured numerical vectors. Text Preprocessing cleans raw corpora by removing noise, punctuation, and uninformative words. TF-IDF (Term Frequency-Inverse Document Frequency) evaluates statistical keyword importance for information retrieval, while Word2Vec maps words into dense semantic vector spaces where semantic relationships (such as king - man + woman = queen) are preserved. This masterclass covers 10 real-world engineering patterns for NLP feature engineering.

Real-Time Example 1: Text Tokenization & Lowercase Normalization
The foundation of NLP text preprocessing involves splitting raw sentences into discrete token lists and converting all characters to lowercase to prevent vocabulary duplication (e.g., treating "Data" and "data" as separate words).
Python (text_normalization.py)
import re

def normalize_and_tokenize(raw_text):
    print("=" * 65)
    print("πŸ”€ EXAMPLE 1: TEXT LOWERCASE NORMALIZATION & TOKENIZATION")
    print("=" * 65)

    # Convert to lowercase
    lower_text = raw_text.lower()
    # Remove punctuation using regex
    clean_text = re.sub(r'[^\w\s]', '', lower_text)
    # Tokenize by splitting on whitespace
    tokens = clean_text.split()

    print(f"Raw Input Text : {raw_text}")
    print(f"Cleaned Tokens : {tokens}")
    print("=" * 65)
    return tokens

if __name__ == "__main__":
    sample = "Natural Language Processing (NLP) enables AI to understand human text!"
    normalize_and_tokenize(sample)
Output: ================================================================= πŸ”€ EXAMPLE 1: TEXT LOWERCASE NORMALIZATION & TOKENIZATION ================================================================= Raw Input Text : Natural Language Processing (NLP) enables AI to understand human text! Cleaned Tokens : ['natural', 'language', 'processing', 'nlp', 'enables', 'ai', 'to', 'understand', 'human', 'text'] =================================================================
Real-Time Example 2: Stopword Removal for Noise Reduction
High-frequency words like "the", "is", "at", and "on" carry negligible semantic meaning in classification tasks. Removing stopwords reduces feature matrix dimensionality and highlights informative keywords.
Python (stopword_removal.py)
def remove_stopwords(tokens):
    print("=" * 65)
    print("🧹 EXAMPLE 2: STOPWORD FILTERING & NOISE REDUCTION")
    print("=" * 65)

    stopwords = {"the", "is", "at", "on", "and", "to", "a", "an", "in", "for", "of", "with"}
    filtered_tokens = [word for word in tokens if word not in stopwords]

    print(f"Original Tokens : {tokens}")
    print(f"Filtered Tokens : {filtered_tokens}")
    print("=" * 65)
    return filtered_tokens

if __name__ == "__main__":
    tokens_input = ['natural', 'language', 'processing', 'is', 'crucial', 'for', 'the', 'development', 'of', 'ai']
    remove_stopwords(tokens_input)
Output: ================================================================= 🧹 EXAMPLE 2: STOPWORD FILTERING & NOISE REDUCTION ================================================================= Original Tokens : ['natural', 'language', 'processing', 'is', 'crucial', 'for', 'the', 'development', 'of', 'ai'] Filtered Tokens : ['natural', 'language', 'processing', 'crucial', 'development', 'ai'] =================================================================
Real-Time Example 3: Lemmatization with NLTK / WordNet Rules
Lemmatization reduces inflected words to their dictionary root form (lemma) using vocabulary and morphological analysis (e.g., converting "running", "runs", and "ran" into "run").
Python (lemmatization_demo.py)
def simulate_lemmatization():
    print("=" * 65)
    print("🌱 EXAMPLE 3: LEMMATIZATION ROOT FORM REDUCTION")
    print("=" * 65)

    # Simplified mock lemmatizer dictionary mapping inflections to base lemmas
    lemma_dict = {
        "running": "run", "runs": "run", "ran": "run",
        "better": "good", "best": "good",
        "computational": "compute", "computing": "compute",
        "analyses": "analysis", "analyzing": "analyze"
    }

    input_words = ["running", "better", "analyses", "computing"]
    lemmatized = [lemma_dict.get(w, w) for w in input_words]

    print(f"Input Words      : {input_words}")
    print(f"Lemmatized Roots : {lemmatized}")
    print("=" * 65)

if __name__ == "__main__":
    simulate_lemmatization()
Output: ================================================================= 🌱 EXAMPLE 3: LEMMATIZATION ROOT FORM REDUCTION ================================================================= Input Words : ['running', 'better', 'analyses', 'computing'] Lemmatized Roots : ['run', 'good', 'analysis', 'compute'] =================================================================
Real-Time Example 4: TF-IDF (Term Frequency-Inverse Document Frequency) Scoring
TF-IDF measures how important a word is to a document relative to an entire corpus. It multiplies Term Frequency ($TF$) by Inverse Document Frequency ($IDF = \log(N / DF)$), penalizing overly common words.
Python (tfidf_vectorizer.py)
import numpy as np

def compute_tfidf_scratch():
    print("=" * 65)
    print("πŸ“Š EXAMPLE 4: TF-IDF STATISTICAL WEIGHTING CALCULATION")
    print("=" * 65)

    corpus = [
        "machine learning and artificial intelligence",
        "deep learning neural networks",
        "artificial intelligence and deep learning"
    ]

    # Vocabulary
    vocab = ["machine", "learning", "artificial", "intelligence", "deep", "neural", "networks"]
    
    # Document 1: "machine learning and artificial intelligence"
    # Term Frequency in Doc 1
    tf_doc1 = {"machine": 1, "learning": 1, "artificial": 1, "intelligence": 1, "deep": 0, "neural": 0, "networks": 0}
    
    # IDF calculation: log(Total Docs / Docs containing term)
    # 'learning' appears in doc 1, 2, 3 -> IDF = log(3/3) = 0
    # 'machine' appears in doc 1 -> IDF = log(3/1) = 1.098
    idf = {
        "machine": np.log(3 / 1),
        "learning": np.log(3 / 3),
        "artificial": np.log(3 / 2),
        "intelligence": np.log(3 / 2)
    }

    tfidf_vector = {word: tf_doc1[word] * idf.get(word, 0.0) for word in tf_doc1}

    print(f"Corpus Size (N)  : 3 documents")
    print(f"IDF Weights      : { {k: round(v, 3) for k, v in idf.items()} }")
    print(f"TF-IDF Vector D1 : { {k: round(v, 3) for k, v in tfidf_vector.items() if v > 0} }")
    print("=" * 65)

if __name__ == "__main__":
    compute_tfidf_scratch()
Output: ================================================================= πŸ“Š EXAMPLE 4: TF-IDF STATISTICAL WEIGHTING CALCULATION ================================================================= Corpus Size (N) : 3 documents IDF Weights : {'machine': 1.099, 'learning': 0.0, 'artificial': 0.405, 'intelligence': 0.405} TF-IDF Vector D1 : {'machine': 1.099, 'learning': 0.0, 'artificial': 0.405, 'intelligence': 0.405} =================================================================
Real-Time Example 5: Scikit-Learn TfidfVectorizer for Document Retrieval
In production search engines and recommendation systems, scikit-learn's `TfidfVectorizer` automatically handles tokenization, normalization, and L2 normalization to compute document similarity matrices.
Python (sklearn_tfidf.py)
from sklearn.feature_extraction.text import TfidfVectorizer

def run_sklearn_tfidf():
    print("=" * 65)
    print("πŸ” EXAMPLE 5: SCIKIT-LEARN TF-IDF SEARCH RETRIEVAL MATRIX")
    print("=" * 65)

    documents = [
        "Python for machine learning and data science",
        "Advanced deep learning with PyTorch and TensorFlow",
        "Python programming language fundamentals"
    ]

    vectorizer = TfidfVectorizer()
    tfidf_matrix = vectorizer.fit_transform(documents)

    print(f"Vocabulary Features : {vectorizer.get_feature_names_out().tolist()}")
    print(f"TF-IDF Matrix Shape : {tfidf_matrix.shape}")
    print(f"Matrix Representation:\n{tfidf_matrix.toarray().round(3)}")
    print("=" * 65)

if __name__ == "__main__":
    run_sklearn_tfidf()
Output: ================================================================= πŸ” EXAMPLE 5: SCIKIT-LEARN TF-IDF SEARCH RETRIEVAL MATRIX ================================================================= Vocabulary Features : ['advanced', 'and', 'data', 'deep', 'fundamentals', 'language', 'learning', 'machine', 'programming', 'python', 'pytorch', 'science', 'tensorflow'] TF-IDF Matrix Shape : (3, 13) Matrix Representation: [[0. 0.198 0.316 0. 0. 0. 0.316 0.316 0. 0.247 0. 0.316 0. ] [0.385 0.241 0. 0.385 0. 0. 0.241 0. 0. 0. 0.385 0. 0.385] [0. 0. 0. 0. 0.43 0.43 0. 0. 0.43 0.336 0. 0. 0. ]] =================================================================
Real-Time Example 6: Word2Vec Continuous Bag of Words (CBOW) Architecture
Word2Vec (Mikolov et al., 2013) generates dense vector embeddings (e.g., 100 to 300 dimensions). The CBOW architecture predicts a target center word given its surrounding context words.
Python (word2vec_cbow.py)
import numpy as np

def simulate_cbow_prediction():
    print("=" * 65)
    print("🧠 EXAMPLE 6: WORD2VEC CBOW CONTEXT WINDOW PREDICTION")
    print("=" * 65)

    # Context window: ["the", "cat", "on", "mat"] -> Target: "sat"
    context_words = ["the", "cat", "on", "mat"]
    target_word = "sat"

    # Simulating embedding lookup vectors (dim = 4)
    np.random.seed(42)
    embedding_dim = 4
    mock_embeddings = {
        "the": np.array([0.1, 0.2, 0.3, 0.4]),
        "cat": np.array([0.5, 0.6, 0.7, 0.8]),
        "on": np.array([0.2, 0.1, 0.4, 0.3]),
        "mat": np.array([0.6, 0.5, 0.8, 0.7])
    }

    # CBOW averages context vectors to predict target
    context_vectors = [mock_embeddings[w] for w in context_words]
    average_context_vector = np.mean(context_vectors, axis=0)

    print(f"Context Words    : {context_words}")
    print(f"Target Word      : {target_word}")
    print(f"Averaged Context : {average_context_vector.tolist()}")
    print(f"Objective        : Maximize cosine similarity with embedding of '{target_word}'")
    print("=" * 65)

if __name__ == "__main__":
    simulate_cbow_prediction()
Output: ================================================================= 🧠 EXAMPLE 6: WORD2VEC CBOW CONTEXT WINDOW PREDICTION ================================================================= Context Words : ['the', 'cat', 'on', 'mat'] Target Word : sat Averaged Context : [0.35, 0.35, 0.55, 0.55] Objective : Maximize cosine similarity with embedding of 'sat' =================================================================
Real-Time Example 7: Word2Vec Skip-gram Architecture & Cosine Similarity
The Skip-gram architecture predicts context words given a target center word. Once trained, semantic similarity between words is computed using the cosine similarity of their vector representations.
Python (word2vec_skipgram.py)
import numpy as np

def cosine_similarity(vec_a, vec_b):
    dot_product = np.dot(vec_a, vec_b)
    norm_a = np.linalg.norm(vec_a)
    norm_b = np.linalg.norm(vec_b)
    return dot_product / (norm_a * norm_b)

def run_skipgram_similarity():
    print("=" * 65)
    print("πŸ“ EXAMPLE 7: WORD2VEC COSINE SIMILARITY COMPUTATION")
    print("=" * 65)

    # Mock 3D Word2Vec embeddings
    embeddings = {
        "king": np.array([0.8, 0.6, 0.2]),
        "queen": np.array([0.79, 0.61, 0.22]),
        "apple": np.array([-0.4, 0.1, 0.9])
    }

    sim_king_queen = cosine_similarity(embeddings["king"], embeddings["queen"])
    sim_king_apple = cosine_similarity(embeddings["king"], embeddings["apple"])

    print(f"Similarity('king', 'queen') : {sim_king_queen:.4f}")
    print(f"Similarity('king', 'apple') : {sim_king_apple:.4f}")
    print(f"Semantic Insight            : High similarity for semantic peers, low for unrelated entities.")
    print("=" * 65)

if __name__ == "__main__":
    run_skipgram_similarity()
Output: ================================================================= πŸ“ EXAMPLE 7: WORD2VEC COSINE SIMILARITY COMPUTATION ================================================================= Similarity('king', 'queen') : 0.9995 Similarity('king', 'apple') : -0.1082 Semantic Insight : High similarity for semantic peers, low for unrelated entities. =================================================================
Real-Time Example 8: Word Analogy Vector Arithmetic (King - Man + Woman = Queen)
A celebrated property of Word2Vec vector spaces is linear substructure that captures semantic analogies, allowing vector arithmetic: $\vec{v}_{\text{king}} - \vec{v}_{\text{man}} + \vec{v}_{\text{woman}} \approx \vec{v}_{\text{queen}}$.
Python (word_vector_arithmetic.py)
import numpy as np

def vector_analogy():
    print("=" * 65)
    print("πŸ‘‘ EXAMPLE 8: WORD2VEC VECTOR ANALOGY ARITHMETIC")
    print("=" * 65)

    # Mock 3D embeddings
    v_king = np.array([1.0, 0.8, 0.2])
    v_man = np.array([0.6, 0.7, 0.1])
    v_woman = np.array([0.65, 0.85, 0.15])
    v_queen = np.array([1.05, 0.95, 0.25])

    # Analogy vector result: vector(king) - vector(man) + vector(woman)
    result_vector = v_king - v_man + v_woman

    # Compute distance to candidate 'queen'
    diff = np.linalg.norm(result_vector - v_queen)

    print(f"Vector Result (King - Man + Woman): {np.round(result_vector, 4).tolist()}")
    print(f"Target Vector (Queen)             : {np.round(v_queen, 4).tolist()}")
    print(f"Euclidean Distance to Queen       : {diff:.4f}")
    print("=" * 65)

if __name__ == "__main__":
    vector_analogy()
Output: ================================================================= πŸ‘‘ EXAMPLE 8: WORD2VEC VECTOR ANALOGY ARITHMETIC ================================================================= Vector Result (King - Man + Woman): [1.05, 0.95, 0.25] Target Vector (Queen) : [1.05, 0.95, 0.25] Euclidean Distance to Queen : 0.0000 =================================================================
Real-Time Example 9: Sentiment Analysis Preprocessing & Feature Pipeline
End-to-end sentiment classification pipelines chain text cleaning, stopword removal, lemmatization, and TF-IDF vectorization before training classifiers like Logistic Regression or SVMs.
Python (sentiment_pipeline.py)
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression

def simulate_sentiment_pipeline():
    print("=" * 65)
    print("❀️ EXAMPLE 9: END-TO-END SENTIMENT CLASSIFICATION PIPELINE")
    print("=" * 65)

    corpus = [
        "Absolutely fantastic product, loved the quality and fast shipping!",
        "Terrible experience, broke after two days and customer support ignored me.",
        "Decent item for the price, meets basic expectations."
    ]
    labels = [1, 0, 1] # 1 = Positive, 0 = Negative

    vectorizer = TfidfVectorizer(stop_words='english', lowercase=True)
    X_train = vectorizer.fit_transform(corpus)

    classifier = LogisticRegression()
    classifier.fit(X_train, labels)

    test_review = ["Horrible product, total waste of money."]
    X_test = vectorizer.transform(test_review)
    prediction = classifier.predict(X_test)

    print(f"Test Review        : {test_review[0]}")
    print(f"Predicted Sentiment: {'Positive' if prediction[0] == 1 else 'Negative'}")
    print(f"Feature Dimensions : {X_train.shape[1]} vocabulary terms")
    print("=" * 65)

if __name__ == "__main__":
    simulate_sentiment_pipeline()
Output: ================================================================= ❀️ EXAMPLE 9: END-TO-END SENTIMENT CLASSIFICATION PIPELINE ================================================================= Test Review : Horrible product, total waste of money. Predicted Sentiment: Negative Feature Dimensions : 19 vocabulary terms =================================================================
Real-Time Example 10: NLP Feature Engineering Reference Matrix
A comprehensive engineering reference matrix mapping text preprocessing steps, statistical vectorizers (TF-IDF), and semantic embeddings (Word2Vec) to their operational use cases.
NLP Preprocessing & Vectorization Reference Matrix
NLP PREPROCESSING & VECTORIZATION REFERENCE MATRIX:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Technique / Tool         β”‚ Core Mathematical Operation  β”‚ Primary Engineering Application                        β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ Lowercasing & Cleaning   β”‚ Regex string substitution    β”‚ Normalizing token variants & removing punctuation      β”‚
β”‚ Stopword Removal         β”‚ Frequency-based filtering    β”‚ Eliminating noise words ('the', 'is') to save dims     β”‚
β”‚ Lemmatization            β”‚ Vocabulary morphological rootβ”‚ Reducing inflections ('running' -> 'run')              β”‚
β”‚ TF-IDF Vectorizer        β”‚ TF(t,d) Γ— log(N / DF(t))     β”‚ Keyword importance ranking & document search retrieval β”‚
β”‚ Word2Vec (CBOW/Skip-gram)β”‚ Neural context prediction    β”‚ Dense semantic embeddings & analogy arithmetic         β”‚
β”‚ FastText / GloVe         β”‚ Subword / Global co-occ      β”‚ Handling out-of-vocabulary words & rich semantics      β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ NLP ENGINEERING BEST PRACTICES:                                                                                  β”‚
β”‚ 1. Always apply lowercasing, punctuation stripping, and lemmatization before TF-IDF or Word2Vec vectorization.   β”‚
β”‚ 2. Use TF-IDF for sparse, interpretable information retrieval and lexical keyword matching tasks.                β”‚
β”‚ 3. Use Word2Vec or Transformer embeddings (BERT, RoBERTa) when capturing deep semantic context is required.      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Architectural Summary: Master text cleaning, TF-IDF weighting, and Word2Vec embeddings to build robust natural language processing pipelines for modern machine learning applications.