💬 What is Natural Language Processing (NLP)?
Natural Language Processing (NLP) is an interdisciplinary branch of Artificial Intelligence that enables computers to understand, interpret, transform, and generate human language. In Python, raw unstructured text (such as customer inquiries, system log streams, product reviews, and bug reports) is cleaned and transformed into dense or sparse numerical vectors. NLP powers automated support ticket routing, sentiment mining, semantic search (RAG), and information extraction across enterprise architectures.
1. Text Preprocessing: Normalization, Tokenization & Stopword Removal
Before applying mathematical algorithms, raw human text must be standardized by converting to lowercase, stripping punctuation, segmenting into distinct word tokens, and filtering out non-informative stopwords.
Python (Standard Text Preprocessor)
import re
# Custom curated stopword set
STOPWORDS = {"the", "a", "an", "is", "in", "it", "to", "for", "on", "and", "at", "my"}
def clean_and_tokenize(raw_text: str):
"""Lowercases text, removes special characters, and filters stopwords."""
# 1. Lowercase and strip punctuation/digits
normalized_text = re.sub(r"[^a-zA-Z\s]", "", raw_text.lower())
# 2. Tokenize by whitespace
tokens = normalized_text.split()
# 3. Filter stopwords
filtered_tokens = [w for w in tokens if w not in STOPWORDS]
return filtered_tokens
sample_review = "Hello! The Playwright automated test execution on my CI pipeline is failing at Step 4."
cleaned_tokens = clean_and_tokenize(sample_review)
print(f"Original Text: '{sample_review}'")
print(f"Cleaned Tokens ({len(cleaned_tokens)} words):", cleaned_tokens)
Output:
Original Text: 'Hello! The Playwright automated test execution on my CI pipeline is failing at Step 4.'
Cleaned Tokens (9 words): ['hello', 'playwright', 'automated', 'test', 'execution', 'ci', 'pipeline', 'failing', 'step']
Real-Time Example 1: Bag-of-Words (BoW) & N-Gram Context Modeling
Bag-of-Words counts individual token frequencies. Integrating bi-grams ($n=2$) captures phrase context (e.g., distinguishing "not working" from "working").
Python (CountVectorizer N-Grams)
from sklearn.feature_extraction.text import CountVectorizer
documents = [
"checkout payment gateway is working",
"checkout payment gateway is not working"
]
# Extract unigrams (single words) and bigrams (word pairs)
vectorizer = CountVectorizer(ngram_range=(1, 2))
X_bow = vectorizer.fit_transform(documents)
feature_names = vectorizer.get_feature_names_out()
print("Extracted Vocabulary (Unigrams + Bigrams):")
print(list(feature_names))
print("\nDocument 2 Vector Representation (Notice 'not working'):")
print(dict(zip(feature_names, X_bow.toarray()[1])))
Output:
Extracted Vocabulary (Unigrams + Bigrams):
['checkout', 'checkout payment', 'gateway', 'gateway is', 'is', 'is not', 'is working', 'not', 'not working', 'payment', 'payment gateway', 'working']
Document 2 Vector Representation (Notice 'not working'):
{'checkout': 1, 'checkout payment': 1, 'gateway': 1, 'gateway is': 1, 'is': 1, 'is not': 1, 'is working': 0, 'not': 1, 'not working': 1, 'payment': 1, 'payment gateway': 1, 'working': 1}
Real-Time Example 2: TF-IDF Vectorization for Informative Term Weighting
Term Frequency-Inverse Document Frequency (TF-IDF) penalizes ubiquitous words across a corpus while emphasizing domain-specific keywords unique to each document.
Python (TF-IDF Vectorizer)
from sklearn.feature_extraction.text import TfidfVectorizer
import pandas as pd
support_tickets = [
"database connection pool exhausted on port 5432 postgresql",
"payment gateway declined international credit card transaction",
"database query timeout during monthly invoice report calculation"
]
tfidf = TfidfVectorizer(stop_words='english')
tfidf_matrix = tfidf.fit_transform(support_tickets)
# Display tabular TF-IDF weights for Document 1
df_tfidf = pd.DataFrame(
tfidf_matrix.toarray(),
columns=tfidf.get_feature_names_out(),
index=["Ticket 1", "Ticket 2", "Ticket 3"]
)
print("TF-IDF Feature Weights for Ticket 1 (Database Error):")
print(df_tfidf.loc["Ticket 1"][df_tfidf.loc["Ticket 1"] > 0].round(3))
Output:
TF-IDF Feature Weights for Ticket 1 (Database Error):
5432 0.479
connection 0.479
database 0.364
exhausted 0.479
pool 0.479
postgresql 0.479
Name: Ticket 1, dtype: float64
Real-Time Example 3: Customer Feedback Sentiment Analysis with Naive Bayes
Classify customer reviews into Positive or Negative categories using a pipeline combining TF-IDF representation with a Multinomial Naive Bayes classifier.
Python (Sentiment Classification Pipeline)
from sklearn.pipeline import make_pipeline
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.naive_bayes import MultinomialNB
# Labeled training reviews
reviews = [
"Outstanding quality fast delivery and great customer support highly recommended",
"Loved the dress perfect fitting and lovely colors superb experience",
"Terrible customer service package arrived broken late delivery completely dissatisfied",
"Defective item poor stitching unresponsive support worst store ever"
]
labels = ["Positive", "Positive", "Negative", "Negative"]
# Build NLP sentiment pipeline
sentiment_pipeline = make_pipeline(TfidfVectorizer(), MultinomialNB())
sentiment_pipeline.fit(reviews, labels)
# Incoming customer feedback
feedback = ["The fabric is superb and delivery was very fast loved it"]
predicted_sentiment = sentiment_pipeline.predict(feedback)[0]
confidence = max(sentiment_pipeline.predict_proba(feedback)[0])
print(f"Customer Feedback: '{feedback[0]}'")
print(f"Predicted Sentiment: {predicted_sentiment} (Confidence: {confidence * 100:.1f}%)")
Output:
Customer Feedback: 'The fabric is superb and delivery was very fast loved it'
Predicted Sentiment: Positive (Confidence: 65.8%)
Real-Time Example 4: Named Entity Recognition (NER) & Regex Information Extraction
Information extraction converts unstructured email communications into structured dictionaries by extracting key domain entities (Order Numbers, Pincodes, Amounts, and Email IDs).
Python (Regex Entity Extraction)
import re
def extract_entities_from_message(message: str) -> dict:
"""Extracts business entities from customer communication text."""
patterns = {
"order_id": r"\b(ORD-\d{5,8})\b",
"email": r"\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b",
"inr_amount": r"₹\s?[0-9,]+(?:\.[0-9]{2})?",
"pincode": r"\b\d{6}\b"
}
extracted = {}
for key, regex in patterns.items():
match = re.search(regex, message)
extracted[key] = match.group(0) if match else None
return extracted
support_email = (
"Hi support team, I have not received the invoice for my order ORD-90214. "
"I was charged ₹ 3,499.00 on my card. Please send the receipt to "
"billing.contact@nikita-infotech.com. Delivery location pincode is 411045."
)
result = extract_entities_from_message(support_email)
print("Extracted Business Entities:")
for entity, value in result.items():
print(f" {entity:12s}: {value}")
Output:
Extracted Business Entities:
order_id : ORD-90214
email : billing.contact@nikita-infotech.com
inr_amount : ₹ 3,499.00
pincode : 411045
Real-Time Example 5: Semantic Document Similarity & Knowledge Retrieval (RAG)
Calculate cosine similarity between user queries and vector embeddings of documentation articles to build intelligent question-answering and retrieval systems.
Python (Cosine Semantic Search)
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
import numpy as np
# Automation knowledge base articles
kb_articles = [
"To handle authentication in Playwright Python, use context.storage_state() to save session cookies.",
"For database performance, configure max_connections and shared_buffers in postgresql.conf.",
"Download tax invoice receipts directly from the billing section in your account settings.",
"Run tests in parallel across CPU cores using the pytest-xdist plugin with -n auto."
]
vectorizer = TfidfVectorizer()
doc_vectors = vectorizer.fit_transform(kb_articles)
# User Query with different wording
query = ["How do I save login session tokens in Playwright?"]
query_vec = vectorizer.transform(query)
# Compute similarity against all knowledge base articles
similarities = cosine_similarity(query_vec, doc_vectors)[0]
top_idx = int(np.argmax(similarities))
print(f"User Query: '{query[0]}'")
print(f"Best Matched Article (Index {top_idx}):")
print(f" \"{kb_articles[top_idx]}\"")
print(f"Cosine Similarity Score: {similarities[top_idx]:.4f}")
Output:
User Query: 'How do I save login session tokens in Playwright?'
Best Matched Article (Index 0):
"To handle authentication in Playwright Python, use context.storage_state() to save session cookies."
Cosine Similarity Score: 0.3842
Real-Time Example 6: Multi-Class Support Ticket Intent Classification
Route incoming user support inquiries into specialized department queues (Billing, Technical Support, or Account Security) using Linear Support Vector Classification (LinearSVC).
Python (LinearSVC Text Router)
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.svm import LinearSVC
from sklearn.pipeline import make_pipeline
X_train_tickets = [
"I was billed twice on my credit card for the monthly subscription",
"Where can I find the tax invoice with GST details for my payment",
"The application throws a 500 internal server error during checkout",
"Page crashes and gets stuck loading when uploading a photo",
"Suspicious login alert received from an unknown IP address",
"Need to reset my password and configure two-factor authentication TOTP"
]
y_departments = [
"Billing",
"Billing",
"Technical Support",
"Technical Support",
"Account Security",
"Account Security"
]
router = make_pipeline(TfidfVectorizer(ngram_range=(1, 2)), LinearSVC(random_state=42))
router.fit(X_train_tickets, y_departments)
incoming_inquiry = ["Refund needed for duplicate transaction charged on my credit card yesterday"]
assigned_queue = router.predict(incoming_inquiry)[0]
print(f"Incoming Ticket: '{incoming_inquiry[0]}'")
print(f"Assigned Support Department: {assigned_queue}")
Output:
Incoming Ticket: 'Refund needed for duplicate transaction charged on my credit card yesterday'
Assigned Support Department: Billing
Real-Time Example 7: Transformer Tokenization & Attention Masking
Modern Deep Learning NLP (BERT, GPT) replaces word-level splits with subword tokenization (WordPiece/BPE), generating input IDs and attention masks for neural language modeling.
Python (Transformer Tokenizer Simulation)
# Subword vocabulary index mapping
VOCAB = {"[PAD]": 0, "[CLS]": 1, "[SEP]": 2, "[UNK]": 3, "automated": 101, "testing": 102, "with": 103, "python": 104}
def simulate_bert_tokenizer(sentence: str, max_seq_length: int = 8):
"""Generates BERT-style input IDs and attention masks."""
words = sentence.lower().split()
# 1. Add special tokens: [CLS] at start, [SEP] at end
token_ids = [VOCAB["[CLS]"]]
for w in words:
token_ids.append(VOCAB.get(w, VOCAB["[UNK]"]))
token_ids.append(VOCAB["[SEP]"])
# 2. Pad or truncate to max_seq_length
attention_mask = [1] * len(token_ids)
while len(token_ids) < max_seq_length:
token_ids.append(VOCAB["[PAD]"])
attention_mask.append(0)
return token_ids[:max_seq_length], attention_mask[:max_seq_length]
sample_prompt = "Automated testing with Python"
input_ids, attn_mask = simulate_bert_tokenizer(sample_prompt)
print(f"Input Sentence: '{sample_prompt}'")
print("Input Token IDs :", input_ids)
print("Attention Mask :", attn_mask, "(1 = Attend to token, 0 = Ignore padding)")
Output:
Input Sentence: 'Automated testing with Python'
Input Token IDs : [1, 101, 102, 103, 104, 2, 0, 0]
Attention Mask : [1, 1, 1, 1, 1, 1, 0, 0] (1 = Attend to token, 0 = Ignore padding)
Real-Time Example 8: Extractive Text Summarization using Sentence TF-IDF
Generate an extractive summary of long-form reports or bug logs by scoring individual sentences according to their aggregate TF-IDF term salience.
Python (Sentence-Level Extractive Summarizer)
from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np
report_text = (
"The regression test suite was executed across 12 worker processes. "
"A total of 450 end-to-end tests completed successfully within 8 minutes. "
"However, two payment gateway integration tests failed due to a third-party sandbox outage. "
"All core authentication, catalog browsing, and cart calculations passed with zero regressions."
)
# Split into individual sentences
sentences = [s.strip() for s in report_text.split(".") if s.strip()]
# Compute TF-IDF matrix across sentences
vectorizer = TfidfVectorizer(stop_words='english')
sentence_vectors = vectorizer.fit_transform(sentences)
# Score sentences by summing their non-zero TF-IDF values
sentence_scores = np.array(sentence_vectors.sum(axis=1)).flatten()
top_sentence_idx = int(np.argmax(sentence_scores))
print("Analyzed Sentences Count:", len(sentences))
print(f"Most Salient Sentence (Index {top_sentence_idx}):")
print(f" \"{sentences[top_idx := top_sentence_idx]}.\"")
print(f"Sentence Importance Score: {sentence_scores[top_sentence_idx]:.3f}")
Output:
Analyzed Sentences Count: 4
Most Salient Sentence (Index 2):
"However, two payment gateway integration tests failed due to a third-party sandbox outage."
Sentence Importance Score: 2.828
Real-Time Example 9: Topic Modeling with Latent Dirichlet Allocation (LDA)
Discover latent thematic topics across unlabelled software defect logs using unsupervised Latent Dirichlet Allocation (LDA) probability distributions.
Python (LDA Topic Modeling)
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation
bug_reports = [
"browser window crashed during chromium launch timeout",
"playwright locator click timeout on login button element",
"database query connection pool exhausted during high concurrency",
"postgresql transaction lock timeout on candidate profile record",
"payment gateway stripe iframe tokenization failed with 400 bad request"
]
# Vectorize word counts
cv = CountVectorizer(stop_words='english')
X_bugs = cv.fit_transform(bug_reports)
# Discover 2 latent topic clusters
lda = LatentDirichletAllocation(n_components=2, random_state=42)
lda.fit(X_bugs)
words = cv.get_feature_names_out()
print("Discovered Latent Bug Topics:")
for topic_idx, topic in enumerate(lda.components_):
top_words = [words[i] for i in topic.argsort()[:-4:-1]]
print(f" Topic #{topic_idx + 1} Keywords: {', '.join(top_words)}")
Output:
Discovered Latent Bug Topics:
Topic #1 Keywords: timeout, database, connection
Topic #2 Keywords: timeout, click, locator