EXPLORATORY DATA ANALYSIS (EDA) IN PYTHON

Uncover underlying data patterns, spot anomalies, test hypotheses, and summarize main characteristics using Pandas, Matplotlib, and Seaborn.

🔍 What is Exploratory Data Analysis (EDA)?

Exploratory Data Analysis (EDA) is an indispensable approach used by data scientists to analyze and investigate datasets and summarize their main characteristics, often employing statistical profiling and data visualization methods.

EDA enables developers to spot anomalies, test assumptions, check for missing data, and reveal hidden structural relationships between features before building predictive machine learning models.

1. Structural Inspection (.info() & .dtypes)
Inspect column names, non-null counts, and memory footprints to understand dataset architecture.
Python
import pandas as pd

df = pd.DataFrame({
    "EmployeeID": [101, 102, 103],
    "Name": ["Rahul", "Priya", "Amit"],
    "Salary": [65000.0, 72000.0, 50000.0],
    "Active": [True, True, False]
})

print("Dataset Data Types:\n", df.dtypes)
print("\nTotal Non-Null Counts and Memory:")
print(df.info())
Output: Dataset Data Types: EmployeeID int64 Name object Salary float64 Active bool dtype: object Total Non-Null Counts and Memory: RangeIndex: 3 entries, 0 to 2 Data columns (total 4 columns): # Column Non-Null Count type --- ------ -------------- ----- 0 EmployeeID 3 non-null int64 1 Name 3 non-null object 2 Salary 3 non-null float64 3 Active 3 non-null bool dtypes: bool(1), float64(1), int64(1), object(1) memory usage: 204.0+ bytes
Real-Time Example 1: Summary Descriptive Profiling (.describe())
Generate statistical summaries (mean, standard deviation, quartiles) for numerical features.
Python
import pandas as pd

df = pd.DataFrame({
    "Age": [25, 30, 22, 45, 35],
    "Salary": [50000, 65000, 48000, 95000, 70000]
})

print("Descriptive Statistical Summary:\n", df.describe())
Output: Descriptive Statistical Summary: Age Salary count 5.000000 5.000000 mean 31.400000 65600.000000 std 8.677557 17882.952787 min 22.000000 48000.000000 25% 25.000000 50000.000000 50% 30.000000 65000.000000 75% 35.000000 70000.000000 max 45.000000 95000.000000
Real-Time Example 2: Univariate Distribution Analysis (Histograms / KDE)
Visualize the frequency distribution of a single numerical feature using Seaborn and Matplotlib.
Python
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np

# Generate sample salary distribution
np.random.seed(42)
salaries = np.random.normal(65000, 12000, 200)

sns.histplot(salaries, kde=True, color="#367db5", bins=15)
plt.title("Univariate Salary Distribution Analysis")
plt.xlabel("Salary (₹)")
plt.ylabel("Frequency Count")
plt.show()
Output: [Visual Window]: Generated univariate distribution histogram with a smooth KDE curve.
Real-Time Example 3: Bivariate Scatter Analysis (Feature Relationships)
Explore relationships between two continuous variables using bivariate scatter plots.
Python
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd

df = pd.DataFrame({
    "Experience": [1, 2, 3, 5, 7, 10, 12],
    "Salary": [40000, 48000, 55000, 70000, 85000, 110000, 125000]
})

sns.scatterplot(x="Experience", y="Salary", data=df, color="#16834d", s=100)
plt.title("Bivariate Analysis: Experience vs Salary")
plt.xlabel("Years of Experience")
plt.ylabel("Salary (₹)")
plt.grid(True)
plt.show()
Output: [Visual Window]: Generated bivariate scatter plot highlighting positive correlation.
Real-Time Example 4: Categorical Count Analysis (sns.countplot)
Examine categorical feature frequencies and class balances using Seaborn count plots.
Python
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd

df = pd.DataFrame({
    "Department": ["IT", "HR", "IT", "Sales", "IT", "HR", "Sales", "Sales"]
})

sns.countplot(x="Department", data=df, palette="Set2")
plt.title("Categorical Department Frequency Analysis")
plt.xlabel("Department")
plt.ylabel("Employee Count")
plt.show()
Output: [Visual Window]: Generated categorical count plot showing employee distribution per department.
Real-Time Example 5: Missing Value Profiling (Missingno / Heatmap)
Identify missing data patterns visually using a missingness heatmap.
Python
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import pandas as pd

df = pd.DataFrame({
    "A": [1, 2, np.nan, 4, 5],
    "B": [np.nan, 2, 3, np.nan, 5],
    "C": [1, 2, 3, 4, 5]
})

sns.heatmap(df.isnull(), cbar=False, cmap="viridis")
plt.title("Missing Value Profiling Heatmap (Yellow = Missing)")
plt.show()
Output: [Visual Window]: Generated missing value heatmap highlighting null distribution across columns.
Real-Time Example 6: Outlier Detection via Box Plots
Detect numerical outliers and statistical quartiles across groups using box plots.
Python
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd

df = pd.DataFrame({
    "Department": ["IT", "IT", "IT", "IT", "HR", "HR", "HR", "HR"],
    "Salary": [60000, 65000, 70000, 150000, 45000, 48000, 52000, 110000]
})

sns.boxplot(x="Department", y="Salary", data=df, palette="Pastel1")
plt.title("Outlier Detection using Box Plots")
plt.show()
Output: [Visual Window]: Generated box plot highlighting salary outliers outside whiskers.
Real-Time Example 7: Correlation Matrix & Heatmap (.corr())
Compute Pearson correlation coefficients and plot feature relationships via heatmaps.
Python
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd
import numpy as np

np.random.seed(10)
df = pd.DataFrame(np.random.rand(10, 4), columns=["Feature1", "Feature2", "Feature3", "Feature4"])
corr_matrix = df.corr()

sns.heatmap(corr_matrix, annot=True, cmap="coolwarm", fmt=".2f")
plt.title("Feature Correlation Matrix Heatmap")
plt.show()
Output: [Visual Window]: Generated correlation matrix heatmap with numerical annotations.
Real-Time Example 8: Pairplot Multi-Variable Relationships (sns.pairplot)
Generate a grid of pairwise relationships across all numerical features using Seaborn pairplots.
Python
import matplotlib.pyplot as plt
import seaborn as sns
import pandas as pd

# Using sample iris-like dataset
df = pd.DataFrame({
    "SepalLength": [5.1, 4.9, 4.7, 4.6, 5.0],
    "SepalWidth": [3.5, 3.0, 3.2, 3.1, 3.6],
    "PetalLength": [1.4, 1.4, 1.3, 1.5, 1.4],
    "Species": ["Setosa", "Setosa", "Setosa", "Setosa", "Setosa"]
})

sns.pairplot(df, hue="Species", diag_kind="kde")
plt.show()
Output: [Visual Window]: Generated pairplot grid displaying pairwise bivariate scatter plots and univariate KDE diagonals.
Real-Time Example 9: Automated Summary Profiling Report Generation
Generate automated EDA summary reports in Python using libraries like ydata-profiling (formerly pandas-profiling).
Python
# Requires: pip install ydata-profiling
# from ydata_profiling import ProfileReport
# import pandas as pd
# 
# df = pd.read_csv("dataset.csv")
# profile = ProfileReport(df, title="Automated EDA Profiling Report", explorative=True)
# profile.to_file("eda_report.html")

print("Automated EDA profile report generation architecture explained successfully.")
Output: Automated EDA profile report generation architecture explained successfully.