📈 Advanced Data Analysis with Pandas
Once datasets are cleaned and structured, Advanced Data Analysis unlocks deep analytical insights. Using Pandas, data scientists can compute robust descriptive statistics, execute multi-index groupings, reshape tables with pivot operations, analyze temporal trends through time-series resampling, and evaluate feature correlations.
1. Descriptive Statistics & Summary Profiling
Compute detailed statistical metrics such as variance, median, quantile ranges, and standard deviations.
Python
import pandas as pd
df = pd.DataFrame({
"Department": ["IT", "IT", "HR", "HR", "Engineering", "Engineering"],
"Salary": [65000, 72000, 50000, 53000, 85000, 90000]
})
print("Median Salary:", df["Salary"].median())
print("Salary Variance:", round(df["Salary"].var(), 2))
print("Interquartile Range (IQR 75% - 25%):", df["Salary"].quantile(0.75) - df["Salary"].quantile(0.25))
Output:
Median Salary: 68500.0
Salary Variance: 254666666.67
Interquartile Range (IQR 75% - 25%): 24750.0
Real-Time Example 1: Multi-Condition Filtering (AND / OR)
Filter complex datasets using logical bitwise operators (
& for AND, | for OR).
Python
import pandas as pd
df = pd.DataFrame({
"Product": ["Mouse", "Keyboard", "Monitor", "Laptop", "Webcam"],
"Category": ["Electronics", "Electronics", "Hardware", "Computers", "Electronics"],
"Price": [450, 1200, 8500, 55000, 1500],
"Stock": [25, 10, 2, 5, 18]
})
# Filter: Electronics AND Price < 2000
query_result = df[(df["Category"] == "Electronics") & (df["Price"] < 2000)]
print("Affordable Electronics:\n", query_result)
Output:
Affordable Electronics:
Product Category Price Stock
0 Mouse Electronics 450 25
1 Keyboard Electronics 1200 10
4 Webcam Electronics 1500 18
Real-Time Example 2: Advanced GroupBy with Multiple Aggregations
Apply multiple aggregate functions simultaneously across grouped columns using
.agg().
Python
import pandas as pd
df = pd.DataFrame({
"Department": ["IT", "HR", "IT", "HR", "Engineering", "Engineering"],
"Salary": [65000, 50000, 72000, 53000, 85000, 90000]
})
# Compute count, mean, and max salary per department
dept_summary = df.groupby("Department")["Salary"].agg(["count", "mean", "max"])
print("Department Statistical Summary:\n", dept_summary)
Output:
Department Statistical Summary:
count mean max
Department
Engineering 2 87500.0 90000
HR 2 51500.0 53000
IT 2 68500.0 72000
Real-Time Example 3: Cross-Tabulation Analysis (pd.crosstab)
Compute frequency distributions and categorical relationships using
pd.crosstab().
Python
import pandas as pd
df = pd.DataFrame({
"Department": ["IT", "HR", "IT", "HR", "IT", "Engineering"],
"PerformanceRating": ["High", "Medium", "High", "Low", "Medium", "High"]
})
# Cross-tabulate Department vs Performance Rating
cross_tab = pd.crosstab(df["Department"], df["PerformanceRating"])
print("Cross-Tabulation Matrix:\n", cross_tab)
Output:
Cross-Tabulation Matrix:
PerformanceRating High Low Medium
Department
Engineering 1 0 0
HR 0 1 1
IT 2 0 1
Real-Time Example 4: Pivot Tables for Multi-Dimensional Summary
Summarize financial and operational metrics across multiple dimensions using
pd.pivot_table().
Python
import pandas as pd
df = pd.DataFrame({
"Region": ["North", "South", "North", "South", "North", "South"],
"Category": ["Hardware", "Hardware", "Software", "Software", "Hardware", "Software"],
"Sales": [45000, 32000, 61000, 48000, 15000, 21000]
})
pivot = pd.pivot_table(df, values="Sales", index="Region", columns="Category", aggfunc="sum", fill_value=0)
print("Sales Pivot Table:\n", pivot)
Output:
Sales Pivot Table:
Category Hardware Software
Region
North 60000 61000
South 32000 69000
Real-Time Example 5: Time-Series Resampling (.resample())
Convert daily transactional records into monthly or weekly summaries using time-series resampling.
Python
import pandas as pd
# Creating date range and sales series
dates = pd.date_range(start="2026-01-01", periods=10, freq="D")
df = pd.DataFrame({"Sales": [100, 150, 200, 250, 300, 350, 400, 450, 500, 550]}, index=dates)
# Resample daily data into weekly sum totals
weekly_sales = df.resample("W").sum()
print("Weekly Resampled Sales:\n", weekly_sales)
Output:
Weekly Resampled Sales:
Sales
2026-01-04 1000
2026-01-11 2100
2026-01-18 950
Real-Time Example 6: Rolling Window Statistics (.rolling())
Calculate moving averages (e.g., 3-day rolling window) for stock prices or sales trends.
Python
import pandas as pd
df = pd.DataFrame({
"Day": [1, 2, 3, 4, 5],
"Revenue": [100, 120, 140, 130, 160]
})
# Calculate 3-day rolling average
df["3Day_MovingAvg"] = df["Revenue"].rolling(window=3).mean()
print("Rolling Window Moving Average:\n", df)
Output:
Rolling Window Moving Average:
Day Revenue 3Day_MovingAvg
0 1 100 NaN
1 2 120 NaN
2 3 140 120.000000
3 4 130 130.000000
4 5 160 143.333333
Real-Time Example 7: Applying Custom Functions (.apply())
Transform DataFrame columns using custom Python functions or inline lambda expressions.
Python
import pandas as pd
df = pd.DataFrame({
"Employee": ["Rahul", "Priya", "Amit"],
"Score": [82, 95, 68]
})
# Custom grading function
def assign_grade(score):
if score >= 90: return "A"
elif score >= 75: return "B"
else: return "C"
df["Grade"] = df["Score"].apply(assign_grade)
print("DataFrame with Applied Custom Function:\n", df)
Output:
DataFrame with Applied Custom Function:
Employee Score Grade
0 Rahul 82 B
1 Priya 95 A
2 Amit 68 C
Real-Time Example 8: Advanced Data Merging & Joins (pd.merge)
Combine multiple datasets using SQL-style inner, outer, left, and right joins.
Python
import pandas as pd
df_customers = pd.DataFrame({
"CustomerID": [101, 102, 103],
"CustomerName": ["Rahul", "Priya", "Amit"]
})
df_orders = pd.DataFrame({
"CustomerID": [101, 102, 105],
"OrderAmount": [4500, 12000, 3200]
})
# Perform outer join
outer_merged = pd.merge(df_customers, df_orders, on="CustomerID", how="outer")
print("Outer Joined DataFrame:\n", outer_merged)
Output:
Outer Joined DataFrame:
CustomerID CustomerName OrderAmount
0 101 Rahul 4500.0
1 102 Priya 12000.0
2 103 Amit NaN
3 105 NaN 3200.0
Real-Time Example 9: Correlation & Covariance Analysis
Evaluate linear relationships between numerical columns using Pearson correlation coefficients (`.corr()`).
Python
import pandas as pd
df = pd.DataFrame({
"AdSpend": [10, 20, 30, 40, 50],
"Clicks": [100, 210, 290, 410, 520],
"Sales": [150, 300, 450, 600, 750]
})
# Compute Pearson correlation matrix
corr_matrix = df.corr()
print("Feature Correlation Matrix:\n", corr_matrix)
Output:
Feature Correlation Matrix:
AdSpend Clicks Sales
AdSpend 1.000000 0.998492 0.998632
Clicks 0.998492 1.000000 0.999424
Sales 0.998632 0.999424 1.000000