ADVANCED PANDAS DATA ANALYSIS WORKFLOWS

Perform advanced data transformations, multi-criteria filtering, statistical profiling, time-series resampling, and cross-tabulation using Pandas in Python.

📈 Advanced Data Analysis with Pandas

Once datasets are cleaned and structured, Advanced Data Analysis unlocks deep analytical insights. Using Pandas, data scientists can compute robust descriptive statistics, execute multi-index groupings, reshape tables with pivot operations, analyze temporal trends through time-series resampling, and evaluate feature correlations.

1. Descriptive Statistics & Summary Profiling
Compute detailed statistical metrics such as variance, median, quantile ranges, and standard deviations.
Python
import pandas as pd

df = pd.DataFrame({
    "Department": ["IT", "IT", "HR", "HR", "Engineering", "Engineering"],
    "Salary": [65000, 72000, 50000, 53000, 85000, 90000]
})

print("Median Salary:", df["Salary"].median())
print("Salary Variance:", round(df["Salary"].var(), 2))
print("Interquartile Range (IQR 75% - 25%):", df["Salary"].quantile(0.75) - df["Salary"].quantile(0.25))
Output: Median Salary: 68500.0 Salary Variance: 254666666.67 Interquartile Range (IQR 75% - 25%): 24750.0
Real-Time Example 1: Multi-Condition Filtering (AND / OR)
Filter complex datasets using logical bitwise operators (& for AND, | for OR).
Python
import pandas as pd

df = pd.DataFrame({
    "Product": ["Mouse", "Keyboard", "Monitor", "Laptop", "Webcam"],
    "Category": ["Electronics", "Electronics", "Hardware", "Computers", "Electronics"],
    "Price": [450, 1200, 8500, 55000, 1500],
    "Stock": [25, 10, 2, 5, 18]
})

# Filter: Electronics AND Price < 2000
query_result = df[(df["Category"] == "Electronics") & (df["Price"] < 2000)]
print("Affordable Electronics:\n", query_result)
Output: Affordable Electronics: Product Category Price Stock 0 Mouse Electronics 450 25 1 Keyboard Electronics 1200 10 4 Webcam Electronics 1500 18
Real-Time Example 2: Advanced GroupBy with Multiple Aggregations
Apply multiple aggregate functions simultaneously across grouped columns using .agg().
Python
import pandas as pd

df = pd.DataFrame({
    "Department": ["IT", "HR", "IT", "HR", "Engineering", "Engineering"],
    "Salary": [65000, 50000, 72000, 53000, 85000, 90000]
})

# Compute count, mean, and max salary per department
dept_summary = df.groupby("Department")["Salary"].agg(["count", "mean", "max"])

print("Department Statistical Summary:\n", dept_summary)
Output: Department Statistical Summary: count mean max Department Engineering 2 87500.0 90000 HR 2 51500.0 53000 IT 2 68500.0 72000
Real-Time Example 3: Cross-Tabulation Analysis (pd.crosstab)
Compute frequency distributions and categorical relationships using pd.crosstab().
Python
import pandas as pd

df = pd.DataFrame({
    "Department": ["IT", "HR", "IT", "HR", "IT", "Engineering"],
    "PerformanceRating": ["High", "Medium", "High", "Low", "Medium", "High"]
})

# Cross-tabulate Department vs Performance Rating
cross_tab = pd.crosstab(df["Department"], df["PerformanceRating"])

print("Cross-Tabulation Matrix:\n", cross_tab)
Output: Cross-Tabulation Matrix: PerformanceRating High Low Medium Department Engineering 1 0 0 HR 0 1 1 IT 2 0 1
Real-Time Example 4: Pivot Tables for Multi-Dimensional Summary
Summarize financial and operational metrics across multiple dimensions using pd.pivot_table().
Python
import pandas as pd

df = pd.DataFrame({
    "Region": ["North", "South", "North", "South", "North", "South"],
    "Category": ["Hardware", "Hardware", "Software", "Software", "Hardware", "Software"],
    "Sales": [45000, 32000, 61000, 48000, 15000, 21000]
})

pivot = pd.pivot_table(df, values="Sales", index="Region", columns="Category", aggfunc="sum", fill_value=0)

print("Sales Pivot Table:\n", pivot)
Output: Sales Pivot Table: Category Hardware Software Region North 60000 61000 South 32000 69000
Real-Time Example 5: Time-Series Resampling (.resample())
Convert daily transactional records into monthly or weekly summaries using time-series resampling.
Python
import pandas as pd

# Creating date range and sales series
dates = pd.date_range(start="2026-01-01", periods=10, freq="D")
df = pd.DataFrame({"Sales": [100, 150, 200, 250, 300, 350, 400, 450, 500, 550]}, index=dates)

# Resample daily data into weekly sum totals
weekly_sales = df.resample("W").sum()

print("Weekly Resampled Sales:\n", weekly_sales)
Output: Weekly Resampled Sales: Sales 2026-01-04 1000 2026-01-11 2100 2026-01-18 950
Real-Time Example 6: Rolling Window Statistics (.rolling())
Calculate moving averages (e.g., 3-day rolling window) for stock prices or sales trends.
Python
import pandas as pd

df = pd.DataFrame({
    "Day": [1, 2, 3, 4, 5],
    "Revenue": [100, 120, 140, 130, 160]
})

# Calculate 3-day rolling average
df["3Day_MovingAvg"] = df["Revenue"].rolling(window=3).mean()

print("Rolling Window Moving Average:\n", df)
Output: Rolling Window Moving Average: Day Revenue 3Day_MovingAvg 0 1 100 NaN 1 2 120 NaN 2 3 140 120.000000 3 4 130 130.000000 4 5 160 143.333333
Real-Time Example 7: Applying Custom Functions (.apply())
Transform DataFrame columns using custom Python functions or inline lambda expressions.
Python
import pandas as pd

df = pd.DataFrame({
    "Employee": ["Rahul", "Priya", "Amit"],
    "Score": [82, 95, 68]
})

# Custom grading function
def assign_grade(score):
    if score >= 90: return "A"
    elif score >= 75: return "B"
    else: return "C"

df["Grade"] = df["Score"].apply(assign_grade)

print("DataFrame with Applied Custom Function:\n", df)
Output: DataFrame with Applied Custom Function: Employee Score Grade 0 Rahul 82 B 1 Priya 95 A 2 Amit 68 C
Real-Time Example 8: Advanced Data Merging & Joins (pd.merge)
Combine multiple datasets using SQL-style inner, outer, left, and right joins.
Python
import pandas as pd

df_customers = pd.DataFrame({
    "CustomerID": [101, 102, 103],
    "CustomerName": ["Rahul", "Priya", "Amit"]
})

df_orders = pd.DataFrame({
    "CustomerID": [101, 102, 105],
    "OrderAmount": [4500, 12000, 3200]
})

# Perform outer join
outer_merged = pd.merge(df_customers, df_orders, on="CustomerID", how="outer")

print("Outer Joined DataFrame:\n", outer_merged)
Output: Outer Joined DataFrame: CustomerID CustomerName OrderAmount 0 101 Rahul 4500.0 1 102 Priya 12000.0 2 103 Amit NaN 3 105 NaN 3200.0
Real-Time Example 9: Correlation & Covariance Analysis
Evaluate linear relationships between numerical columns using Pearson correlation coefficients (`.corr()`).
Python
import pandas as pd

df = pd.DataFrame({
    "AdSpend": [10, 20, 30, 40, 50],
    "Clicks": [100, 210, 290, 410, 520],
    "Sales": [150, 300, 450, 600, 750]
})

# Compute Pearson correlation matrix
corr_matrix = df.corr()

print("Feature Correlation Matrix:\n", corr_matrix)
Output: Feature Correlation Matrix: AdSpend Clicks Sales AdSpend 1.000000 0.998492 0.998632 Clicks 0.998492 1.000000 0.999424 Sales 0.998632 0.999424 1.000000