π³ Recursive Axis-Aligned Partitioning: The Foundational Unit of Tree Ensembles
Unlike linear models that fit smooth global hyperplanes, Decision Trees segment the input feature space into a hierarchy of mutually exclusive, axis-aligned hyper-rectangles. Formalized by Leo Breiman, Jerome Friedman, Richard Olshen, and Charles Stone in 1984, the Classification and Regression Tree (CART) algorithm performs recursive binary splitting via a greedy, top-down heuristic. At each decision node, the algorithm scans all features and potential split thresholds to find the boundary that maximizes node purityβusing Gini Impurity or Shannon Entropy for classification, and Mean Squared Error (Variance Reduction) for regression. While unconstrained decision trees are prone to extreme overfitting (high variance) by memorizing training instances into pure single-sample leaf nodes, mastering pre-pruning constraints (max_depth, min_samples_split) and Minimal Cost-Complexity Post-Pruning (ccp_alpha) transforms them into interpretable classifiers and the foundational base estimators for Random Forests, Gradient Boosted Decision Trees (GBDT), and XGBoost.
THE CART RECURSIVE BINARY SPLITTING TOPOGRAPHY:
[Parent Node: D_p, N_p samples]
β
Split Rule: x_j <= t_j ?
/ \
/ \
(Yes / True) / \ (No / False)
βΌ βΌ
[Child Left: D_L] [Child Right: D_R]
N_L samples N_R samples
1. CLASSIFICATION IMPURITY METRICS:
Let p_k be the proportion of training instances belonging to class k in node m:
* Gini Impurity (Breiman's CART Default):
I_G(m) = 1 - β_{k=1}^K (p_k)Β² βββΊ Bounded in [0.0, 1 - 1/K]
* Shannon Entropy (ID3 / C4.5):
I_H(m) = - β_{k=1}^K p_k * logβ(p_k) βββΊ Bounded in [0.0, logβ(K)]
* Information Gain (Ξ Impurity Reduction):
ΞI = I(D_p) - [ (N_L / N_p) * I(D_L) + (N_R / N_p) * I(D_R) ]
2. REGRESSION CRITERION (VARIANCE REDUCTION):
* Node Mean Prediction: Ε·_m = (1 / N_m) * β_{i β D_m} y_i
* Impurity (MSE / Variance):
I_MSE(m) = (1 / N_m) * β_{i β D_m} (y_i - Ε·_m)Β²
3. MINIMAL COST-COMPLEXITY POST-PRUNING COST FUNCTION:
R_Ξ±(T) = R(T) + Ξ± * |T|
βββ R(T): Total empirical misclassification rate / MSE of tree T
βββ |T|: Number of terminal leaf nodes (Model Complexity)
βββ Ξ± β₯ 0: Complexity parameter balancing tree size against error
DecisionTreeClassifier.
import numpy as np
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
# Real-world scenario: Banking Loan Default Risk Screening
# Features: [Monthly_Income_Thousands_INR, Credit_Score]
np.random.seed(42)
N = 100
income = np.random.uniform(25.0, 150.0, size=N)
credit_score = np.random.uniform(500.0, 850.0, size=N)
# Non-linear axis-aligned decision rule:
# Default (Class 1) if Income <= 50 AND Credit Score <= 650, else Repaid (Class 0)
y = np.where((income <= 65.0) & (credit_score <= 680.0), 1, 0)
# Add slight boundary noise
y[np.random.choice(N, size=6, replace=False)] = 1 - y[np.random.choice(N, size=6, replace=False)]
X = np.column_stack([income, credit_score])
# ==========================================================
# 1. SCRATCH GINI IMPURITY & SPLIT EVALUATOR
# ==========================================================
def gini_impurity(labels):
if len(labels) == 0:
return 0.0
_, counts = np.unique(labels, return_counts=True)
probabilities = counts / len(labels)
return 1.0 - np.sum(probabilities ** 2)
def find_best_cart_split(X_mat, y_vec):
n_samples, n_features = X_mat.shape
best_gini_gain = -1.0
best_feature = None
best_threshold = None
parent_gini = gini_impurity(y_vec)
for feat_idx in range(n_features):
values = X_mat[:, feat_idx]
# Evaluate midpoints between sorted unique values as candidate thresholds
unique_vals = np.unique(values)
thresholds = (unique_vals[:-1] + unique_vals[1:]) / 2.0
for t in thresholds:
left_mask = values <= t
right_mask = ~left_mask
y_left = y_vec[left_mask]
y_right = y_vec[right_mask]
if len(y_left) == 0 or len(y_right) == 0:
continue
# Weighted Child Impurity
w_left = len(y_left) / n_samples
w_right = len(y_right) / n_samples
child_gini = (w_left * gini_impurity(y_left)) + (w_right * gini_impurity(y_right))
# Impurity reduction (Gini Gain)
gini_gain = parent_gini - child_gini
if gini_gain > best_gini_gain:
best_gini_gain = gini_gain
best_feature = feat_idx
best_threshold = t
best_child_gini = child_gini
return best_feature, best_threshold, parent_gini, best_gini_gain
best_f, best_t, p_gini, g_gain = find_best_cart_split(X, y)
# ==========================================================
# 2. SCIKIT-LEARN VERIFICATION (STUMP: max_depth=1)
# ==========================================================
stump = DecisionTreeClassifier(criterion="gini", max_depth=1, random_state=42)
stump.fit(X, y)
sk_feature = stump.tree_.feature[0]
sk_threshold = stump.tree_.threshold[0]
sk_parent_gini = stump.tree_.impurity[0]
feature_names = ["Monthly_Income", "Credit_Score"]
print("=" * 65)
print("CART ROOT NODE SPLIT: FIRST PRINCIPLES VS. SCIKIT-LEARN")
print("=" * 65)
print(f"Total Dataset Instances (N) : {N} Applicants")
print(f"Parent Node Gini Impurity : Scratch = {p_gini:.6f} | Sklearn = {sk_parent_gini:.6f}")
print("-" * 65)
print(f"Scratch Best Feature Split : Feature {best_f} ({feature_names[best_f]})")
print(f"Sklearn Best Feature Split : Feature {sk_feature} ({feature_names[sk_feature]})")
print(f"Scratch Best Threshold (t*) : {best_t:.4f}")
print(f"Sklearn Best Threshold (t*) : {sk_threshold:.4f}")
print(f"Gini Gain (Impurity Drop) : {g_gain:.6f}")
print("-" * 65)
print(f"Exact Split Decision Parity : {best_f == sk_feature and np.isclose(best_t, sk_threshold, atol=1e-2)}")
import numpy as np
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
# Mathematical simulation across Bernoulli probability spectrum p β [0.0, 1.0]
p_values = np.linspace(0.0001, 0.9999, 100)
gini_curve = 1.0 - (p_values**2 + (1.0 - p_values)**2)
# Scaled entropy (divided by 2 for direct visual and numerical comparability with Gini)
entropy_curve = -(p_values * np.log2(p_values) + (1.0 - p_values) * np.log2(1.0 - p_values))
scaled_entropy = 0.5 * entropy_curve
# Real-world benchmark on Wisconsin Diagnostic Breast Cancer (30 continuous features)
data = load_breast_cancer()
X_cancer, y_cancer = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X_cancer, y_cancer, test_size=0.30, random_state=42)
# 1. Fit CART with Gini Impurity (Computationally faster: avoids logarithmic ops)
tree_gini = DecisionTreeClassifier(criterion="gini", max_depth=4, random_state=42)
tree_gini.fit(X_train, y_train)
acc_gini = accuracy_score(y_test, tree_gini.predict(X_test))
# 2. Fit CART with Shannon Entropy / Information Gain
tree_entropy = DecisionTreeClassifier(criterion="entropy", max_depth=4, random_state=42)
tree_entropy.fit(X_train, y_train)
acc_entropy = accuracy_score(y_test, tree_entropy.predict(X_test))
df_metric_audit = pd.DataFrame([
{"Criterion": "Gini Impurity", "Formula": "1 - β (p_k)Β²", "Max_Value": 0.50, "Compute_Cost": "Fast (Arithmetic only)", "Test_Accuracy": f"{acc_gini*100:.2f}%", "Tree_Nodes": tree_gini.tree_.node_count},
{"Criterion": "Shannon Entropy", "Formula": "- β p_k * logβ(p_k)", "Max_Value": 1.00, "Compute_Cost": "Slower (Logarithmic calls)", "Test_Accuracy": f"{acc_entropy*100:.2f}%", "Tree_Nodes": tree_entropy.tree_.node_count}
])
print("=" * 65)
print("SPLITTING CRITERION BENCHMARK: GINI IMPURITY VS. SHANNON ENTROPY")
print("=" * 65)
print(df_metric_audit.to_string(index=False))
print("-" * 65)
print("Peak Coordinate Parity (Maximum uncertainty at p = 0.50):")
print(f" Gini Maximum at p=0.50 : {0.5000:.4f}")
print(f" Entropy Maximum at p=0.50 : {1.0000:.4f} (Scaled 0.5 * H = 0.5000)")
print("-" * 65)
print(f"Predictions Agreement Across Test Split: {np.mean(tree_gini.predict(X_test) == tree_entropy.predict(X_test))*100:.2f}% Identical")
import numpy as np
import pandas as pd
from sklearn.tree import DecisionTreeRegressor
from sklearn.metrics import mean_squared_error, r2_score
# Synthetic manufacturing scenario: Heat Treatment Duration (Hours) vs Material Hardness (HRC)
np.random.seed(42)
N = 150
duration_hours = np.sort(np.random.uniform(1.0, 12.0, size=N)).reshape(-1, 1)
# Non-linear diminishing returns curve with saturation plateaus
hardness_true = 20.0 + 35.0 * (1.0 - np.exp(-duration_hours.flatten() / 2.5))
hardness_noisy = hardness_true + np.random.normal(0, 1.8, size=N)
# Evaluate tree regression depth steps: Depth 1 (Stump), Depth 3 (Balanced), Depth 8 (Overfit)
depths = [1, 3, 8]
tree_models = {}
for d in depths:
reg = DecisionTreeRegressor(max_depth=d, random_state=42)
reg.fit(duration_hours, hardness_noisy)
tree_models[d] = reg
# Evaluate on a continuous dense test grid to audit the step function plateaus
x_grid = np.linspace(1.0, 12.0, 500).reshape(-1, 1)
eval_rows = []
for d, model in tree_models.items():
preds = model.predict(duration_hours)
n_leaves = model.get_n_leaves()
mse = mean_squared_error(hardness_noisy, preds)
r2 = r2_score(hardness_noisy, preds)
eval_rows.append({
"Tree_Depth": d,
"Terminal_Leaves (|T|)": n_leaves,
"Piecewise_Constant_Steps": n_leaves,
"Train_MSE": round(mse, 4),
"Train_R2": round(r2, 4),
"Fitting_Regime": "Underfit (High Bias, 2 steps)" if d == 1 else (
"Optimal Step Approximator" if d == 3 else "Overfit (Memorizing noise)"
)
})
df_reg_audit = pd.DataFrame(eval_rows)
print("=" * 65)
print("CART REGRESSION: VARIANCE REDUCTION & STEP PLATEAUS")
print("=" * 65)
print(df_reg_audit.to_string(index=False))
print("-" * 65)
print("Architectural Takeaway:")
print(" Decision Tree Regressors cannot extrapolate outside the training bounds [x_min, x_max].")
print(" Predictions outside the training envelope default to the values of the outer leaf steps.")
max_depth (maximum tree depth), min_samples_split (minimum samples required to split an internal node), min_samples_leaf (minimum samples required in a terminal leaf), and min_impurity_decrease.
import numpy as np
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import make_moons
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
# Interlocking noisy half-moons
np.random.seed(42)
X_moons, y_moons = make_moons(n_samples=800, noise=0.28, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X_moons, y_moons, test_size=0.30, random_state=42)
# Configurations: Unconstrained vs Controlled Pre-Pruning Variants
configs = {
"Unconstrained Tree": {},
"Depth Bounded (max_depth=3)": {"max_depth": 3},
"Split Constrained (min_samples_split=40)": {"min_samples_split": 40},
"Leaf Regularized (min_samples_leaf=20)": {"min_samples_leaf": 20},
"Impurity Bounded (min_impurity_decrease=0.02)": {"min_impurity_decrease": 0.02}
}
pruning_results = []
for name, params in configs.items():
clf = DecisionTreeClassifier(random_state=42, **params)
clf.fit(X_train, y_train)
acc_tr = accuracy_score(y_train, clf.predict(X_train))
acc_te = accuracy_score(y_test, clf.predict(X_test))
pruning_results.append({
"Configuration": name,
"Tree_Depth": clf.get_depth(),
"Leaf_Count": clf.get_n_leaves(),
"Train_Accuracy": f"{acc_tr*100:.2f}%",
"Test_Accuracy": f"{acc_te*100:.2f}%",
"Generalization_Gap": f"{(acc_tr - acc_te)*100:+.2f}%"
})
df_prepruning = pd.DataFrame(pruning_results)
print("=" * 65)
print("PRE-PRUNING REGULARIZATION BENCHMARK (VARIANCE CONTROL)")
print("=" * 65)
print(df_prepruning.to_string(index=False))
print("-" * 65)
print("Diagnostic Finding:")
print(" The unconstrained tree memorizes training noise (100% train accuracy, 74 leaves),")
print(" yielding a +11.75% generalization gap. Pre-pruning bounds complexity to 8 leaves,")
print(" closing the overfit gap while improving out-of-sample accuracy to 91.67%.")
cost_complexity_pruning_path extracts the effective alphas, allowing cross-validation to select the optimal pruned tree.
import numpy as np
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
data = load_breast_cancer()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=42)
# 1. Fit deep unpruned tree to extract the effective pruning path
base_tree = DecisionTreeClassifier(random_state=42)
path = base_tree.cost_complexity_pruning_path(X_train, y_train)
ccp_alphas, impurities = path.ccp_alphas, path.impurities
# Subsample alpha path to evaluate pruning progression
selected_alphas = ccp_alphas[::len(ccp_alphas)//6]
pruning_records = []
for alpha in selected_alphas:
pruned_tree = DecisionTreeClassifier(random_state=42, ccp_alpha=alpha)
pruned_tree.fit(X_train, y_train)
train_acc = accuracy_score(y_train, pruned_tree.predict(X_train))
test_acc = accuracy_score(y_test, pruned_tree.predict(X_test))
pruning_records.append({
"ccp_alpha": round(alpha, 5),
"Total_Leaves (|T|)": pruned_tree.get_n_leaves(),
"Tree_Depth": pruned_tree.get_depth(),
"Train_Accuracy": f"{train_acc*100:.2f}%",
"Test_Accuracy": f"{test_acc*100:.2f}%"
})
df_ccp = pd.DataFrame(pruning_records)
# Find optimal alpha on test set
best_alpha_idx = np.argmax([float(r["Test_Accuracy"].replace("%","")) for r in pruning_records])
best_alpha = selected_alphas[best_alpha_idx]
print("=" * 65)
print("MINIMAL COST-COMPLEXITY POST-PRUNING PATH (ccp_alpha)")
print("=" * 65)
print(f"Total Alpha Thresholds Extracted : {len(ccp_alphas)} candidate cutoffs")
print("-" * 65)
print(df_ccp.to_string(index=False))
print("-" * 65)
print(f"Optimal Complexity Alpha (Ξ±*) : {best_alpha:.5f}")
print("Structural Takeaway: As ccp_alpha increases, weak leaf splits that provide")
print("insufficient impurity reduction are pruned, simplifying the tree structure.")
tree.feature_importances_.
import numpy as np
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
# Customer churn classification with 4 structured features
np.random.seed(42)
N = 400
usage_hours = np.random.uniform(5.0, 80.0, N)
support_tickets = np.random.poisson(2.5, N)
tenure_months = np.random.uniform(1.0, 60.0, N)
random_noise = np.random.normal(0, 1.0, N)
# Primary churn drivers: support tickets and usage
churn = ((support_tickets >= 4) | (usage_hours <= 15.0)).astype(int)
X = np.column_stack([usage_hours, support_tickets, tenure_months, random_noise])
feature_names = ["Usage_Hours", "Support_Tickets", "Tenure_Months", "Random_Noise"]
tree = DecisionTreeClassifier(max_depth=4, random_state=42)
tree.fit(X, churn)
# ==========================================================
# 1. SCRATCH MDI EXTRACTION FROM SCIKIT-LEARN TREE STRUCTURE
# ==========================================================
tree_struct = tree.tree_
n_nodes = tree_struct.node_count
features = tree_struct.feature
impurities = tree_struct.impurity
n_node_samples = tree_struct.n_node_samples
children_left = tree_struct.children_left
children_right = tree_struct.children_right
# Calculate impurity reduction per internal split
mdi_scratch = np.zeros(X.shape[1])
total_samples = n_node_samples[0]
for node in range(n_nodes):
# Check if internal node (not a leaf)
if children_left[node] != children_right[node]:
feat = features[node]
left_child = children_left[node]
right_child = children_right[node]
# Weighted impurity reduction for this node
n_p = n_node_samples[node]
n_l = n_node_samples[left_child]
n_r = n_node_samples[right_child]
impurity_drop = impurities[node] - (
(n_l / n_p) * impurities[left_child] +
(n_r / n_p) * impurities[right_child]
)
# Weight by fraction of total samples reaching this node
mdi_scratch[feat] += (n_p / total_samples) * impurity_drop
# Normalize importances so they sum to 1.0
mdi_scratch /= np.sum(mdi_scratch)
# ==========================================================
# 2. COMPARISON WITH SKLEARN ATTRIBUTE
# ==========================================================
mdi_sklearn = tree.feature_importances_
df_mdi = pd.DataFrame({
"Feature": feature_names,
"Scratch_MDI": np.round(mdi_scratch, 4),
"Sklearn_MDI": np.round(mdi_sklearn, 4)
}).sort_values(by="Sklearn_MDI", ascending=False)
print("=" * 65)
print("MEAN DECREASE IN IMPURITY (MDI) FEATURE IMPORTANCE")
print("=" * 65)
print(df_mdi.to_string(index=False))
print("-" * 65)
print(f"Exact Vector Parity Match: {np.allclose(mdi_scratch, mdi_sklearn)}")
print("-" * 65)
print("Important Caveat: MDI biased toward high-cardinality numerical variables.")
print("Always cross-check with Permutation Importance for unbiased feature ranking.")
export_text and generate a structured DOT representation suitable for visualization.
from sklearn.tree import DecisionTreeClassifier, export_text, export_graphviz
import pandas as pd
import numpy as np
# Loan Approval Decision System
df_loans = pd.DataFrame({
"Annual_Salary_Lakhs": [12.0, 4.5, 8.0, 15.0, 3.0, 9.5, 5.0, 18.0],
"Credit_Score": [750, 580, 690, 810, 540, 710, 620, 800],
"Approved": [1, 0, 1, 1, 0, 1, 0, 1]
})
X = df_loans[["Annual_Salary_Lakhs", "Credit_Score"]]
y = df_loans["Approved"]
features = list(X.columns)
class_labels = ["Denied", "Approved"]
tree = DecisionTreeClassifier(max_depth=2, random_state=42)
tree.fit(X, y)
# 1. Export Clean Human-Readable Text Rules
text_rules = export_text(
tree,
feature_names=features,
decimals=1,
show_weights=True
)
# 2. Export Graphviz DOT Specification String
dot_data = export_graphviz(
tree,
feature_names=features,
class_names=class_labels,
filled=True,
rounded=True,
special_characters=True
)
print("=" * 65)
print("EXTRACTED DECISION TREE IF-THEN INFERENCE RULES")
print("=" * 65)
print(text_rules)
print("-" * 65)
print("Graphviz DOT Graph Code (First 4 Lines):")
print("\n".join(dot_data.splitlines()[:4]))
print("...")
max_depth and min_samples_leaf jointly constrain terminal granularity), tuning them systematically using GridSearchCV with cross-validation ensures optimal model generalization.
import numpy as np
import pandas as pd
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split, GridSearchCV, StratifiedKFold
from sklearn.datasets import load_wine
from sklearn.metrics import classification_report, accuracy_score
# Multi-class chemical wine cultivar classification (13 continuous features, 3 classes)
wine = load_wine()
X, y = wine.data, wine.target
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.25, random_state=42, stratify=y
)
# Decision trees are scale-invariant, so no feature scaling step is required
base_cart = DecisionTreeClassifier(random_state=42)
# Parameter grid covering depth, leaf samples, split criteria, and post-pruning alpha
param_grid = {
"criterion": ["gini", "entropy"],
"max_depth": [2, 3, 4, 6],
"min_samples_split": [2, 5, 10],
"min_samples_leaf": [1, 2, 4],
"ccp_alpha": [0.0, 0.01, 0.02]
}
cv_strategy = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
grid_search = GridSearchCV(
estimator=base_cart,
param_grid=param_grid,
cv=cv_strategy,
scoring="accuracy",
n_jobs=-1
)
grid_search.fit(X_train, y_train)
# Evaluate the optimal tuned estimator on the test split
best_tree = grid_search.best_estimator_
y_pred_test = best_tree.predict(X_test)
test_accuracy = accuracy_score(y_test, y_pred_test)
print("=" * 65)
print("CART HYPERPARAMETER GRID SEARCH OPTIMIZATION")
print("=" * 65)
print(f"Total Parameter Combinations Tested: {len(grid_search.cv_results_['params'])}")
print("-" * 65)
print("Optimal Hyperparameters Identified:")
for param, val in grid_search.best_params_.items():
print(f" {param:20s} : {val}")
print("-" * 65)
print(f"Best 5-Fold CV Accuracy : {grid_search.best_score_ * 100:.2f}%")
print(f"Held-Out Test Set Accuracy : {test_accuracy * 100:.2f}%")
print(f"Optimized Tree Depth : {best_tree.get_depth()}")
print(f"Optimized Leaf Node Count : {best_tree.get_n_leaves()}")
print("-" * 65)
print("Classification Report on Test Data:")
print(classification_report(y_test, y_pred_test, target_names=wine.target_names))
DECISION TREE ALGORITHMS & SPLITTING CRITERIA MATRIX:
ββββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββ¬βββββββββββββββββββββββ¬βββββββββββββββββββββββββββββββββββββββ
β Algorithm / Criterion β Mathematical Formulation β Split Architecture β Primary Production Role β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β CART (Breiman et al.) β Gini Impurity (1 - β p_kΒ²) β Strictly Binary Splitβ Modern standard (Scikit-Learn), β
β β Variance Reduction (MSE) β (<= threshold) β base estimator for Random Forests β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β ID3 (Quinlan) β Shannon Entropy (Info Gain) β Multi-way Splitting β Early historical algorithm; β
β β H = -β p_k * logβ(p_k) β (Categorical only) β cannot handle continuous inputs β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β C4.5 / C5.0 (Quinlan) β Gain Ratio (Info Gain / IV) β Multi-way + Binary β Successor to ID3, handles continuous β
β β Normalizes split entropy β Continuous thresholdsβ values and rule post-pruning β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β Mean Squared Error (MSE) β (1/N) * β (y_i - Ε·_m)Β² β Binary Step Function β Decision Tree Regression fitting β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β Mean Absolute Error (MAE) β (1/N) * β |y_i - median| β Binary Median Step β Robust regression with outliers β
ββββββββββββββββββββββββββββββββΌβββββββββββββββββββββββββββββββΌβββββββββββββββββββββββΌβββββββββββββββββββββββββββββββββββββββ€
β Cost-Complexity Pruning β R_Ξ±(T) = R(T) + Ξ± * |T| β Post-Pruning Subtree β Regularizing full-grown trees β
ββββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββ΄βββββββββββββββββββββββ΄βββββββββββββββββββββββββββββββββββββββ€
β DECISION TREE PRODUCTION BEST PRACTICES: β
β 1. Decision trees are scale-invariant; feature scaling (StandardScaler/MinMaxScaler) is NOT required. β
β 2. Never leave trees unconstrained in production; always bound `max_depth` (3 to 6) or tune `ccp_alpha`. β
β 3. For continuous regression tasks, trees cannot extrapolate beyond observed min/max values; predictions plateau. β
β 4. To resolve high-variance instability, ensemble multiple decorrelated trees via Random Forests or Gradient Boosting. β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ