Data Analysis for Engineers/Module 5

Module 5 · Section 6 of 12

Lesson 5.5 - Regression Diagnostics

Target: ~10 min read - 20 min hands-on

Overview

A high R^2 doesn't guarantee a trustworthy model. Residual plots reveal whether errors are randomly scattered (good) or follow a pattern. Heteroscedasticity is residual spread that changes across the range of predictions - it breaks a core assumption. Multicollinearity (predictors correlated with each other) inflates coefficient uncertainty; the Variance Inflation Factor (VIF) quantifies it. Overfitting is fitting training noise rather than the true pattern.

Why This Matters (Engineering Context)

A demand model that looks perfect on history but breaks on next year's forecast is a classic overfitting failure - diagnostics are how you catch it before it drives a bad capacity decision.

Code-Along

from statsmodels.stats.outliers_influence import variance_inflation_factor
from sklearn.linear_model import LinearRegression

y = demand_df["demand_mw"].values
X_multi = demand_df[["avg_temp_c", "activity_index"]].values
y_pred_multi = LinearRegression().fit(X_multi, y).predict(X_multi)
residuals = y - y_pred_multi                          # actual minus predicted

# --- Residual plots ---
fig, axes = plt.subplots(1, 2, figsize=(11, 4.5))
# residuals vs predicted: want a shapeless horizontal band. A funnel = heteroscedasticity.
axes[0].scatter(y_pred_multi, residuals, alpha=0.6, color="teal")
axes[0].axhline(0, color="red", linestyle="--")
axes[0].set_xlabel("Predicted Demand (MW)"); axes[0].set_ylabel("Residual (MW)")
axes[0].set_title("Residuals vs Predicted (checking heteroscedasticity)")
axes[1].hist(residuals, bins=15, color="slateblue", edgecolor="white")   # want roughly Normal
axes[1].set_title("Residual Distribution"); axes[1].set_xlabel("Residual (MW)")
plt.tight_layout(); plt.show()

# --- VIF: how much a predictor is explained by the OTHER predictors ---
# VIF ~ 1 = independent; > 5-10 = collinear (its coefficient becomes unreliable).
X_vif = demand_df[["avg_temp_c", "activity_index"]].copy()
X_vif.insert(0, "const", 1.0)                          # statsmodels needs an explicit intercept column
vif_data = pd.DataFrame({
    "feature": X_vif.columns,
    "VIF": [variance_inflation_factor(X_vif.values, i) for i in range(X_vif.shape[1])],
})
print("Variance Inflation Factors:")
print(vif_data)

# --- Overfitting demo: fit degree-1 and degree-15 polynomials to the same data ---
x_sorted_idx = np.argsort(demand_df["avg_temp_c"].values)   # sort so the fitted line plots cleanly
x_sorted = demand_df["avg_temp_c"].values[x_sorted_idx]
y_sorted = y[x_sorted_idx]
coeffs_low = np.polyfit(x_sorted, y_sorted, deg=1)     # a straight line
coeffs_high = np.polyfit(x_sorted, y_sorted, deg=15)   # absurdly flexible for 60 points

fig, ax = plt.subplots(figsize=(7, 5))
ax.scatter(x_sorted, y_sorted, alpha=0.4, color="gray", label="Data")
x_smooth = np.linspace(x_sorted.min(), x_sorted.max(), 200)
ax.plot(x_smooth, np.polyval(coeffs_low, x_smooth), color="steelblue", linewidth=2, label="Degree 1 (sensible)")
ax.plot(x_smooth, np.polyval(coeffs_high, x_smooth), color="red", linewidth=1.5, label="Degree 15 (overfit)")
ax.set_xlabel("Temperature (C)"); ax.set_ylabel("Demand (MW)")
ax.set_title("Overfitting: a wiggly high-degree fit vs a sensible linear fit"); ax.legend()
ax.set_ylim(y_sorted.min() - 100, y_sorted.max() + 100)
plt.tight_layout(); plt.show()

Run it: the residuals-vs-predicted plot should look like a random horizontal band with no funnel shape - the linear model's assumptions are reasonably satisfied. VIF values well below 5 indicate avg_temp_c and activity_index aren't problematically correlated (ignore the const row's own VIF). The degree-15 polynomial visibly wiggles near the data edges - overfitting, even though it passes closer to every training point.

Practice Exercises

  1. Check (visually or numerically) whether residual spread increases at higher predicted demand - is there evidence of heteroscedasticity here?
  2. Create a near-duplicate predictor (activity_index * 1.02 + noise) and recompute VIF
    • what happens?
  3. Evaluate the degree-1 and degree-15 polynomials on 10 new held-out temperature points
    • which generalizes better?
# Try the practice exercises here

Knowledge Check

  1. What residual-plot pattern suggests heteroscedasticity?
  2. What does a high VIF (> 5 or > 10) indicate?
  3. Why can a very high-complexity model do worse on new data despite fitting training data almost perfectly?
Answer key
  1. A funnel shape - spread widening or narrowing as predicted values increase
  2. That predictor is highly correlated with others, inflating its coefficient's uncertainty
  3. It has learned training noise rather than the true pattern - overfitting

49 / 63 sections · Course home · Join the coaching cohort