Data Analysis for Engineers/Module 5

Module 5 · Section 9 of 12

Lesson 5.8 - Model Evaluation

Target: ~10 min read - 20 min hands-on

Overview

Evaluating a model on its own training data always looks optimistic - a train/test split (and, more robustly, cross-validation) estimates real performance on new data. For classification, a confusion matrix breaks down correct/incorrect predictions by class; precision and recall matter more than raw accuracy when classes are imbalanced (rare failures); the ROC curve shows the trade-off across every decision threshold.

Why This Matters (Engineering Context)

For a predictive-maintenance model, missing an actual failure (false negative) is usually far more costly than a false alarm - precision and recall (not just accuracy) tell you whether the model is usable for that decision.

Code-Along

from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.metrics import confusion_matrix, precision_score, recall_score, roc_curve, auc, ConfusionMatrixDisplay
from sklearn.ensemble import RandomForestClassifier

# Hold out 30% for testing. stratify=y_fail keeps the class ratio the same in both splits.
X_train, X_test, y_train, y_test = train_test_split(X, y_fail, test_size=0.3, random_state=1, stratify=y_fail)
forest_eval = RandomForestClassifier(n_estimators=200, max_depth=5, random_state=1).fit(X_train, y_train)

# training accuracy flatters; test accuracy is the honest number
print(f"Training accuracy: {forest_eval.score(X_train, y_train):.3f}")
print(f"Test accuracy:     {forest_eval.score(X_test, y_test):.3f}")

# cross_val_score: refit on 5 different train/test folds -> a mean and a spread
cv_scores = cross_val_score(forest_eval, X, y_fail, cv=5)
print(f"\n5-fold CV accuracy: {cv_scores.mean():.3f} +/- {cv_scores.std():.3f}")

y_pred_test = forest_eval.predict(X_test)
cm = confusion_matrix(y_test, y_pred_test)          # [[TN, FP], [FN, TP]]
print("\nConfusion matrix:\n", cm)
# precision = of predicted-fail, how many really failed; recall = of real fails, how many we caught
print(f"\nPrecision: {precision_score(y_test, y_pred_test):.3f}")
print(f"Recall:    {recall_score(y_test, y_pred_test):.3f}")

# ROC: sweep the probability threshold; predict_proba[:, 1] = P(class 1)
y_proba = forest_eval.predict_proba(X_test)[:, 1]
fpr, tpr, thresholds = roc_curve(y_test, y_proba)
roc_auc = auc(fpr, tpr)                              # area under ROC; 0.5 = no skill, 1.0 = perfect

fig, axes = plt.subplots(1, 2, figsize=(11, 4.5))
ConfusionMatrixDisplay(cm, display_labels=["OK", "Failed"]).plot(ax=axes[0], colorbar=False)
axes[0].set_title("Confusion Matrix (test set)")
axes[1].plot(fpr, tpr, color="darkorange", label=f"ROC (AUC={roc_auc:.3f})")
axes[1].plot([0, 1], [0, 1], "k--", label="Random guess")
axes[1].set_xlabel("False Positive Rate"); axes[1].set_ylabel("True Positive Rate")
axes[1].set_title("ROC Curve"); axes[1].legend()
plt.tight_layout(); plt.show()

Run it: test accuracy is typically slightly lower than training accuracy - the honest signal of performance on unseen data. The ROC curve bows up and to the left of the diagonal, with AUC comfortably above 0.5, reflecting that the sensors carry real predictive signal.

Practice Exercises

  1. Compare precision and recall - which is higher, and what does that imply about the type of error (false positive vs false negative) the model makes more often?
  2. Change test_size to 0.5 and re-run - how much does test accuracy move vs the 0.3 split? What does that say about a single split vs cross-validation?
  3. Find a classification threshold (other than 0.5) that maximizes recall while keeping precision above 0.7, using the thresholds array from roc_curve.
# Try the practice exercises here

Knowledge Check

  1. Why does evaluating a model on its own training data overstate performance?
  2. What does the diagonal line on an ROC curve represent?
  3. In predictive maintenance, why might recall matter more than precision?
Answer key
  1. The model has already adapted to that data, including its noise
  2. A random-guess classifier with no skill - AUC 0.5
  3. Missing an actual failure (false negative, lowering recall) is usually costlier than a false alarm

52 / 63 sections · Course home · Join the coaching cohort