Lesson 2.10 - Exploratory Data Analysis

Target: ~10 min read - 25 min hands-on

Overview

This capstone lesson runs a full EDA workflow end-to-end on the site-energy dataset: load -> inspect -> clean -> summarize -> spot patterns. This is the sequence you'll repeat on nearly every dataset in this course and in professional work - and it's the direct rehearsal for Mini-Project 2.

Why This Matters (Engineering Context)

A disciplined EDA habit - always check shape, dtypes, and missingness before drawing conclusions - is what separates a defensible analysis from one that gets challenged in review.

Code-Along

# A compact end-to-end EDA pass: inspect -> clean -> summarize -> spot patterns.

# --- Step 1: Inspect --- shape, column types, and where the gaps are
print("Shape:", energy_df.shape)
print("\nDtypes:\n", energy_df.dtypes)
print("\nMissing values:\n", energy_df.isna().sum())

# --- Step 2: Clean --- drop the injected duplicate rows, then fill each
# missing energy_kwh with that SITE's own mean. groupby + transform returns a
# result aligned to the original rows, so it assigns straight back in.
eda_df = energy_df.drop_duplicates().copy()
eda_df["energy_kwh"] = eda_df.groupby("site")["energy_kwh"] \
    .transform(lambda s: s.fillna(s.mean()))

# --- Step 3: Summarize ---
print("\nOverall energy stats:")
print(eda_df["energy_kwh"].describe())

# group by (year, site) and sum; unstack() then pivots the "site" level of the
# index out into columns, giving a year x site grid.
annual_by_site = eda_df.groupby([eda_df["date"].dt.year, "site"])["energy_kwh"] \
    .sum().unstack().round(0)
print("\nAnnual energy total by site:")
print(annual_by_site)

# --- Step 4: Spot patterns --- average daily energy per calendar month;
# idxmax() returns the month number with the highest average.
eda_df["month"] = eda_df["date"].dt.month
peak_month = eda_df.groupby("month")["energy_kwh"].mean().idxmax()
print(f"\nHighest-consumption month on average (by day): month #{peak_month}")

Note: with the cooling-load logic built into Lesson 2.2's generator (elevated demand for day-of-year 60-305, roughly March-October), the peak month should land in that window - a good sanity check that the dataset behaves the way domain knowledge predicts.

Practice Exercises

1. Extend the EDA to compute the lowest-consumption month the same way (idxmin()).
2. Identify which site has the highest variance in daily energy (.groupby("site")["energy_kwh"].var()) - what might that suggest operationally?
3. Write 2-3 sentences summarizing what this EDA found, as if it were the opening paragraph of an energy-audit report.

# Try the practice exercises here

Knowledge Check

1. What are the four steps of the EDA workflow demonstrated in this lesson?
2. Why fill missing energy with each site's own mean rather than the dataset-wide mean?
3. What sanity check confirms the synthetic dataset's cooling-load logic is working?