Lesson 2.6 - Adding & Transforming Columns

Target: ~9 min read - 20 min hands-on

Overview

.apply() runs a function across every row or column; lambda gives you a quick, throwaway function inline. This lesson computes derived metrics - like a load-band classification (Low / Normal / High / Peak) - as new columns.

Why This Matters (Engineering Context)

Encoding a classification as a reusable function turns a raw number into an interpretable category your report can act on directly - and the same function can be applied to any future dataset without retyping the thresholds.

Code-Along

# --- Custom function + .apply(): categorize each energy reading into a band ---

def load_band(kwh):
    """Qualitative daily-consumption band (simplified)."""
    if kwh < 1500:
        return "Low"
    elif kwh < 3000:
        return "Normal"
    elif kwh < 5000:
        return "High"
    else:
        return "Peak"

# .apply() runs load_band() on every value in the energy_kwh column, one at a
# time, and stores each returned label in a new "load_band" column.
energy_df["load_band"] = energy_df["energy_kwh"].apply(load_band)

# value_counts() tallies how many rows landed in each band.
print(energy_df["load_band"].value_counts())

# --- lambda version: a short inline function for a one-line derived column ---

# A lambda is an unnamed function defined inline - used here instead of a full
# def block for a one-line calculation. It converts Celsius to Fahrenheit
# (c * 9/5 + 32); the `if pd.notna(c) else np.nan` part just makes the
# "leave missing values missing" intent explicit.
energy_df["ambient_f"] = energy_df["ambient_c"].apply(lambda c: c * 9 / 5 + 32 if pd.notna(c) else np.nan)

# Preview the relevant columns together as a sanity check.
energy_df[["site", "date", "energy_kwh", "load_band", "ambient_c", "ambient_f"]].head()

Note: rows with a NaN energy_kwh pass through load_band; because NaN < 1500 is False for every comparison, they fall through to the final else and get labeled "Peak" - which is wrong. Real pipelines guard against NaN explicitly (Practice Exercise 1).

Practice Exercises

1. Fix load_band to handle NaN gracefully by returning "Unknown", then re-run and re-check .value_counts().
2. Add a new column cooling_flag that's True when ambient_c > 30, using a lambda.
3. Use .apply() (not a loop) to create a weekday_name column from the date column.

# Try the practice exercises here

Knowledge Check

1. What's the main advantage of lambda over a full def for simple transformations?
2. What happens if .apply() calls a function that errors on a NaN value?
3. Why encode a qualitative classification as a function instead of hardcoding it inline for each dataset?