Data Analysis for Engineers/Module 2

Module 2 · Section 4 of 12

Lesson 2.3 - Selecting & Filtering

Target: ~9 min read - 20 min hands-on

Overview

.loc[] selects by label (column/index names), .iloc[] selects by integer position. Boolean masks - df[df["column"] > value] - filter rows by a condition, and they're the single most-used pandas pattern in this course.

We'll filter the energy dataset down to one sector, then to a date range, then combine both conditions.

Why This Matters (Engineering Context)

Almost every engineering dataset is filtered by site, asset, sector, or date range before analysis - you rarely want the entire estate at once when you're scoping a study to one building, one production line, or one billing period.

Code-Along

# --- .loc and .iloc: label-based vs position-based indexing ---

# .loc[row_label, column_label] — access by LABEL
# Gets the value in row labeled 0, column named "site"
print(energy_df.loc[0, "site"])

# .iloc[row_position, column_position] — access by INTEGER POSITION
# Gets the value in the 1st row (position 0), 1st column (position 0)
print(energy_df.iloc[0, 0])

# .loc supports slicing by label and selecting multiple columns at once
# Note: with .loc, slice 0:2 is INCLUSIVE of both endpoints (rows 0, 1, and 2)
print(energy_df.loc[0:2, ["site", "energy_kwh"]])

# --- Boolean mask: filter rows using a True/False condition ---

# Creates a Series of True/False values — True where sector is "Data Center"
dc_mask = energy_df["sector"] == "Data Center"

# Using the mask to index the DataFrame keeps only the True rows
dc_df = energy_df[dc_mask]
print("\nData Center rows:", len(dc_df), "out of", len(energy_df))

# --- Combine multiple conditions with & (and) ---
# Each condition must be wrapped in parentheses when combined this way.
# Filters for rows that are BOTH Data Center sector AND energy_kwh >= 6000
high_dc = energy_df[(energy_df["sector"] == "Data Center") & (energy_df["energy_kwh"] >= 6000)]
print("High-consumption Data Center days (>= 6000 kWh):", len(high_dc))

# --- Date-range filter ---
# Compares the "date" column against string dates (pandas parses these
# automatically since the column is a datetime dtype) to select rows
# falling within August 2023, across all sites
aug_2023 = energy_df[(energy_df["date"] >= "2023-08-01") & (energy_df["date"] <= "2023-08-31")]
print("Rows in August 2023 (all sites):", len(aug_2023))

Run it: exact counts depend on the random seed set in Lesson 2.2 - as long as you ran that cell first, your numbers will match a classmate's exactly, since np.random.seed(42) makes the synthetic data fully reproducible.

Practice Exercises

  1. Filter energy_df to just the "Delta Mill" site, and print how many of its rows have energy_kwh above that site's own median.
  2. Use .loc[] to select only the date and ambient_c columns for the first 5 rows.
  3. Write a boolean mask for rows where ambient_c is above 30 C and energy_kwh is missing (.isna()) - how many are there?
# Try the practice exercises here

Knowledge Check

  1. What is the key difference between .loc[] and .iloc[]?
  2. What operator combines two boolean conditions in a pandas mask (not Python's and)?
  3. True or False: df[df["col"] > 5] returns a new DataFrame, not a modification of the original.
Answer key
  1. .loc[] selects by label, .iloc[] selects by integer position
  2. & (with each condition wrapped in parentheses)
  3. True

14 / 63 sections · Course home · Join the coaching cohort