Lesson 5.2 - Hypothesis Testing

Target: ~10 min read - 20 min hands-on

Overview

Hypothesis testing gives you a disciplined way to answer "is this difference real, or just random noise?" A t-test compares two group means; ANOVA compares three or more; a chi-square test checks whether categorical proportions differ from expected. The p-value is the probability of seeing a difference at least this large if there were truly no difference - a small p-value (< 0.05 conventionally) suggests the observed difference is unlikely to be chance alone.

Why This Matters (Engineering Context)

"Is Supplier A's part really closer to nominal than Supplier B's, or did we get lucky with this lot?" is exactly what a t-test answers rigorously, instead of a gut feeling from comparing two averages.

Code-Along

np.random.seed(50)

# --- Two-sample t-test: do two groups have different MEANS? ---
supplier_a = np.random.normal(40.00, 0.040, 25)
supplier_b = np.random.normal(40.03, 0.050, 25)     # generated with a deliberate +0.03 offset

# ttest_ind returns (t statistic, p-value). p < alpha -> difference unlikely to be chance.
t_stat, p_value = stats.ttest_ind(supplier_a, supplier_b)
print(f"t-statistic: {t_stat:.3f}, p-value: {p_value:.4f}")
alpha = 0.05
print("significant difference" if p_value < alpha else "no significant difference detected")

# --- ANOVA: compare THREE OR MORE group means at once ---
supplier_c = np.random.normal(39.99, 0.035, 25)
f_stat, p_anova = stats.f_oneway(supplier_a, supplier_b, supplier_c)
print(f"\nANOVA F-statistic: {f_stat:.3f}, p-value: {p_anova:.4f}")

# --- Chi-square test of independence: does pass/fail depend on supplier? ---
observed = np.array([          # rows = suppliers, columns = [pass, fail]
    [23, 2],   # Supplier A
    [18, 7],   # Supplier B
    [24, 1],   # Supplier C
])
# returns (chi2, p-value, degrees of freedom, table expected if the two were independent)
chi2_stat, p_chi2, dof, expected = stats.chi2_contingency(observed)
print(f"\nChi-square: {chi2_stat:.3f}, p-value: {p_chi2:.4f}, dof: {dof}")
print("Expected counts under independence:\n", np.round(expected, 1))

Run it: with these generated means (40.00 vs 40.03 mm) the t-test returns a small p-value, correctly detecting the deliberate offset. The chi-square test checks whether pass/fail rate depends on supplier - with Supplier B's higher fail count, expect a small p-value there too.

Practice Exercises

1. Re-run the t-test with much smaller samples (5 per supplier instead of 25) - does the p-value still detect the difference? What does that say about sample size and power?
2. Run a t-test on two samples with identical generating means (both np.random.normal(40, 0.04, 25)) - confirm the p-value is usually well above 0.05.
3. Interpret the ANOVA result in one sentence: what does it tell you, and not tell you, about which specific pair of suppliers differs?

# Try the practice exercises here

Knowledge Check

1. What does a p-value represent?
2. When would you use ANOVA instead of a t-test?
3. What does the chi-square test of independence check for?