Chapter 07 · Inference
The Hypothesis-Testing Engine
Every hypothesis test, however exotic its name, runs the same engine: assume nothing is going on, measure how surprising the data would be under that assumption, and decide. Learn the engine once and every test in the next chapters is just a different fuel.
In 60 seconds
- H₀ (null) = "no effect / no difference"; Hₐ (alternative) = what you want evidence for.
- The p-value is the probability, if H₀ were true, of a test statistic at least as extreme as the one observed. Small p → reject H₀.
- Two ways to be wrong: Type I (reject a true H₀; probability α) and Type II (miss a real effect; probability β). Power = 1 − β.
- "Not significant" is not "no effect": with noisy data, tests have little power.
The engine in five steps
- State the hypotheses. H₀ always contains the equality (μ = μ₀). Hₐ is two-tailed (μ ≠ μ₀), left-tailed (μ < μ₀) or right-tailed (μ > μ₀) — decided by the question, before seeing the data.
- Choose the significance level α — the Type I error risk you accept (commonly 0.05).
- Compute the test statistic: how many standard errors the estimate sits from the H₀ value.
- Find the p-value (or compare with the critical value).
- Decide and interpret in plain words: reject H₀ if p ≤ α; otherwise "the data do not provide sufficient evidence".
Two ways to be wrong
| H₀ is true (no hydrocarbons) | H₀ is false (real discovery) | |
|---|---|---|
| Reject H₀ (declare a find) | Type I error — a costly appraisal campaign chases nothing. P = α | Correct — power = 1 − β |
| Do not reject H₀ (walk away) | Correct | Type II error — a real field is abandoned. P = β |
For a fixed sample size, lowering α raises β: you cannot make both errors small at once without more data. Pick α by asking which mistake costs more.
Critical values vs p-values
The critical-value approach asks: is the statistic inside the rejection region? The p-value approach asks: how much tail area lies beyond it? They always agree; the p-value carries more information because it shows how strong the evidence is (p = 0.049 and p = 0.0001 are both "significant at 5 %", but they are not equally convincing).
| p-value | Evidence against H₀ |
|---|---|
| p > 0.10 | weak or none |
| 0.05 < p ≤ 0.10 | moderate |
| 0.01 < p ≤ 0.05 | strong |
| p ≤ 0.01 | very strong |
Case 1: are U.S. crude inventories drifting?
Each week the EIA reports U.S. commercial crude stocks. Over 2015–2024, did the average weekly change differ from zero — a systematic build or draw?
Weekly change in U.S. crude oil stocks, 2015–2024
Thousand barrels per week · curve = normal with the same mean and SD
One-mean z-test (σ known)
One-mean t-test (σ unknown)
n weeks
Mean change x̄
SD s
t statistic
Decision: p = 0.62 > 0.05, so we do not reject H₀: μ = 0. Over the decade, builds and draws balance out within noise. Note the huge week-to-week SD (≈ 5.5 million barrels) compared with the mean change — that ratio is what makes the test weak.
Power: could we even have seen a drift?
Suppose stocks truly built by δ thousand barrels per week. Power is the chance the test would detect it. With the observed SD, a 300 kbbl/week drift — 15.6 million barrels a year — would be detected only 24% of the time with ten years of data, and 7% with one year.
Power rises with the true effect, the sample size and α, and falls with the variability. Before running any test, ask whether it has a realistic chance of detecting the effect you care about — otherwise "not significant" tells you almost nothing.
When normality is doubtful: Wilcoxon signed-rank test
The t-test assumes a normal population (or large n). The Wilcoxon signed-rank test only assumes a symmetric distribution. It ranks the absolute differences |x − μ₀|, gives each rank the sign of its difference, and sums the positive ranks (W). Because it uses ranks, a single extreme week (a hurricane, a pandemic) cannot dominate. Here: p = 0.87 — the same conclusion. Methods like this, which do not assume a particular distribution, are called nonparametric; t and z are parametric.
Case 2: is "one in three" still true?
A long-quoted industry rule of thumb says roughly one wildcat in three finds hydrocarbons. Do recent Norwegian results contradict it?
One-proportion z-test
Uses p₀ (not p̂) in the standard error, because the null distribution is what we evaluate. Needs n p₀ and n (1 − p₀) ≥ 5.
H₀: p = 1/3, Hₐ: p ≠ 1/3. With p̂ = 0.5085 from n = 295: z = 6.38, p < 0.0001. Very strong evidence that Norwegian wildcats of 2015–2024 succeed more often than one in three. (Why? Mature basins, better seismic, and many "near-field" wildcats close to known fields — a reminder that the population matters.)
Statistical vs practical significance
With enough data, a trivially small difference becomes "significant"; with too little, an important one does not. Always report the effect size with a confidence interval next to the p-value, and judge importance in engineering or business units.
Which one-mean procedure?
| Situation | Procedure |
|---|---|
| σ known, normal population or large n, no outliers | one-mean z-test |
| σ unknown, normal population or large n, no outliers | one-mean t-test |
| Symmetric but non-normal, or outliers present | Wilcoxon signed-rank test |
| Skewed, small n | transform, or bootstrap / permutation (ch 12) |
Do it in code
s = pd.read_sql("select week, stocks_kbbl from crude_stocks_weekly order by week", con).set_index("week").stocks_kbbl
chg = s.diff().loc["2015":"2024"].dropna()
stats.ttest_1samp(chg, 0) # t = 0.50, p = 0.62
stats.wilcoxon(chg) # p = 0.87
stats.binomtest(150, 295, 1/3) # exact test of p = 1/3, p < 1e-9
from statsmodels.stats.power import TTestPower
TTestPower().power(effect_size=300/chg.std(), nobs=len(chg), alpha=0.05) # 0.24with c as (select week, stocks_kbbl - lag(stocks_kbbl) over (order by week) as d
from crude_stocks_weekly)
select count(*) as n, avg(d) as xbar, stddev_samp(d) as s,
avg(d) / (stddev_samp(d) / sqrt(count(*))) as t -- 0.50
from c
where d is not null and extract(year from week) between 2015 and 2024;Key takeaways
- One engine: hypotheses → α → test statistic → p-value → decision in plain words.
- p is computed assuming H₀ is true; it is not the probability that H₀ is true.
- Plan for power. Noisy data make "not significant" uninformative.
- Rank-based tests (Wilcoxon) protect against outliers and non-normality.
Self-check
1 · Set up the hypotheses
A refinery claims its new control system raised average weekly utilisation above 92 %. State H₀ and Hₐ.
H₀: μ = 92 (no improvement); Hₐ: μ > 92 — a right-tailed test, because the claim is directional.
2 · Read the p-value
A test gives p = 0.03. Is it correct to say "there is a 3 % chance H₀ is true"?
No. It means: if H₀ were true, results at least this extreme would occur 3 % of the time. The probability that H₀ is true is a Bayesian question (chapter 12).
3 · Error type
The stocks test failed to reject H₀. If stocks were in fact drifting upward, which error occurred?
A Type II error — a real effect was missed. With power around 24 % for a 300 kbbl/week drift, that is quite likely.