Chapter 09 · Inference

Categorical Data & Chi-Square

When the data are categories — oil, gas or dry; North Sea or Barents; this decade or that — means and t-tests do not apply. Chi-square procedures compare observed counts with the counts a hypothesis predicts, and one family of formulas answers three different questions.

The chi-square distribution

χ²-curves are right-skewed, start at 0, are indexed by degrees of freedom, have mean = df, and look more and more normal as df grows. The test statistic is compared with the right tail only — large values mean "observed counts are far from expected".

Goodness of fit: is exploration seasonal?

North Sea exploration uses semi-submersibles that drill year-round, but Arctic and winter weather could still push spud dates into summer. H₀: wildcats are spudded uniformly across the 12 months (each month gets 1/12). Data: 694 Norwegian wildcats, 2000–2024.

χ² = 6.01 with 11 df, p = 0.87. The spread across months is entirely consistent with chance — no seasonality. Even the Barents Sea alone (n = 99; χ² = 9.97, p = 0.53) shows none: modern rigs and ice-free southern Barents waters keep drilling year-round. "Fail to reject" is a finding too.

Contingency tables and association

Cross-tabulate two categorical variables on the same wells — the sea area and what was found — and look at the conditional distribution of fluid within each area (each row as percentages). If the variables were unrelated, every row would look like the marginal distribution (the "all areas" row).

The rows clearly differ in what is found — the North Sea is the oil province, the Norwegian and Barents Seas lean to gas — while the "none" share is similar everywhere (chapter 03 saw that success does not depend on area).

Independence test

χ² = 66.7 with 6 df, p < 0.0001 (about 2 × 10⁻¹²). Sea area and fluid type are associated. Where does the association come from? The standardised residual (O − E)/√E for each cell flags the cells that contribute most:

Cell colour: blue = fewer wells than independence predicts, red = more. Each cell shows observed (expected) and the residual. |residual| > 2 is a useful flag.

The North Sea has far more oil-only and far fewer gas finds than expected; the Norwegian Sea has far more gas and less oil; the Barents Sea has more gas and much less oil. Geology — source rocks and burial depth — in one table.

Homogeneity test: has the mix changed over the decades?

The homogeneity test uses exactly the same arithmetic as the independence test, but the question is different: here each decade is treated as a separate population, and we ask whether the distribution of fluid found is the same in all of them.

χ² = 43.5 with 18 df, p = 0.0007: the distributions are not homogeneous. The share of dry wells fell from the pioneering 1960s–70s, and the 2020s show more oil and oil+gas finds as exploration concentrates near existing infrastructure.

All three also work for a discrete quantitative variable with a few possible values (e.g. discoveries per campaign 0–5), treated as categories.

Do it in code

w = pd.read_sql("select * from v_wildcats", con)
obs = w.query("2000 <= entry_year <= 2024").entry_month.value_counts().sort_index()
stats.chisquare(obs)                                  # goodness of fit vs uniform: p = 0.87

ct = pd.crosstab(w.main_area, w.fluid)
chi2, p, dof, expected = stats.chi2_contingency(ct)   # 66.7, 2e-12, 6
resid = (ct - expected) / np.sqrt(expected)           # standardised residuals
v = np.sqrt(chi2 / (ct.values.sum() * (min(ct.shape) - 1)))   # Cramér's V = 0.155

with o as (select main_area, fluid, count(*)::numeric as obs
           from v_wildcats group by 1, 2),
     t as (select o.*,
                  sum(obs) over (partition by main_area) as row_total,
                  sum(obs) over (partition by fluid)     as col_total,
                  sum(obs) over ()                       as n
           from o)
select main_area, fluid, obs,
       round(row_total * col_total / n, 1)                                        as expected,
       round((obs - row_total * col_total / n) / sqrt(row_total * col_total / n), 2) as std_resid,
       sum((obs - row_total * col_total / n)^2 / (row_total * col_total / n)) over () as chi2
from t order by 1, 2;

Self-check