Toolkit
Data, Code & Reproducibility
Every example in this course is a real analysis of an open dataset. This page lists the sources and licences, describes the database built from them, and shows how to rerun the whole pipeline.
Datasets and licences
| Dataset | Publisher & licence | Coverage | Used in |
|---|---|---|---|
| Europe Brent and Cushing WTI spot prices, daily (series RBRTE, RWTC) | U.S. Energy Information Administration — U.S. government work, public domain | 1987 – today | 02, 04, 05, 06, 08, 12 |
| Weekly U.S. ending stocks of crude oil excl. SPR (WCESTUS1) | U.S. EIA — public domain | 1982 – today | 07 |
| Weekly refinery utilisation, U.S. and PADD 1–5 | U.S. EIA — public domain | U.S. 1990 –, PADDs 2010 – | 04, 06, 08, 11, 12 |
| Weekly U.S. regular retail gasoline price | U.S. EIA — public domain | 1990 – today | 10, 12 |
| Country energy dataset (production, consumption, per-capita energy …) | Our World in Data, compiled from the Energy Institute Statistical Review of World Energy and others — CC BY 4.0 | historical series to 2024 (snapshot year 2023) | 01, 02 |
| Exploration wellbores, field reserves, monthly field production | Norwegian Offshore Directorate FactPages — Norwegian Licence for Open Government Data (NLOD 2.0) | 1966 – today | 01 – 09, 11, 12 |
| Monthly commodity prices ("Pink Sheet") | World Bank — CC BY 4.0 | 1960 – today | 10 |
Data were downloaded on 18 October 2026; all numbers on the site refer to that snapshot. Raw downloads total about 22 MB. Derived extracts used by the pages are JSON files in statistics/data/: results.json, spot_daily.json, ncs_wells.json, ncs_fields.json, ncs_production.json, refinery_util_weekly.json, crude_stocks_weekly.json, gasoline_crude_monthly.json, owid_countries.json, owid_top_producers.json and wb_monthly.json.
Pipeline
- Fetch the raw files with 01_fetch.sh and 05_fetch_padd.sh.
- Load them into PostgreSQL with etl.py (run by 03_etl.sh inside a Python container built from this Dockerfile). It also writes the JSON extracts.
- Create the helper views with 01_views.sql.
- Analyse with analysis.py (06_analysis.sh), which computes every quoted number into
results.json. - Check the SQL shown on the pages with 02_chapter_checks.sql.
The pages read results.json and fill numbers into the text, so rerunning the pipeline on fresh data updates the prose automatically. Interactive widgets recompute from the JSON extracts in your browser using the course's small statistics library.
Database schema (energystats)
| Table / view | Grain | Key columns |
|---|---|---|
spot_price_daily | day | date, brent, wti (USD/bbl) |
crude_stocks_weekly | week | week, stocks_kbbl |
refinery_utilization_weekly | week × region | week, region (us, padd1–padd5), utilization_pct |
gasoline_retail_weekly | week | week, gasoline_usd_gal |
gasoline_crude_monthly | month | month, gasoline, brent, wti (monthly averages) |
owid_country_energy | country × year | country, year, iso_code, population, oil_production, energy_per_capita, … |
ncs_field_reserves | field | field, recoverable_oe_msm3, recoverable_oil_msm3, recoverable_gas_bsm3 |
ncs_exploration_wells | wellbore | wellbore, purpose, content, main_area, entry_date, entry_year, water_depth_m, drilling_days |
ncs_field_production_monthly | field × month | field, year, month, oil_msm3, gas_bsm3, oe_msm3, water_msm3 |
wb_commodity_prices_monthly | month × commodity | month, commodity, price |
v_wildcats (view) | wildcat wellbore | main_area, entry_year, entry_month, discovery (bool), fluid (oil / gas / oil+gas / none), decade |
v_brent_monthly (view) | month | month, brent, wti |
Definitions used throughout
- Wildcat: exploration wellbore with purpose "WILDCAT" (first test of a prospect). Appraisal wells are excluded from success rates.
- Discovery: reported content oil, gas, oil/gas, gas/condensate or oil/gas/condensate. Dry, shows and water count as no discovery; wells with content "not applicable/available" are excluded.
- Returns: log returns in percent, 100·ln(Pt/Pt−1); monthly returns use monthly average prices.
- Field size: latest recoverable oil-equivalent estimate per field, million standard m³.
Attribution
Contains data from the Norwegian Offshore Directorate (NLOD 2.0), Our World in Data (CC BY 4.0), the World Bank (CC BY 4.0) and the U.S. Energy Information Administration (public domain). Analyses and any errors are this site's own.