Data provenance

Every number on this site is traceable back to a specific data source, a specific script that pulls it, and a specific cache file. Every pull is reproducible with the code at github.com/ericscheier/emburdensynth.

The coverage table below is built at render time from the cache files themselves — the numbers you see are the current counts in each RDS, not frozen literals from when this page was first written.

Source Coverage Retrieval script
WB WDI (indicators) 179 countries × 1960–2024 × 7 indicators build_wb_wdi_1960.R
WB Gini historical 171 countries × 1970–2024 build_temporal_deep_history.R
WB GCD 2010 92 countries × 11 sectors (pre-pipeline)
WB Clean-fuel access 200 countries × 2000–2023 build_who_air_pollution.R
WGI (governance) 214 countries × 1996–2022 × 6 dimensions fix_wgi_v2.R
Eurostat HBS 34 countries × 1988–2020 × 5 quintiles build_eurostat_quintile_spread.R
Eurostat energy poverty ~30 countries × 2010–2023 (cached prior)
Ember Electricity Data 213 countries × 2000–2025 × 9 fuel categories build_ember_by_fuel.R
OWID Energy 220 countries × 1900–2025 × 11 variables build_owid_energy.R
OWID CO2 218 countries × 1750–2024 × per-capita + cumulative + by-fuel build_owid_co2.R
EIA International 222 countries × 1980–2023 build_eia_international.R
WHO PM2.5 200 countries × 1990–2023 build_who_air_pollution.R
FAOSTAT food security 183 countries × 2000–2023 build_fao_food_security.R
UNDP HDI 204 countries × 1990–2022 build_undp_hdi.R
NASA GISTEMP v4 1880–2025 global + zonal build_nasa_giss.R
WorldPop 1 km rasters 129 countries × 2020 (pre-pipeline)
GADM v4.1 boundaries 255 countries × admin-0 (via emburdengeo)
ESMAP Multi-Tier Framework 28 countries in manifest (surveys click-through XLSX) build_esmap_mtf_tiers.R
NREL ResStock + RECS + EULP US only, adapted globally emburdendata::download_*
CDC HHI + WONDER + PLACES US census tracts emburdendata

Why so many sources?

Household energy burden is not a single-dataset object. Estimating it requires:

  • Household expenditure surveys to anchor the sector shares (GCD, Eurostat HBS)
  • Spatial density rasters to allocate the surveys to cells (WorldPop, RWI)
  • Utility tariff schedules to convert dollars into kWh (URDB)
  • Degree-day climatology to compute physics-required demand (WB CCKP)
  • Grid generation mix to attribute CO2 per kWh (Ember) — and, since Round 20, to attribute transformity (sej/J) per kWh through Odum’s emergy framework
  • Access + affordability + QoS surveys (WB electricity access, WHO clean-fuel, ESMAP MTF tiers) to complete the burden picture

Each source has its own vintage, coverage, and gaps; the pipeline fills gaps with kNN imputation and documents the source-tier at every cell.

Reproducibility guarantees

  • Every downloader script prefers open, direct-download APIs over commercial data
  • Fallback chains ensure graceful degradation (e.g. EIA International falls back to OWID Energy if API key absent)
  • All downloads write to a consistent local cache (~/.cache/emburdendata/) via rappdirs
  • Cache files are content-addressed (renaming triggers re-download)
  • Full pull scripts + cache locations are documented in docs/references.bib (19 dataset entries, auto-generated)

Three pulls we could not fully automate

  • Yale Environmental Performance Index (EPI) — no direct-download URL published; requires a web form. Not blocking.
  • EM-DAT (CRED disaster database) — requires free CRED registration before bulk XLSX download. See manual setup guide.
  • ESMAP MTF tier surveys — click-through XLSX from the WB Microdata Library; the manifest (28 covered countries) is auto-cached but tier-share workbooks require a manual download per country. See manual setup guide.

Everything else is fully scripted.