Full audit · regenerated from each repo's main
Dataset registry — Python lecture family
1Hosting patterns in use
Eight distinct patterns. The 2026-07-15 audit found seven; adding lecture-wasm surfaced the sibling repo URL pattern, and the migration added data-lectures while retiring legacy-repo URL entirely.
| Pattern | What it looks like | Refs | Where |
|---|---|---|---|
| data-lectures | reads this repo's published tree — the target state | 7 | intro, programming, python.myst |
| own-repo URL | fetches a file committed in its own repo via a GitHub raw URL | 17 | advanced.myst, intro, programming, python.myst |
| local path | pd.read_csv('…') relative path — no URL; breaks in Colab/download | 8 | advanced.myst, intro |
| sibling repo URL | fetches another lecture repo's committed copy by URL | 11 | wasm |
| external data repo | QuantEcon/high_dim_data via raw and media (LFS) hosts | 12 | intro, wasm |
| written by the lecture itself, then read back | 11 | dp, intro, jax, programming, python.myst | |
| live API | fetched at build time from a third-party service | 16 lectures | advanced.myst, intro, programming, python.myst, wasm |
5 URL spellings for "raw file on GitHub"
- github.com/{org}/{repo}/raw/refs/heads/{ref}/…
- github.com/{org}/{repo}/raw/{ref}/…
- media.githubusercontent.com/media/{org}/{repo}/{ref}/…
- raw.githubusercontent.com/{org}/{repo}/refs/heads/{ref}/…
- raw.githubusercontent.com/{org}/{repo}/{ref}/…
Down from six — the blob/master…?raw=true spelling retired with the
legacy-repo refs. One canonical spelling is a styleguide question for
QuantEcon.manual#108.
2Registry A — static dataset files (36)
Every distinct file a lecture reads, grouped by the repo that owns the bytes. lecture-wasm mirrors intro's lectures and reads intro's copies by URL, so it appears as a consumer, not an owner.
| File | Contents | Lecture(s) | Hosting | Flags / notes |
|---|---|---|---|---|
| data-lectures — 4 files | ||||
| countries.csv | WorldData.info country reference table | pandas_panel | data-lectures | ✓ migrated |
| employ.csv | Eurostat employment in Europe — by age and sex, 2007–2016 | pandas_panel | data-lectures | ✓ migrated |
| lingcod_msy_recovery.csv | Pacific Coast lingcod — biomass and fishing pressure relative to MSY | msy_fishery | data-lectures | ✓ migrated |
| realwage.csv | OECD real minimum wages — 32 countries, 2006–2016 | pandas_panel | data-lectures | ✓ migrated |
| lecture-python-intro — 19 files | ||||
| SCF_plus_mini.csv | SCF+ wealth/income survey extract | inequality, inequality (wasm) | external data repo | built by generating_mini.md (executable MyST) in high_dim_data from SCF_plus.dta |
| SCF_plus_mini_no_weights.csv | SCF+ extract, no survey weights | mle, mle (wasm) | external data repo | variant of the SCF_plus_mini pipeline. Was pinned to feature branch update_scf_noweights (critical flag in the 2026-07-15 audit); reads high_dim_data main since lecture-python-intro#793 |
| assignat.xlsx | Assignat issuance / price data, French Revolution | french_rev, french_rev (wasm) | own-repo URL sibling repo URL | |
| caron.npy | French Revolution money balances (Caron) | french_rev, french_rev (wasm) | local path sibling repo URL | ⚠ breaks in Colab/notebook download hand-prepared NumPy array, no construction record |
| chapter_3.xlsx | Hyperinflation tables, Sargent "Ends of Four Big Inflations" | inflation_history, inflation_history (wasm) | own-repo URL sibling repo URL | hand-transcribed by authors |
| cities_brazil.csv | Brazilian city populations | heavy_tails, heavy_tails (wasm) | external data repo | via media.githubusercontent (LFS) manual download from worldpopulationreview.com, documented in high_dim_data README |
| cities_us.csv | US city populations | heavy_tails, heavy_tails (wasm) | external data repo | via media.githubusercontent (LFS) manual download from worldpopulationreview.com, documented in high_dim_data README |
| dette.xlsx | French government debt series | french_rev, french_rev (wasm) | own-repo URL sibling repo URL | |
| fig_3.xlsx | French Revolution fiscal data | french_rev, french_rev (wasm) | own-repo URL sibling repo URL | |
| forbes-billionaires.csv | Forbes billionaires list | heavy_tails, heavy_tails (wasm) | external data repo | via media.githubusercontent (LFS) webscrape_forbes.ipynb in high_dim_data (Forbes API) |
| forbes-global2000.csv | Forbes Global 2000 firm size | heavy_tails, heavy_tails (wasm) | external data repo | via media.githubusercontent (LFS) webscrape_forbes.ipynb in high_dim_data (Forbes API) |
| graph.txt | 15-node weighted digraph for the shortest-path problem | short_path (wasm) | sibling repo URL | maintained as %%file blocks in intro, dp and jax; lecture-wasm instead fetches intro's committed copy by URL (requests.get), making that copy load-bearing — it was merely "shadowed" in the 2026-07-15 audit |
| japan_population_by_age.xlsx | Japan population by age group (prob_dist bimodality example) | prob_dist | local path | ⚠ breaks in Colab/notebook download new since the 2026-07-15 audit added by lecture-python-intro#790; upstream source not yet recorded |
| life-expectancy-vs-gdp-per-capita.csv | Our World in Data, life expectancy vs GDP per capita | simple_linear_regression, simple_linear_regression (wasm) | own-repo URL sibling repo URL | OWID export, as downloaded |
| longprices.xls | Long-run price levels (Sargent–Velde) | inflation_history, inflation_history (wasm) | own-repo URL sibling repo URL | hand-transcribed by authors |
| mpd2020.xlsx | Maddison Project Database 2020 (GDP per capita, long run) | long_run_growth, long_run_growth (wasm) | own-repo URL sibling repo URL | |
| nom_balances.npy | Nominal balances, French Revolution | french_rev, french_rev (wasm) | local path sibling repo URL | ⚠ breaks in Colab/notebook download hand-prepared NumPy array, no construction record |
| us_adult_heights.csv | Heights of US adults, from NHANES (prob_dist motivating example) | prob_dist | local path | ⚠ breaks in Colab/notebook download new since the 2026-07-15 audit added by lecture-python-intro#790; NHANES-derived, extraction not scripted |
| usa-gini-nwealth-tincome-lincome.csv | US Gini coefficients, wealth & income (long series) | inequality, inequality (wasm) | own-repo URL sibling repo URL | built by inequality/data.ipynb (committed beside it) from SCF_plus_mini |
| lecture-python-programming — 1 files | ||||
| test_pwt.csv | Penn World Table 7.0 extract | pandas | own-repo URL | |
| lecture-python.myst — 6 files | ||||
| fp.dta | Treisman (2016) Russia billionaires replication data | mle | own-repo URL | |
| hansen_singleton_1982_data.csv | FRED-derived consumption/returns snapshot | hansen_singleton_1982 | own-repo URL | make_data.py + README beside the data (exact series and sample documented) |
| hansen_singleton_1983_data.csv | FRED-derived snapshot | hansen_singleton_1983 | own-repo URL | make_data.py + README beside the data |
| maketable1.dta | Acemoglu–Johnson–Robinson (2001) replication, table 1 | ols | own-repo URL | |
| maketable2.dta | AJR (2001) replication, table 2 | ols | own-repo URL | |
| maketable4.dta | AJR (2001) replication, table 4 | ols | own-repo URL | |
| lecture-python-advanced.myst — 6 files | ||||
| acs_data_summary.csv | ACS occupation summary | match_transport | local path | ⚠ breaks in Colab/notebook download construction (ACS filters, grouping, sorting) described in lecture prose only |
| bbh_macro_quarterly.csv | Macro quarterly series (Bhandari et al.) | subjective_beliefs_business_cycles | local path | ⚠ breaks in Colab/notebook download extracted from the paper's replication package (Zenodo DOI, CC BY 4.0); extraction not scripted |
| bbh_michigan_monthly.csv | Michigan survey monthly series (Bhandari et al.) | subjective_beliefs_business_cycles | local path | ⚠ breaks in Colab/notebook download same Zenodo replication package; extraction not scripted |
| dataBHS.mat | US consumption/income series, MATLAB replication bundle | five_preferences | local path | ⚠ breaks in Colab/notebook download lives at lectures/ root |
| fred_data.csv | FRED snapshot — GS1, GS5, GS10, DFII5, DFII10, USREC | risk_aversion_or_mistaken_beliefs | own-repo URL | lecture names the FRED series; no build script anywhere |
| hansen_jagannathan_1991_data.json | Hansen–Jagannathan (1991) asset-returns bundle | hansen_jagannathan_1991 | own-repo URL | lecture documents 3 sources (FRED yields deflated by CPIAUCSL, …); no build script |
3Registry B — data embedded in lecture source (8 files)
Written to the working directory by the lecture itself (%%file,
%%writefile, or an in-lecture open(…, 'w')), then read back.
Self-contained everywhere — including Colab — at the cost of data living inside narrative source.
| File | Lecture(s) | Series |
|---|---|---|
| graph.txt | short_path | dp · intro · jax — same data maintained in 3 repos |
| newfile.txt | python_essentials | programming |
| numbers.txt | debugging | programming |
| output.txt | python_essentials | programming |
| output2.txt | python_essentials | programming |
| test_table.csv | python_advanced_features | programming |
| us_cities.txt | python_advanced_features, python_essentials | programming |
| web_graph_data.txt | finite_markov | python.myst |
4Registry C — live API data (16 lectures)
Fetched at build time from third-party services — the build-reproducibility surface, and the blocker for the WASM/JupyterLite target (pyodide cannot reach most of these APIs; that is lecture-wasm's core problem, meta#143).
| Provider / access | Series or tickers | Lecture | Series | Why live |
|---|---|---|---|---|
| FRED fredgraph.csv URL | PCND, PCESV, DPCERD3Q086SBEA, CNP16OV | doubts_or_variability | advanced.myst | incidental |
| FRED fredgraph.csv URL | USREC | subjective_beliefs_business_cycles | advanced.myst | incidental |
| FRED pandas_datareader | UNRATE, USREC, M0892AUSM156SNBR, UMCSENT, CPILFESL, INDPRO | business_cycle | intro | mixed snapshot pipeline already exists (data-lectures business_cycle_data.csv) but is unadopted |
| FRED fredgraph.csv URL | UNRATE | pandas | programming | API is the lesson |
| FRED pandas_datareader | UNRATE | unemployment_linear | python.myst | incidental |
| FRED pandas_datareader | UNRATE | unemployment_shocks | python.myst | incidental |
| FRED pandas_datareader | UNRATE, USREC, M0892AUSM156SNBR, UMCSENT, CPILFESL, INDPRO | business_cycle | wasm | mixed mirror of intro's business_cycle; cannot run under pyodide |
| FRED pandas_datareader | historical mirror — intro has since moved off pandas_datareader here | heavy_tails | wasm | incidental mirror of an older intro heavy_tails |
| World Bank wbgapi | NY.GDP.MKTP.KD.ZG, SL.UEM.TOTL.NE.ZS, FS.AST.PRVT.GD.ZS | business_cycle | intro | mixed teaches wb.series.info querying and metadata as part of the narrative |
| World Bank wbgapi | NY.GDP.MKTP.CD | heavy_tails | intro | incidental |
| World Bank wbgapi | SI.POV.GINI, NY.GDP.PCAP.KD | inequality | intro | mixed demonstrates searching for the Gini series ID with wbgapi |
| World Bank wbgapi | GC.DOD.TOTL.GD.ZS | pandas | programming | API is the lesson the section is literally "Using wbgapi and yfinance to Access Data" |
| World Bank wbgapi | NY.GDP.MKTP.KD.ZG, SL.UEM.TOTL.NE.ZS, FS.AST.PRVT.GD.ZS | business_cycle | wasm | mixed mirror of intro's business_cycle; cannot run under pyodide |
| World Bank wbgapi | SI.POV.GINI, NY.GDP.PCAP.KD | inequality | wasm | mixed mirror of intro's inequality |
| Yahoo Finance yfinance | CT=F | commod_price | intro | incidental |
| Yahoo Finance yfinance | AMZN, BTC-USD | heavy_tails | intro | incidental |
| Yahoo Finance yfinance | AMZN, COST | prob_dist | intro | incidental |
| Yahoo Finance yfinance | 11-ticker exercise list | pandas | programming | API is the lesson |
| Yahoo Finance yfinance | ^IXIC | kesten_processes | python.myst | incidental |
| Yahoo Finance yfinance | CT=F | commod_price | wasm | incidental mirror of intro's commod_price |
| Yahoo Finance yfinance | AMZN, BTC-USD | heavy_tails | wasm | incidental mirror of intro's heavy_tails |
| Yahoo Finance yfinance | AMZN, COST | prob_dist | wasm | incidental mirror of intro's prob_dist |
5Cross-series reuse
Files consumed by more than one repo. Before the migration the only true cross-series files were the pandas_panel trio; the wasm mirror now multiplies every intro dataset into a second consumer.
| Dataset | Consumers | How it's shared today |
|---|---|---|
| SCF_plus_mini.csv | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| SCF_plus_mini_no_weights.csv | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| assignat.xlsx | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| caron.npy | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| chapter_3.xlsx | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| cities_brazil.csv | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| cities_us.csv | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| countries.csv | programming + python.myst | all consumers read data-lectures — the target state |
| dette.xlsx | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| employ.csv | programming + python.myst | all consumers read data-lectures — the target state |
| fig_3.xlsx | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| forbes-billionaires.csv | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| forbes-global2000.csv | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| life-expectancy-vs-gdp-per-capita.csv | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| longprices.xls | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| mpd2020.xlsx | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| nom_balances.npy | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| realwage.csv | programming + python.myst | all consumers read data-lectures — the target state |
| usa-gini-nwealth-tincome-lincome.csv | intro + wasm | wasm mirrors intro and reads intro's copy by URL |
| graph.txt | dp + intro + jax | duplicated as %%file blocks — 3 copies of the same data to keep in sync |
6Committed but unreferenced data files (35)
Files in a repo's tree that no executed code cell reads. Orphan-sweep candidates are tracked in meta#337; shadowed and exercise-download files are functional, just invisible to a URL-level audit.
| File | State | Why |
|---|---|---|
| lecture-python-intro — 3 files | ||
| lectures/datasets/GDP_per_capita_world_bank.csv | orphan | no reference anywhere; the QuantEcon/data copy was dropped at Phase 2 |
| lectures/datasets/Metadata_Country_API_NY.GDP.PCAP.CD_DS2_en_csv_v2_4770417.csv | orphan | metadata twin of the World Bank GDP-per-capita orphan |
| lectures/datasets/fig_3.ods | orphan | source-format twin of fig_3.xlsx; referenced by nothing |
| lecture-python-programming — 5 files | ||
| lectures/_static/lecture_specific/pandas/data/ticker_data.csv | orphan | no reference anywhere |
| lectures/_static/lecture_specific/python_advanced_features/numbers.txt | debugging lecture writes it with %%file | |
| lectures/_static/lecture_specific/python_advanced_features/test_table.csv | exercise download | prose exercise link tells the reader to download this exact URL — keep |
| lectures/_static/lecture_specific/python_foundations/test_table.csv | orphan | duplicate of the python_advanced_features copy |
| lectures/_static/lecture_specific/python_foundations/us_cities.txt | python_essentials writes it with %%writefile | |
| lecture-python.myst — 2 files | ||
| lectures/_static/lecture_specific/finite_markov/web_graph_data.txt | finite_markov writes it with %%file | |
| lectures/web_graph_data.txt | shadowed duplicate at lectures/ root | |
| lecture-dp — 10 files | ||
| lectures/_static/lecture_specific/finite_markov/web_graph_data.txt | orphan | inherited from python.myst; no finite_markov data reference in this repo |
| lectures/_static/lecture_specific/match_transport/acs_data_summary.csv | orphan | inherited copy, no consuming lecture in this repo |
| lectures/_static/lecture_specific/mle/fp.dta | orphan | inherited from python.myst, no consuming lecture |
| lectures/_static/lecture_specific/ols/maketable1.dta | orphan | inherited from python.myst, no consuming lecture |
| lectures/_static/lecture_specific/ols/maketable2.dta | orphan | inherited from python.myst, no consuming lecture |
| lectures/_static/lecture_specific/ols/maketable4.dta | orphan | inherited from python.myst, no consuming lecture |
| lectures/_static/lecture_specific/pandas_panel/countries.csv | orphan | inherited copy; the consuming lectures live in other repos and now read data-lectures |
| lectures/_static/lecture_specific/pandas_panel/employ.csv | orphan | inherited copy; the consuming lectures live in other repos and now read data-lectures |
| lectures/_static/lecture_specific/pandas_panel/realwage.csv | orphan | inherited copy; the consuming lectures live in other repos and now read data-lectures |
| lectures/graph.txt | short_path regenerates it via %%file | |
| lecture-wasm — 14 files | ||
| lectures/_static/lecture_specific/inequality/usa-gini-nwealth-tincome-lincome.csv | wasm's inequality reads intro's copy by URL | |
| lectures/_static/lecture_specific/simple_linear_regression/life-expectancy-vs-gdp-per-capita.csv | wasm's simple_linear_regression reads intro's copy by URL | |
| lectures/datasets/GDP_per_capita_world_bank.csv | orphan in intro, mirrored into wasm | |
| lectures/datasets/Metadata_Country_API_NY.GDP.PCAP.CD_DS2_en_csv_v2_4770417.csv | orphan in intro, mirrored into wasm | |
| lectures/datasets/assignat.xlsx | wasm's french_rev reads intro's copy by URL | |
| lectures/datasets/caron.npy | wasm's french_rev reads intro's copy by URL | |
| lectures/datasets/chapter_3.xlsx | wasm's inflation_history reads intro's copy by URL | |
| lectures/datasets/dette.xlsx | wasm's french_rev reads intro's copy by URL | |
| lectures/datasets/fig_3.ods | orphan in intro, mirrored into wasm | |
| lectures/datasets/fig_3.xlsx | wasm's french_rev reads intro's copy by URL | |
| lectures/datasets/longprices.xls | wasm's inflation_history reads intro's copy by URL | |
| lectures/datasets/mpd2020.xlsx | wasm's long_run_growth reads intro's copy by URL | |
| lectures/datasets/nom_balances.npy | wasm's french_rev reads intro's copy by URL | |
| lectures/graph.txt | wasm's short_path fetches intro's committed graph.txt by URL, not this copy | |
| continuous_time_mcs — 1 files | ||
| lectures/old_stuff.txt | orphan | scratch file at lectures/ root; referenced by nothing |
7Provenance — how each file came to exist
The axis the manifest convention formalizes (verbatim / constructed / dynamic snapshot, meta#336). For migrated datasets this comes from their manifests; for everything else, from the curated audit annotations.
| File | Provenance | Recorded in |
|---|---|---|
| verbatim — 8 | ||
| countries.csv | WorldData.info country reference table | manifest |
| dataBHS.mat | US consumption/income series, MATLAB replication bundle | audit annotation |
| fp.dta | Treisman (2016) Russia billionaires replication data | audit annotation |
| life-expectancy-vs-gdp-per-capita.csv | OWID export, as downloaded | audit annotation |
| maketable1.dta | Acemoglu–Johnson–Robinson (2001) replication, table 1 | audit annotation |
| maketable2.dta | AJR (2001) replication, table 2 | audit annotation |
| maketable4.dta | AJR (2001) replication, table 4 | audit annotation |
| mpd2020.xlsx | Maddison Project Database 2020 (GDP per capita, long run) | audit annotation |
| constructed · builder committed — 7 | ||
| SCF_plus_mini.csv | built by generating_mini.md (executable MyST) in high_dim_data from SCF_plus.dta | audit annotation |
| SCF_plus_mini_no_weights.csv | variant of the SCF_plus_mini pipeline. Was pinned to feature branch update_scf_noweights (critical flag in the 2026-07-15 audit); reads high_dim_data main since lecture-python-intro#793 | audit annotation |
| forbes-billionaires.csv | webscrape_forbes.ipynb in high_dim_data (Forbes API) | audit annotation |
| forbes-global2000.csv | webscrape_forbes.ipynb in high_dim_data (Forbes API) | audit annotation |
| hansen_singleton_1982_data.csv | make_data.py + README beside the data (exact series and sample documented) | audit annotation |
| hansen_singleton_1983_data.csv | make_data.py + README beside the data | audit annotation |
| usa-gini-nwealth-tincome-lincome.csv | built by inequality/data.ipynb (committed beside it) from SCF_plus_mini | audit annotation |
| constructed · pipeline lost — 9 | ||
| acs_data_summary.csv | construction (ACS filters, grouping, sorting) described in lecture prose only | audit annotation |
| bbh_macro_quarterly.csv | extracted from the paper's replication package (Zenodo DOI, CC BY 4.0); extraction not scripted | audit annotation |
| bbh_michigan_monthly.csv | same Zenodo replication package; extraction not scripted | audit annotation |
| employ.csv | Eurostat employment in Europe — by age and sex, 2007–2016 | manifest |
| fred_data.csv | lecture names the FRED series; no build script anywhere | audit annotation |
| hansen_jagannathan_1991_data.json | lecture documents 3 sources (FRED yields deflated by CPIAUCSL, …); no build script | audit annotation |
| lingcod_msy_recovery.csv | Pacific Coast lingcod — biomass and fishing pressure relative to MSY | manifest |
| realwage.csv | OECD real minimum wages — 32 countries, 2006–2016 | manifest |
| us_adult_heights.csv | added by lecture-python-intro#790; NHANES-derived, extraction not scripted | audit annotation |
| author-assembled — 11 | ||
| assignat.xlsx | Assignat issuance / price data, French Revolution | audit annotation |
| caron.npy | hand-prepared NumPy array, no construction record | audit annotation |
| chapter_3.xlsx | hand-transcribed by authors | audit annotation |
| cities_brazil.csv | manual download from worldpopulationreview.com, documented in high_dim_data README | audit annotation |
| cities_us.csv | manual download from worldpopulationreview.com, documented in high_dim_data README | audit annotation |
| dette.xlsx | French government debt series | audit annotation |
| fig_3.xlsx | French Revolution fiscal data | audit annotation |
| japan_population_by_age.xlsx | added by lecture-python-intro#790; upstream source not yet recorded | audit annotation |
| longprices.xls | hand-transcribed by authors | audit annotation |
| nom_balances.npy | hand-prepared NumPy array, no construction record | audit annotation |
| test_pwt.csv | Penn World Table 7.0 extract | audit annotation |
| toy — 1 | ||
| graph.txt | maintained as %%file blocks in intro, dp and jax; lecture-wasm instead fetches intro's committed copy by URL (requests.get), making that copy load-bearing — it was merely "shadowed" in the 2026-07-15 audit | audit annotation |
8Draft rules for styleguide/datasets.md
Distilled from the live-API pedagogy analysis (§4) and the migration pilots; feeding QuantEcon.manual#108.
A lecture reads data from a stable snapshot URL (data-lectures). Live API calls are the exception and require a reason recorded in the lecture source.
A live call is justified when the fetch workflow is itself the lesson (the pandas lecture's wbgapi/yfinance sections) or when currency is the point. “The lecture needs series X” is not a reason — that's what snapshots are for.
A refresh builder in data-lectures producing the snapshot (the business_cycle_data.csv pattern, already prototyped). Breakage becomes a one-line URL switch, and the WASM build always uses the twin — pyodide cannot reach the live APIs at all.
Where live access stays, fredgraph.csv-style URLs beat pandas_datareader-style wrappers: one less dependency, and the git history shows the wrappers are what broke.
Source, series IDs, retrieval date, license and refresh cadence — the manifest schema in this repo is the template; the provenance classes in §7 say which fields are required.