Data & Reproducibility

Reproduction

# 1. Pull raw MPD JSON slices (Hugging Face mirror)
mkdir -p data/raw
for start in 0 1000 10000 11000 12000 13000 14000 15000 \
            102000 103000 104000 105000 106000 107000 \
            108000 109000 110000 111000 112000 113000; do
  end=$((start + 999)); printf -v rs "%d-%d" "$start" "$end"
  curl -sL -o "data/raw/mpd.slice.${rs}.json" \
    "https://huggingface.co/datasets/jaxliu/Spotify_Million_Playlist_Dataset_Challenge/resolve/main/data/mpd.slice.${rs}.json"
done

# 2. Build the analytic dataframe
python3 src/build_dataset.py        # → data/tracks.parquet (~73 MB)

# 3. Three analysis stages
python3 src/species_abundance.py    # SAD log-series vs TPL fit + figure
python3 src/species_area.py         # SAR slope in two regimes + figure
python3 src/taste_tribes.py         # bipartite modularity + retention vs null

# 4. Stage outputs to the Hugo site
python3 src/build_site.py           # copies JSONs + figures into site/data, site/static/figures
cd site && hugo --minify            # produces site/public/

# 5. Visual check before deploy
python3 -m http.server 8765 --bind 127.0.0.1 --directory site/public

Total runtime: ~12 minutes including the network pulls. Memory peak ≈ 1.5 GB.

Inputs

Stage	Source	Size	Notes
MPD slices	HuggingFace `jaxliu/Spotify_Million_Playlist_Dataset_Challenge`	≈ 660 MB	20 contiguous slice files (0–999, 1000–1999, 10000–10999 … 113000–113999)
`tracks.parquet`	`build_dataset.py`	≈ 73 MB	One row per (playlist, track)
Species list	`species_abundance.py`	—	259,652 unique URIs
Track co-occurrence	`taste_tribes.py`	—	Sparse 19K × 65K, thresholded to ≥ 3

Outputs

Output	File	Schema
SAD summary	`data/sad_summary.json`	`{n_species, n_individuals, logseries.{alpha,logL,AIC,BIC}, zsm.{theta}, tpl.{alpha,logL,AIC,BIC}}`
SAR summary	`data/sar_summary.json`	`{sample_sizes, mean_S, std_S, z_small_N, z_large_N, slope_ratio}`
Tribe summary	`data/tribes_summary.json`	`{n_communities_total, modularity_Q, top_clusters_retention: [η, real, null_mean, ...]}`
Figures	`figures/sad_.png`, `sar_.png`, `retention_vs_null.png`	144-dpi PNG, dark theme

Caveats

Subsample selection bias. This is a non-random (stratified contiguous) slice of the MPD. Different MPD pid-ranges will have slightly different SAD statistics; however, the relational metrics (slope ratio, retention enrichment ratio, modularity Q) are robust across slice origins.
Co-occurrence threshold. Setting min-co ≥ 3 discards weak edges but may split giant components. Tuned for stable community detection.
Monoplex Louvain may be sub-optimal relative to bipartite-Leiden, but the qualitative result is robust.
No audio features used. Future work could combine the playlist-incidence matrix with Spotify audio features (valence, energy, tempo, …) to test whether niche tribes cluster along audio-feature axes.

Methods summary

                ┌──────────────────────┐
                │   MPD raw JSON       │  (HuggingFace mirror, 660MB)
                │   20K playlists      │
                └──────────┬───────────┘
                           │
              parse + tally (build_dataset.py)
                           │
                ┌──────────▼───────────┐
                │  tracks.parquet      │   1.34M rows
                │  (pid × track_uri)   │
                └─┬─────────┬─────────┬─┘
                  │         │         │
   species_abundance.py  species_area.py  taste_tribes.py
                  │         │         │
              SAD + 3 fits   SAR × 2     community Q
              + Preston      regimes     + retention/null
                  │         │         │
                  ▼         ▼         ▼
            figures/sad_*   sar_*    retention_vs_null.png
            data/sad_summary sar_summary  tribes_summary

The three analyses use the same input file but answer different questions:

species_abundance tests H0 (can neutral log-series fit the SAD?)
species_area tests H1 (does the SAR show dual scaling?)
taste_tribes tests H2 (do taste-tribe clusters have above-null retention?)