Home on MPD Ecology — Playlists as Ecosystems

Playlists as Ecosystems

Tue, 07 Jul 2026 00:00:00 +0000

1. Introduction

The species-abundance distribution (SAD) and species-area relationship (SAR) are two of the most widely studied patterns in ecology. They are the fingerprints ecologists use to compare ecosystems as different as tropical rainforest, the deep-sea abyssal plain, and bacterial mats in Yellowstone hot springs. The unified neutral theory of biodiversity (Hubbell, 2001) predicts that the SAD should take a specific analytical form (the zero-sum multinomial), and the SAR should follow a power law with z ∈ [0.20, 0.35] for contiguous habitats.

Data & Reproducibility

Tue, 07 Jul 2026 00:00:00 +0000

Reproduction

# 1. Pull raw MPD JSON slices (Hugging Face mirror)
mkdir -p data/raw
for start in 0 1000 10000 11000 12000 13000 14000 15000 \
 102000 103000 104000 105000 106000 107000 \
 108000 109000 110000 111000 112000 113000; do
 end=$((start + 999)); printf -v rs "%d-%d" "$start" "$end"
 curl -sL -o "data/raw/mpd.slice.${rs}.json" \
 "https://huggingface.co/datasets/jaxliu/Spotify_Million_Playlist_Dataset_Challenge/resolve/main/data/mpd.slice.${rs}.json"
done

# 2. Build the analytic dataframe
python3 src/build_dataset.py # → data/tracks.parquet (~73 MB)

# 3. Three analysis stages
python3 src/species_abundance.py # SAD log-series vs TPL fit + figure
python3 src/species_area.py # SAR slope in two regimes + figure
python3 src/taste_tribes.py # bipartite modularity + retention vs null

# 4. Stage outputs to the Hugo site
python3 src/build_site.py # copies JSONs + figures into site/data, site/static/figures
cd site && hugo --minify # produces site/public/

# 5. Visual check before deploy
python3 -m http.server 8765 --bind 127.0.0.1 --directory site/public

Total runtime: ~12 minutes including the network pulls. Memory peak ≈ 1.5 GB.

Figures

Tue, 07 Jul 2026 00:00:00 +0000

Cover gallery

Four atmospheric covers for the project, generated to anchor each analytical section in a visual idiom. All cover imagery is editorial photography / 3D concept art — no real brands or institution names.

Cover 1 — "Playlists as habitats". A single observer amid translucent vertical panels of track data, cinematic teal-and-amber lighting. Reference: Teenage Engineering OP-1 product photography and Brompton specification sheets. Used as the home hero.

Methodology

Tue, 07 Jul 2026 00:00:00 +0000

Data acquisition

The Spotify Million Playlist Dataset (MPD) is the canonical corpus for music-recommendation research, released by Spotify and the RecSys Challenge 2018 organisers (Chen et al., 2018). The full corpus (1,000,000 playlists, ≈3.4 GB compressed across 100 JSON slice files) is mirrored on Hugging Face at jaxliu/Spotify_Million_Playlist_Dataset_Challenge. We use 12 contiguous slices of the dataset (20,000 raw playlists), covering pids 0-15999 and 102000-113999.

Reproducibility

All code is open-source and runs against data/tracks.parquet. The pipeline is:

Results

Tue, 07 Jul 2026 00:00:00 +0000

Hypothesis scorecard

#	Hypothesis	Predicted	Observed	Verdict
H1	Dual-scaling species-area relationship (slope ratio > 1.10)	slope ratio ≈ 1.1–1.4	1.28	CONFIRMED
H2	Taste-tribe retention enrichment ≥ 1.5× over degree-preserving null	≥ 5 clusters	8/8 clusters 0.64–0.91× (below null)	REFUTED
H0	Pure-neutral theory (log-series SAD with no deviation)	ΔAIC ≥ 10 favoring log-series	ΔAIC = 1.35 × 10⁶ favoring log-series	CONFIRMED

Species-abundance distribution — H0 confirmed

Model	Parameters	log L	AIC	ΔAIC
Log-series	α = 516.0	548,711	−1,097,419	0
Zero-sum multinomial	θ̂ ≈ 1.2 × 10⁶	(χ²)	≈ −1,097,317	+102
Truncated power law	α = 2.90	−128,248	+256,500	+1,353,919