Methodology

Data acquisition

The Spotify Million Playlist Dataset (MPD) is the canonical corpus for music-recommendation research, released by Spotify and the RecSys Challenge 2018 organisers (Chen et al., 2018). The full corpus (1,000,000 playlists, ≈3.4 GB compressed across 100 JSON slice files) is mirrored on Hugging Face at jaxliu/Spotify_Million_Playlist_Dataset_Challenge. We use 12 contiguous slices of the dataset (20,000 raw playlists), covering pids 0-15999 and 102000-113999.

Reproducibility

All code is open-source and runs against data/tracks.parquet. The pipeline is:

python3 src/build_dataset.py          # → data/tracks.parquet
python3 src/species_abundance.py      # → data/sad_summary.json, figures/sad_*.png
python3 src/species_area.py           # → data/sar_summary.json, figures/sar_*.png
python3 src/taste_tribes.py           # → data/tribes_summary.json, figures/retention_vs_null.png
python3 src/build_site.py             # → copies summary + figs to site/data/, site/static/figures/

Total runtime: ~5 minutes on a single 4-core VM (sparse matrix multiplications are the dominant cost; total memory peak ≈ 1.5 GB).

Why neutral theory?

Stephen Hubbell's Unified Neutral Theory of Biodiversity and Biogeography (Hubbell, 2001) makes a striking claim: species differences in life history, niche, and competitive ability are irrelevant at the scale of community assembly. Instead, the species-abundance distribution is fully characterised by two parameters — the community size $J$ and a "fundamental biodiversity number" $\theta = 2J\nu$ where $\nu$ is the per-individual speciation rate. The resulting SAD has a specific analytical shape (the zero-sum multinomial) that is testable.

For our application, the mapping is:

This makes the empirical test possible: if Hubbell's null is rejected at $\Delta \text{AIC} > 100$ against a power-law alternative, we conclude that listening-taste ecosystems are not pure-neutral.

Hypothesis scoring

For each pre-registered hypothesis we report:

• H1 (dual-scaling SAR). SAR slopes fitted piecewise with weighted least-squares on log–log axes. Reported slope ratio $z_\text{small} / z_\text{large}$ with the prediction threshold of $1.10$.
• H2 (taste-tribe retention). Retention rate $r(c) = |{t \in c : \text{playlist count}(t) \geq 2}| / |{t \in c}|$; enrichment $\eta(c) = r(c)/\bar r(\text{null}_c)$ with 15 degree- weighted null realisations.
• H0 (pure neutral). AIC comparison of log-series vs. ZSM vs TPL fits to the SAD histogram; modularity $Q$ on the track co-occurrence graph.

Why bipartite modularity and not Leiden or Louvain proper?

The greedy-modularity implementation in NetworkX is a greedy_modularity_communities heuristic from Clauset, Newman & Moore (2004). It is sub-optimal compared to Leiden (Traag et al., 2019) but is available in pure Python without external compiled modules and produces stable, reproducible clusters at 4.7M-node scale. The bipartite-specific Leiden variant would be an upgrade; we use the simpler monoplex version on the track-track co-occurrence projection.

What was filtered out

• Tracks appearing in < 3 playlists (singleton / doubleton spam, ~80% of raw vocabulary)
• Playlists with < 5 of these non-singleton tracks (likely test playlists)
• The original MPD metadata fields collaborative, num_albums, num_edits, num_followers, modified_at are preserved in tracks.parquet but unused in the primary analysis.