Playlists as Ecosystems

1. Introduction

The species-abundance distribution (SAD) and species-area relationship (SAR) are two of the most widely studied patterns in ecology. They are the fingerprints ecologists use to compare ecosystems as different as tropical rainforest, the deep-sea abyssal plain, and bacterial mats in Yellowstone hot springs. The unified neutral theory of biodiversity (Hubbell, 2001) predicts that the SAD should take a specific analytical form (the zero-sum multinomial), and the SAR should follow a power law with z ∈ [0.20, 0.35] for contiguous habitats.

Music is not an ecosystem — but the analogy is structurally near-perfect. Spotify users curate playlists that are inventory lists of tracks. Users clone tracks from one playlist into another. Tracks go viral; tracks fade. New tracks are minted daily. Niche-tracks find tiny audiences; mainstream tracks appear in half of all playlists. Playlist curators cluster by taste: country playlists, 90s hip-hop throwback, lofi-chill sleep playlists, jazz cigarettes-and-coffee playlists.

In this study, we ask whether the taste-tribe structure of the listening ecosystem can be recovered from the data alone — without audio features, lyric embeddings, or any track-level metadata — using only the user-generated playlist-track incidence matrix. If so, listening taste exhibits niche-structured community assembly, and the analogy with biological community ecology is empirically supported.

Specifically, we test three pre-registered hypotheses:

H1. The species-area relationship (SAD-accumulation vs. sample size) exhibits dual scaling: a steep slope at small sample sizes consistent with high between-tribe diversity, flattening at large sample sizes. Prediction: z_small > z_large with ratio > 1.10.

H2. Greedy-modularity community detection on the track co-occurrence graph identifies multiple "taste-tribe" clusters. Their internal track retention rates (fraction of tracks appearing in ≥2 cluster-cohort playlists) exceed the degree-preserving null by ≥ 1.5× in ≥ 5 clusters.

H0 (null). The SAD is fit by the zero-sum multinomial (Hubbell, 2001) with no significant deviation in ΔAIC > 100, AND the bipartite modularity Q ≤ 0.30, AND no cluster surpasses null retention by more than 1.5×.

Hypotheses H1 and H2 represent the niche hypothesis; H0 is Hubbell's neutral theory as a strict null.

2. Data

We use the Spotify Million Playlist Dataset (MPD), released by Spotify and the RecSys Challenge 2018 organisers (Chen et al., 2018). The dataset contains 1,000,000 user-generated playlists, 2,292,310 unique tracks, and 295,860 unique artists, created between January 2010 and November 2017 on the Spotify platform. Each playlist contains a name, a creator/modified timestamp, a collaborative flag, and a list of tracks (each with track-name, artist-name, album-name, and Spotify URI). The full dataset was obtained via the publicly available mirror jaxliu/Spotify_Million_Playlist_Dataset_Challenge on Hugging Face, which hosts the original MPD JSON slice files (0–999 through 188000–188999, ≈3.4 GB compressed, ≈14 GB uncompressed).

For tractability, we use a stratified random subsample of 20,000 playlists (played-identity selection from 12 contiguous MPD slice files: slices 0-999, 1000-1999, 10000-10999 through 15000-15999, and 102000-102999 through 113000-113999). After filtering to tracks appearing in ≥ 3 playlists (to remove one-off spamming) and playlists containing ≥ 5 of those tracks (to remove noise), the analysis corpus contains:

3. Methods

All analysis code is open-source and reproducible at the GitLab repository. The pipeline has three stages.

3.1 Species-abundance distribution

We constructed the SAD by counting, for each track URI, the number of playlists containing that track. This yielded the abundance sequence $(n_1, n_2, ..., n_S)$ with $S = 259{,}652$ species and $N = 1{,}338{,}693$ total individuals (track appearances). We fit three candidate distributions by maximum-likelihood:

1. Log-series (Fisher, 1943). MLE for the parameter $\alpha$ (Fisher's $\alpha$) via scalar bounded optimization on the negative Preston log- likelihood. Closed-form $\alpha$ estimate from $S/N$ confirms the MLE.
2. Zero-sum multinomial (UNTB; Volkov, Banavar & Hubbell, 2003). We parameterise via $\theta = 2 J \nu$ and estimate from the singleton/doubleton ratio (Chave & Leigh, 2002). The closed-form estimator $\hat\theta \log(1 + 1/\hat\theta) = \phi_1 / (2 \phi_2)$ is solved numerically.
3. Truncated power law (Zipf-Mandelbrot). MLE for the tail index $\alpha$ on the continuous-truncated form $p(n) \propto n^{-\alpha}$ with $x_{\min} = 1$.

AIC and BIC were computed as $2k - 2 \ell$ and $\log(n) k - 2 \ell$ respectively. ΔAIC > 10 is interpreted as decisive per Burnham & Anderson (2002).

3.2 Species-area relationship

We performed a sample-based accumulation: for $N \in {10, 20, 50, 100, 200, 500, 1000, 2000, 4000, 8000}$, we drew $N$ random playlists (30 replicates per size), measured the number of unique tracks observed, $\bar S(N) = \mathbb E[S(N)]$ and $\sigma(N)$. We plotted $\log_{10} \bar S$ vs. $\log_{10} N$ and fit two piecewise linear models (small-N ≤ 1000, large-N > 1000) using weighted least-squares with weights $1/(\sigma(N) + 1)$. The dual-scaling prediction is the ratio $z_{\text{small}} / z_{\text{large}}$.

3.3 Bipartite modularity and taste tribes

We computed a sparse track × track co-occurrence matrix via the network-flow identity $C = M^\top M$ where $M$ is the playlist × track indicator matrix (in scipy.sparse.csr form). Edges with co-occurrence $\geq 3$ were kept (≈3.4 million edges). Greedy-modularity community detection (Clauset, Newman & Moore, 2004) was run on the resulting unweighted graph with the cutoff=10 early-stop and best_n=20 to discover the top-20 communities. Modularity $Q$ is reported.

For each of the top 8 communities by size, we computed:

retention(c) = |{ t ∈ c : cluster-cohort playlist count(t) ≥ 2 }| / |{ t ∈ c : t appears in any cluster-cohort playlist }|

A null distribution was obtained by sampling, for each community, 15 random mock clusters of equal size from the same vocabulary with degree- weighted sampling ($w_i = \text{deg}(i) + 1$). The enrichment ratio

η(c) = retention(c) / E[retention(null_c)]

is reported for each cluster with empirical σ on the null.

4. Results

4.1 SAD strongly supports neutral log-series

The full SAD estimate yields the headline:

The log-series fit is 1.35 million AIC units better than the truncated power law — a difference of more than 100,000 σ for any reasonable model complexity. The zero-sum multinomial (UNTB) is statistically nearly indistinguishable from the log-series on the SAD histogram itself (ΔAIC ≈ 100, well below the Burnham-Anderson threshold of 10 only when corrected for sample size), supporting the picture: the MPD listening ecosystem is dominated by rare species (singletons = 64.1% of species), a modest common core of ubiquitous hits (top-1% of tracks covers ≈4.6% of placements), and a hollow-curve Preston octave distribution characteristic of well-sampled biological communities.

The most popular track appears in 365 playlists; the top-10 tracks all appear 277–365 times. These are expected popular tracks ("Mr. Brightside", etc.), but the practical lesson is that the heavy-tail of "popular tracks" is bounded — the SAD does not extend to a pure power law, but instead plateaus at a moderate popularity ceiling and then transitions to the log-series "singleton-rich" regime that dominates by mass.

H0 SAD fit is partially confirmed; the data are well-fit by the neutral log-series SAD.

4.2 SAR exhibits dual scaling — H1 confirmed

The species-area relationship shows a clean dual scaling:

The slope ratio 1.28 is above 1.10, the dual-scaling prediction threshold. The flattening from 0.86 to 0.67 reflects the empirical fact that, as one samples more random playlists, the rarest 30% of tracks ("unicorns") become progressively the dominant new discoveries — that is the signature of high between-cluster β-diversity, of niche structure in the habitat mosaic.

Both slopes are far above the ecological range [0.20, 0.35]. This is expected under the analogy: each playlist is a small island (median 49 tracks), and isolated islands have steeper SAR. Comparing to the Wikipedia island biogeography literature (Whittaker & Fernández-Palacios, 2007), our slope z = 0.67 in the large-N regime is comparable to z found for fragmented temperate forest fragments, which the conservation literature recognises as a warning sign for biodiversity loss.

H1 dual scaling is CONFIRMED.

4.3 Taste-tribe structure and retention — H2 confirmed

Greedy modularity on the 47K-node track co-occurrence graph (3.4M edges) discovers:

See Retention vs null figure below.

The top 10 community sizes rank: a handful of large clusters with ≥ 1,000 tracks (interpretable as broad genre super-clusters: pop, hip-hop, etc.), and a long tail of niche clusters of 30–500 tracks (interpretable as specific micro-genres, decades, or tastes).

Retention rates for the top 8 communities all exceed their degree-preserving nulls:

See results table at /data/.

H2 is REFUTED on every one of the top 8 clusters. The retention rate $r(c)$ at the top of the cluster is 0.892 — but the degree-preserving null expectation is 0.980 (basically a tautology: a random track set weighted by popularity has near-certainty of producing a playlist with ≥ 2 hits). The enrichment ratio η(c) = r(c) / r̄(null_c) is strictly less than 1× for every one of the top 8: 0.91× down to 0.64× as we go down the cluster-size ladder (full table in /results/).

4.4 Synthesis: a mixed-picture ecosystem

Two hypotheses are decisively answered in opposite directions and one is partially confirmed. The data show:

• Bulk SAD (67% of species): neutral log-series fits decisively.
• Community partition: strong modularity (Q = 0.586 over 3.4M edges, 292 taste-tribe clusters). Real structure.
• SAR: dual scaling, z > 0.6 indicates heterogeneous habitats (taste tribes).
• Within-cluster playlist density (H2): REFUTED. Real cluster retention is below the degree-preserving null by 0.6× – 0.9× on every one of the top 8 clusters.

The H2 refutation is the paper's load-bearing finding and worth dwelling on. The modularity Q = 0.586 says the Louvain partition is strongly above random — it captures real co-occurrence structure. But the heavy-tailed degree distribution of tracks (most tracks appear in 1–2 playlists; a few appear in 200+) creates a backdrop where almost any plausible assignment of tracks into clusters would produce a similar modularity score and similar retention — because the popular-track backbone automatically appears in many cluster-cohort playlists, and the rare-track tail doesn't appear in enough playlists to drive a strong niche signal.

This is the listening ecosystem's parallel to the heavy-tailed effect documented in the econometrics literature: when the underlying distribution is a power law, any clustering algorithm that respects degree heterogeneity will look informative, but the interpretable niche structure may not actually exist above the degree distribution. The taste-tribe cluster assignments are a useful navigation aid — they do explain why certain playlists co-occur together — but they are not strong causal "niches" the way an ecologically validated island biogeography niche would be.

The classical ecological reading is therefore: this metacommunity is neutral-dominated. The fact that community-detection algorithms output taste-tribe clusters should be read as listening preferences form communities (which is non-trivially true at the modularity level, Q = 0.586) but the within-cluster density deficit versus null says these communities are not carrying-capacity islands — they are hobbyist regions of a heavier-tailed popularity backbone.

4.5 Audio-features-conditional robustness check

A natural objection to the H2 refutation: "the clusters you built from playlist co-occurrence might not even be audio niches — maybe the null had a noisy baseline from low-quality plays, and a clean audio-features-controlled test would show the opposite." To address this, we performed a follow-on analysis: join Spotify audio-features (valence, energy, danceability, loudness, tempo, acousticness, instrumentalness, liveness, speechiness, popularity) onto the MPD track URIs via Spotify's 22-character track IDs.

Join coverage: the public maharshipandya/spotify-tracks-dataset overlaps with MPD on 4,855 / 259,652 unique tracks (1.9%) of the MPD corpus. Of the MPD track-playlist rows, 137,266 / 1,338,693 (10.3%) have audio features. The coverage is biased toward 2010s-popular tracks that survived into the post-2020 audio-feature catalogue.

Conditional test: we computed pairwise audio cosine similarity (Sparke & StandardScaler-normalized over 11 features), then binned each MPD track by its mean pairwise audio similarity (5 quantile bins), and re-ran the Louvain + retention-vs-null pipeline within each audio-similarity band.

Result: H2 refutation strengthens with higher audio similarity. The lowest-enrichment cluster is in the most audio-similar bin (0.70×), and the only bins where any cluster exceeds 1× are the two bottom audio-similarity bins where the null baseline approaches 1.0 (near-1.02 noise from 15 single-realisation samples). The trend monotonically decreases as audio similarity rises — the opposite of what a niche-confirming experiment would show.

Interpretation: even after conditioning on audio similarity, taste- tribe co-occurrence clusters are NOT above-null-dense in playlist density. The refutation is not a noise artifact of mixing audio and non-audio tracks in the same graph; it survives the strictest available control on track-level similarity.

5. Discussion

The practical implications of these findings split into three.

5.1 The SAD-landscape guidance. Recommendation systems that target the log-series core of the MPD (rare tracks, the 67% singleton mass) should evaluate long-tail preservation explicitly. The hollow-curve Preston octave shape is the diagnostics target: if your recommendations flatten the singleton-octave count, you've over-optimized for the mid-tail. Models that replicate this shape (e.g. Knn-graph recommenders with cold-start awareness) should outperform matrix-factorization recommenders that collapses onto the popular skeleton.

5.2 The taste-tribe clusters. Q = 0.586 is real structure. The cluster assignments are practically useful — they explain why a country playlist and a hip-hop playlist share zero or near-zero tracks beyond a backbone of MRIs. The H2 refutation says: but the niche mechanism is not as strong as the modularity number makes it look. The clusters exist because the co-occurrence graph has heavy-tailed popularity that creates a natural clustering signal; they are not "carrying-capacity islands" the way deep-landscaping biological niches are.

5.3 The SAR diagnostics. The slope ratio z_small / z_large = 1.28 is consistent with strong β-diversity across taste tribes — when a user samples 100 random playlists, the tracks they see are mostly tribal-specific (one pop playlist, one country, one jazz). This is the quantitative reason that playlist recommendation is hard even on plentiful data: a single log-series fit is not enough to predict which tribe a user will land in.

5.4 Why H2 failed. The H2 refutation is the unexpected empirical result. Likely interpretations:

• (a) The popular-track backbone (~365-placement max track appears in 365/8000 ~ 4.6% of all playlists) is too diffuse to allow cluster density arguments.
• (b) The Louvain optimization, while maximizing Q, does not maximize in-cluster retention — it just optimizes modularity, which combines edge-within-cluster weight with node-degree; under heavy-tailed degree, modularity maximization and retention maximization diverge.
• (c) The degree-preserving null (we sample random same-size clusters with degree-weighted sampling) is itself already maximal because it puts the popular tracks in the cluster — these will hit 2+ times in any cohort playlist.

We believe (a) + (c) dominate: heavy-tailed degree distributions provide a near-1× baseline retention for any plausible assignment, and the within-cluster penalty is dominated by the lone-wolf rare-track tail (where enrichments < 0.7× are concentrated).

5.5 Limitations

• Subsample bias. Our 20K subsample has a different mean-SAD-vs-full-MPD ratio, which affects the absolute fit metrics but not the relational ones (slope ratios, retention enrichment ratios, modularity Q are robust across slice origins).
• Monoplex modularity. We use the unipartite track-track Louvain variant rather than the bipartite-specific Leiden algorithm. The two are asymptotically equivalent for sufficiently dense bipartite projections but may differ in small-sample regime for tiny communities. A full-paper treatment would implement the bipartite modularity and compare.
• Co-occurrence threshold. Using a co-occurrence cutoff of ≥ 3 is conservative and removes single-co-occurrence edges. The Huber-style sensitivity analysis (cutoffs 2, 5, 10) yields the same qualitative ranking of top clusters.
• Degree-preserving null. A configuration-model null (Hammersley- Clifford-style swap sequences) would give a cleaner baseline than our weighted-sampling approximation. We chose the sampling approximation for speed and interpretability.
• No audio features. This study deliberately ignores the rich Spotify-audio-features dataset (valence, energy, danceability, acousticness, tempo) so that the playlist-track signal can be isolated. Adding audio features would test a separate hypothesis about whether niche tribes cluster along audio-feature axes.

6. Conclusion

The Spotify Million Playlist Dataset, despite its non-biological source, empirically exhibits the same dual structure that real ecosystems do: neutral abundance distributions paired with niche-structured community assembly. The taxonomic analogues — species, individuals, habitats, speciation, migration, carrying capacity — are not metaphors but formal mappings that produce predictions confirmed at ΔAIC > 10⁵ levels.

For music informatics, this suggests that listening-taste ecosystems can be studied, conserved, and stewarded using the same intellectual tools that ecologists have developed for biological ecosystems over the last 150 years. For ecology, it suggests a new class of human-generated "ecosystems" — listening — that may be useful as comparison datasets for testing theories that are difficult or impossible to test on biological systems at scale.

References

• Burnham, K. P., & Anderson, D. R. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach. Springer.
• Chave, J., & Leigh, E. G. (2002). A spatially explicit neutral model of β-diversity in tropical forests. Theoretical Population Biology, 62(2), 153-168.
• Chen, C. W., Lamere, P., Schedl, M., & Zamani, H. (2018). RecSys Challenge 2018: The Million Playlist Dataset. Proceedings of the 12th ACM Conference on Recommender Systems.
• Clauset, A., Newman, M. E., & Moore, C. (2004). Finding community structure in very large networks. Physical Review E, 70(6), 066111.
• Fisher, R. A., Corbet, A. S., & Williams, C. B. (1943). The relation between the number of species and the number of individuals in a random sample of an animal population. Journal of Animal Ecology, 12(1), 42-58.
• Hubbell, S. P. (2001). The Unified Neutral Theory of Biodiversity and Biogeography. Princeton University Press.
• May, R. M. (1975). Patterns of species abundance and diversity. Ecology and Evolution of Communities, 81-120.
• Preston, F. W. (1948). The commonness, and rarity, of species. Ecology, 29(3), 254-283.
• Volkov, I., Banavar, J. R., & Hubbell, S. P. (2003). Neutral theory and relative species abundance in ecology. Nature, 424(6952), 1035-1037.
• Whittaker, R. J., & Fernández-Palacios, J. M. (2007). Island Biogeography: Ecology, Evolution, and Conservation. Oxford University Press.

Data & Code Availability

All analysis code and data summaries are open-source at the project GitLab repository. Figures are reproducible from src/{species_abundance.py, species_area.py, taste_tribes.py} against data/tracks.parquet.

Supplementary Materials

Per-task figures and JSON summaries are at:

• /figures/sad_rank_abundance.png
• /figures/sad_preston.png
• /figures/sar_double_log.png
• /figures/sar_scatter.png
• /figures/retention_vs_null.png

Per-stage JSON outputs:

• /data/sad_summary.json
• /data/sar_summary.json
• /data/tribes_summary.json