How does Spotify know what you like? Expert sheds light on 'recommender' systems at U of T event

Streaming giant Spotify’s popularity is due in part to its uncanny ability to sift through a library of more than 82 million tracks to match the tastes of each of its 500 million monthly active users.

How does it do it? The answer lies in the platform’s recommender machine learning algorithms, which develop custom content individual users based on their past behaviour. 

Mounia Lalmas, senior director of research at Spotify, shed light on some of the techniques used by the world’s largest music streaming platform at a recent event hosted by the University of Toronto’s Schwartz Reisman Institute for Technology and Society (SRI).

She emphasized the importance of “creating personalized listening experiences” and delved into how Spotify processes data to match users and tracks, providing insight into the types of data used, which includes not only the taste preferences of users, but also their interactions with the app, including time of day and history.

“Really, what is important for Spotify are two sides: the artists and creators, and the listeners and users,” said Lalmas, who leads a team of interdisciplinary researchers at Spotify who are working on personalization. “We focus on both, and how we view things is really a symbiotic relationship where each side matters equally.”

Rather than seeking to predict the next click, Lalmas – who holds an honorary professorship at University College London and is a distinguished research fellow at the University of Amsterdam – said the goal is to develop algorithms that guide users’ long-term journeys, and that this necessarily shifts what factors determine a “good recommendation.” She rounded out her presentation by sharing recent methods her team has developed, including how the platform is supporting users to explore diverse content and helping to connect users with new artists.

The event, held at the Rotman School of Management, also featured SRI Research Lead Ashton Anderson, an assistant professor in U of T’s department of computer science in the Faculty of Arts & Science. Anderson’s research in the field of computational social science led him to Spotify in 2018 – and a collaboration with Lalmas – to explore how algorithms impact user journeys.

Creating personalized listener experiences

At the heart of Spotify’s listening experience lies a sophisticated recommendation engine composed of three layers: data, models, and experience. 

Beginning with data, Lalmas highlighted the role of user playlists and listening history, as well as instrumentation – the actions taken on the platform – in shaping recommendations. For instance, a user who spends a lot of time scrolling is likely seeking out new content rather than something they are already familiar with, she said, adding that Spotify uses metadata provided by the music label, as well as the audio profile of the song to further factor into its recommendations.

Next, user and track data are mapped onto an embedding space that enables the platform to assess relationships between content. In natural language processing, embeddings enable systems to grasp the relative proximity between words – for instance, the word “dog” would be closer to “cat” than to “tree.” Every track, artist and user can be represented in this space, which serves as a backbone for the platform’s models, while proximity within these spaces determines similarity and cohesion.

The last layer is experience, which consists of the app that Spotify users interact with. Recommendation engines power tailored playlists in this layer, including “Discover Weekly,” and determine how content is surfaced on a user’s homepage. This layer provides crucial feedback (for example, a user who skips a suggested song), which helps the engine to continuously refine its recommendations, Lalmas said.

Balancing wants and needs

Lalmas said that recommendation systems should deliver a lifetime of content, with the goal of building trust. To do this, algorithms must be fine-tuned to strike a balance between satisfying users’ wants and needs.

Imagine, for example, that you opened a newly recommended playlist and you skipped the first five songs because they weren’t your style. Feeling frustrated, you would probably close the playlist. On the other hand, if the app continually suggests only familiar songs, you’d soon become bored. The key to engage listeners, Lalmas said, is striking a delicate balance between the two.

Diversifying listening experiences

Lalmas wrapped up her talk by sharing some of the progress her team has made towards understanding how to diversify listening experiences, including a survey of recent research publications.

In a paper presented at the 2022 ACM Conference on Web Search and Data Mining, her team proposed and evaluated a machine learning algorithm that models a user’s recent sequential listening patterns (which change quickly) and their all-time listening profile (which changes slowly). The result is a more accurate user profile developed over time, improving recommendations.

In a collaboration with Anderson, Lalmas and team modelled the evolution of user preferences over time. Their findings showed that changes in musical preferences were directional, meaning that those who listened to hard rock may eventually develop an interest in blues rock, but those who started with blues rock would not shift to hard rock. By identifying these preferential pathways, it can become possible to introduce users to new music different enough to diversify their listening – but not different enough for engagement to drop.

Another collaborative project between Anderson and Lalmas analyzed the long-term consequences of Spotify recommendations on users’ listening habits. They labelled users as “generalists” or “specialists” based on the diversity of songs they listen to, and found that recommender systems can be fine-tuned to encourage diverse listening without affecting engagement by steering consumption towards less popular content. This study highlights a common challenge with recommender systems: the tendency to promote the most popular items, which can lead to “rich-get-richer” scenarios.

The influence of algorithmic recommendations on consumption has motivated researchers to investigate their technical design and governance. Meanwhile, finding new ways to align recommender systems with social values has emerged as an important area of research.

SRI Director and Chair Gillian Hadfield is currently engaged in a multi-stakeholder project exploring how to improve user satisfaction with Facebook’s recommender systems, which proposes that partnerships between academia and Big Tech are vital to shed light on the social impacts of algorithms. Recommender systems were also the focus of a session at SRI’s 2022 Absolutely Interdisciplinary conference, which discussed the importance of policy to shape the design of algorithms in the context of democratic rights.

Lalmas noted that diversity is key when creating personalized experiences, and that “a diverse and dynamic diet” generates long-term benefits for users and platforms alike.