Engagement is not enough!

Thinking about alternative metrics to optimize content curation in online environments

Jana Lasser     |     Graz University of Technology     |     jana.lasser@tugraz.at     |     @janalasser

WhoAmI

Physicist, computer scientist, modelling complex social systems

Focus 1: Quantifying "bad" behaviour on social media

Misinformation sharing: "Social media sharing of low quality news sources by political elites", Lasser et al. 2022, in PNAS nexus.

Misinformation sharing: "From Alternative conceptions of honesty to alternative facts in communications by U.S. politicians", Lasser et al. 2023, arXiv.

Hate speech: "Collective moderation of hate, toxicity, and extremity in online discussions", Lasser et al. 2023, arXiv.

Focus 2: Agent-based models of interventions in complex social systems

COVID-19 spread: "Assessing the impact of SARS-CoV-2 prevention measures in schools by means of agent-based simulations calibrated to cluster tracing data", Lasser et al. 2022, in Nature Communications.

Healthcare system resilience: "Stress-testing the Resilience of the Austrian Healthcare System Using Agent-Based Simulation", Lasser et al. 2022, in Nature Communications.

Social media platforms as information architectures

Social media platforms disseminate information from all domains of life: entertainment, educational, health, news, political, commercial, art, ...

Example Twitter: ~500 million tweets are created and ~150 billion tweets are sent to the devices of ~42 million users every day.

For every user, from a pool of ~500 million tweets about 3600 tweets (0.007%) are selected .

Traffic rules: How is the content selected? How is the content ranked?

Twitter's timeline creation pipeline

How is content selected?

Source: Twitter's Recommendation Algorithm, Twitter, March 31, 2023.

Twitter's timeline creation pipeline

Social: people a user follows

Source: Twitter's Recommendation Algorithm, Twitter, March 31, 2023.

Twitter's timeline creation pipeline

Engagement: reactions of other users to content

Source: Twitter's Recommendation Algorithm, Twitter, March 31, 2023.

Twitter's timeline creation pipeline

Similarity: content and users that are similar to the user

Source: Twitter's Recommendation Algorithm, Twitter, March 31, 2023.

Twitter's timeline creation pipeline

Final ranking: probability the user will engage with the content

Source: Twitter's Recommendation Algorithm, Twitter, March 31, 2023.

Twitter's timeline creation pipeline

Filters: "author diversity" and "content balance"

Source: Twitter's Recommendation Algorithm, Twitter, March 31, 2023.

Engagement might not be enough

The business model of large social media platforms is based on capturing user attention for advertising revenue.

This might be a good metric to optimize user entertainment. What about education, health, news, politics, commerce, art, ...?

There is mounting evidence that social media platforms threaten social cohesion [1]. Optimizing for engagement might be part of the problem [2].

[1] "A systematic review of worldwide causal and correlational evidence on digital media and democracy", Lorenz-Spreen et al. 2023, Nature Human Behaviour.

[2] "Facebook Struggles to Balance Civility and Growth", Roose et al. 2021, New York Times.

Beyond engagement

How can we replace or supplement engagement as metric for content delivery?

See also: Building Human Values into Recommender Systems: An Interdisciplinary Synthesis, Stray et al., 2022, arXiv.

Beyond engagement

How can we replace or supplement engagement as metric for content delivery?

See also: Building Human Values into Recommender Systems: An Interdisciplinary Synthesis, Stray et al., 2022, arXiv.

Beyond engagement

How can we replace or supplement engagement as metric for content delivery?

See also: Building Human Values into Recommender Systems: An Interdisciplinary Synthesis, Stray et al., 2022, arXiv.

Beyond engagement

How can we replace or supplement engagement as metric for content delivery?

See also: Building Human Values into Recommender Systems: An Interdisciplinary Synthesis, Stray et al., 2022, arXiv.

Challenges

Value trade-offs
Who decides which values count and how much?

How do short-term metrics relate to long-term outcomes?
Social media platforms are complex systems. Behaviour is hard to predict.

Many long-term outcomes are not measureable from user behaviour
Measuring these metrics requires surveys. Integration of survey signals & long-term outcomes into recommender systems is an open challenge.