Causally estimating the effect of YouTube's recommender system using counterfactual bots

Homa Hosseinmardi^{1

2}, Amir Ghasemian³, Miguel Rivera-Lanas⁴, Manoel Horta Ribeiro⁵, Robert West⁵, Duncan J Watts^{1

2

6}

Affiliations

¹ Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104.
² Annenberg School of Communication, University of Pennsylvania, Philadelphia, PA 19104.
³ Yale Institute for Network Science, Yale University, New Haven, CT 06511.
⁴ Heinz College of Information Systems and Public Policy, Carnegie Mellon University, Pittsburgh, PA 15213.
⁵ School of Computer and Communication Sciences, École Polytechnique Fédérale de Lausanne, 1015 Ecublens, Switzerland.
⁶ Operations, Information, and Decisions Department, University of Pennsylvania, Philadelphia, PA 19104.

PMID: 38349876
PMCID: PMC10895271
DOI: 10.1073/pnas.2313377121

Causally estimating the effect of YouTube's recommender system using counterfactual bots

Homa Hosseinmardi et al. Proc Natl Acad Sci U S A. 2024.

. 2024 Feb 20;121(8):e2313377121.

doi: 10.1073/pnas.2313377121. Epub 2024 Feb 13.

Authors

Homa Hosseinmardi^{1

2}, Amir Ghasemian³, Miguel Rivera-Lanas⁴, Manoel Horta Ribeiro⁵, Robert West⁵, Duncan J Watts^{1

2

6}

Affiliations

¹ Department of Computer and Information Science, University of Pennsylvania, Philadelphia, PA 19104.
² Annenberg School of Communication, University of Pennsylvania, Philadelphia, PA 19104.
³ Yale Institute for Network Science, Yale University, New Haven, CT 06511.
⁴ Heinz College of Information Systems and Public Policy, Carnegie Mellon University, Pittsburgh, PA 15213.
⁵ School of Computer and Communication Sciences, École Polytechnique Fédérale de Lausanne, 1015 Ecublens, Switzerland.
⁶ Operations, Information, and Decisions Department, University of Pennsylvania, Philadelphia, PA 19104.

PMID: 38349876
PMCID: PMC10895271
DOI: 10.1073/pnas.2313377121

Abstract

In recent years, critics of online platforms have raised concerns about the ability of recommendation algorithms to amplify problematic content, with potentially radicalizing consequences. However, attempts to evaluate the effect of recommenders have suffered from a lack of appropriate counterfactuals-what a user would have viewed in the absence of algorithmic recommendations-and hence cannot disentangle the effects of the algorithm from a user's intentions. Here we propose a method that we call "counterfactual bots" to causally estimate the role of algorithmic recommendations on the consumption of highly partisan content on YouTube. By comparing bots that replicate real users' consumption patterns with "counterfactual" bots that follow rule-based trajectories, we show that, on average, relying exclusively on the YouTube recommender results in less partisan consumption, where the effect is most pronounced for heavy partisan consumers. Following a similar method, we also show that if partisan consumers switch to moderate content, YouTube's sidebar recommender "forgets" their partisan preference within roughly 30 videos regardless of their prior history, while homepage recommendations shift more gradually toward moderate content. Overall, our findings indicate that, at least since the algorithm changes that YouTube implemented in 2019, individual consumption patterns mostly reflect individual preferences, where algorithmic recommendations play, if anything, a moderating role.

Keywords: algorithmic audits; experiment design; online extremism; recommender systems.

PubMed Disclaimer

Conflict of interest statement

Competing interests statement:The authors declare no competing interest.

Figures

**Fig. 1.**
Overview of the counterfactual bot method to disentangle YouTube’s recommender system from user preferences utilizing counterfactual bots. Each panel shows the trajectories (one per row) that bots traverse within the corresponding experiment. By measuring the difference in the partisanship of watched videos by control bot ( $y^{cont .}$ ) and watched videos by algorithmic counterfactual bots ( $y^{alg .}$ ), our design eliminates the “preference” or “choice” component ( ${\hat{y}}^{pref .} = y^{cont .} - y^{alg .}$ ) of observed consumption, allowing us to estimate the causal effect of algorithmic recommendations. (A) Estimating bias of the recommender: Four bots watch the same history in the learning phase, whereas in the observation phase, the control bot continues to follow the real user’s historical trajectory and the “counterfactual” bots follow simple algorithmic rules: “up next” (choosing the top-ranked video from the sidebar), “random sidebar” (choosing a random video from the sidebar), and “random home” (choosing a random video from the homepage). (B) Estimating “forgetting time” of the recommender: Two bots start at the same time, watching the same trajectory in the learning period. The control bot will continue watching from the same trajectory in the observation phase, while the counterfactual bot will switch to watching videos of moderate content. To estimate the effects of different-length histories, half the bots have “short” (30 video) histories prior to switching (top two rows), while the other half have “long” (120 video) histories (bottom two rows).

**Fig. 2.**
Examples of traversed trajectories for four focal users with different mixtures of (A) center ( $ψ^{C}$ ) and (B–D) far-right ( $ψ^{fR}$ ) consumption in the counterfactual experiment, Fig. 1A. The first half is the learning phase (all four bots watch the same videos at each step) and the second half (shaded gray area) is the observation phase (each of the four bots follows a separate rule). The y-axis shows the partisanship of watched videos at each step. The dashed line shows zero partisanship. Solid lines show the average partisan score of all 60 watched videos in the observation phase for each path.

**Fig. 3.**
Partisan score of the 60 watched videos for the control and counterfactual bots during the observation phase for focal users with different mixtures of (A) center ( $ψ^{C}$ ) and (B–D) far-right ( $ψ^{fR}$ ) video consumption. Each box-plot shows the median, interquartile range, and full range of the average partisanship (the y-axis range is limited to [−0.24,0.24] for better visualization). The dashed line shows zero partisanship, and the dotted lines represent one and two standard deviations away from the mean (zero) of partisan scores (see *SI Appendix* for details of the sample of videos and partisan scoring method).

**Fig. 4.**
Marginal effect of bursty viewership of partisan videos (calculated using ggeffects R-package) on the user preference role in future consumption. Preference increases with higher bursts of partisan consumption.

**Fig. 5.**
Forgetting time: A comparison of the average partisan score and the fraction of recommended fR videos (insets) for control (red line) and counterfactual (green line) bots for sidebar (A) and homepage (B) recommendations respectively. The control bot watches 120 videos from a fR focal user, while the counterfactual bot after watching the same 30-video history as in the control bot, transitions to videos from a *center* focal user spanning 90 videos. (A) Sidebar response to this change in consumption is immediate and partisan score converge to zero. (B) For homepage, the average partisan score converges to moderate range; however, even after 90 post-switch videos the average fraction of fR videos remains nonzero (albeit much lower than for the control). For better visualization, the y-axis range across all panels is the same.

**Fig. 6.**
Effect of history length: A comparison of the average partisan score and the fraction of recommended *far-right* videos (insets) for counterfactual bots only with short (green) and long (purple) histories of fR viewership for sidebar (A) and homepage (B) respectively. In the control arm, a bot watches a 30-video fR history followed by a 120-video *center* history, while in the treatment arm, the bot is exposed to an additional fR history lasting 120 videos. (A) On the sidebar, both longer and shorter history exhibit the same drop rate in terms of average partisan score and average fraction of fR content. (B) On the homepage longer history reduces the drop rate of partisanship on both metrics. Even after 90 steps, the average partisan score of bots with longer history remains higher than that of the shorter path. For better visualization, the y-axis range across all panels is the same.

See this image and copyright information in PMC

References

1. A. Schomer, “US YouTube advertising 2020” eMarketer (2020). https://www.emarketer.com/content/us-youtube-advertising-2020. Accessed 27 December 2021.
1. T. Konitzer et al., Measuring news consumption with behavioral versus survey data. Pew Research. https://www.pewresearch.org/journalism/2020/12/08/measuring-news-consump.... Accessed 27 December 2023.
1. M. Iqbal, Twitter revenue and usage statistics. Business of Apps. https://www.businessofapps.com/data/twitter-statistics/. Accessed 27 December 2023.
1. Hosseinmardi H., et al. , Examining the consumption of radical content on YouTube. Proc. Natl. Acad. Sci. U.S.A. 118, e2101967118 (2021). - PMC - PubMed
1. M. A. Brown et al., Echo chambers, rabbit holes, and algorithmic bias: How YouTube recommends content to real users. SSRN. https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4114905. Accessed 27 December 2023.

LinkOut - more resources

Full Text Sources
- Atypon
- PubMed Central
Miscellaneous
- NCI CPTAC Assay Portal

Save citation to file

Email citation

Add to Collections

Add to My Bibliography

Your saved search

Create a file for external citation management software

Your RSS Feed

Causally estimating the effect of YouTube's recommender system using counterfactual bots

Affiliations

Causally estimating the effect of YouTube's recommender system using counterfactual bots

Authors

Affiliations

Abstract

Conflict of interest statement

Figures

References

LinkOut - more resources

Full Text Sources

Miscellaneous