Like Walking a Tightrope: User-Centric Perspectives on Content in Need of Moderation

Norm-violating content on social media platforms is on the rise, fueling prejudice and polarization. To counter such developments in a timely manner, platforms rely on automated content moderation. The issue, however, is that current understandings of automated moderation are mostly elite-driven, neglecting individual user perspectives, introducing biases and discrimination, and overlooking that users’ perspectives on what constitutes a norm violation in the first place may differ substantially from those of experts and platforms. Therefore, this study uses focus groups to explore what type of content users themselves think is in need of moderation. Findings suggest that users, while acknowledging the importance of moderating polarized topics (e.g., migration) and uncivil discourse styles (e.g., insults), are more concerned with the motivations of perpetrators and the vulnerability of targets. Our findings show that, if platforms want to incorporate user perspectives when designing automated moderation systems, they must move beyond merely detecting problematic message characteristics and focus on more tailored approaches, alongside accountability measures for perpetrators and safety measures for victims.