Fear Speech Shows the Limits of Moderation

Recent research suggests that "Fear Speech" is often more common and problematic than hate speech, yet is difficult to address with content moderation.

Today we are featuring a guest post by Kiran Garimella, a professor of library and information science at Rutgers whose lab uses large-scale data to tackle societal issues such as misinformation, political polarization, and hate speech.  We are featuring his lab’s work because we have been heavily citing it in a wide array of venues. 

For example, we cite it in a paper we are presenting today at Columbia University’s  Knight First Amendment Institute on “The Algorithmic Management of Polarization and Violence on Social Media”, in collaboration with Jonathan Stray and Helene Puig Larrauri.  The paper argues that the design of social media often facilitates more destructive conflict, and then suggests design solutions rather than relying on content moderation, which has limited effectiveness in addressing the “fear speech” that Kiran’s lab studies.

It’s an argument that we’ve made in several venues recently, and the feedback we’ve gotten in each setting has helped sharpen our argument and build allies along the way.  We are now in the process of strategizing with several other organizations about how we can inform the many state level design codes that are in the works.  In February, we presented the idea of “regulating design, not speech” as a solution to societal cohesion issues at the Tech + Social Cohesion conference and followed that up with a March appearance on the Lawfare podcast. In late March, we convened a panel at a Yale Law school conference where the head of Adversarial Planning from Niantic Labs, an ex-employee of Facebook/Twitter, and the lead of the science board at the Prosocial Design Network discussed specific design solutions.  Later that week, at the University of Michigan, we applied the same paradigm to international contexts, where content moderation approaches are even more challenged by the situational factors that are not accounted for in objective global policies.  

In all of these venues, we referenced the below work on “Fear Speech”, which echoed my experience at Facebook.  We reference this work because fear speech is not inherently bad and information about what to be afraid of often keeps us safe, such that content moderation policies will often fail to address such content. However, as their recently published paper in PNAS referenced below shows, such content is often more common and engaged with, as compared with concepts like “hate speech”, which content moderation focuses on.  Design based solutions that remove the incentive to use fear to gain attention may be more appropriate interventions than penalizing expressions of fear, which can be genuine.  Below is the guest essay from Kiran and his team.

— 

On the Rise of Fear Speech on Social Media

By Kiran Garimella

In recent years, most mainstream social media platforms have been heavily moderated to prevent the spread of online hate speech, which is usually fertile in toxic words and is directed towards an individual or a community. However, our research [1,2] finds the rise of newer and more subtle techniques which are being deployed in a concerted effort, which attempt to incite fear about a target community. We call it fear speech, based on literature from conflict studies [3]. We studied the prevalence of fear speech in two different contexts: right wing extremism on WhatsApp in India and on the far right social media site Gab.com in the US. In both contexts, we found that fear speech is being used more and more in a planned manner, and it is highly effective. In this post, I explain why being conscious of this new type of speech is important and what we could do about this.

Take a look at this example: 

“Hundreds of South Americans are marching through Mexico, aiming to cross the US Border and demand asylum in the US. No one in Mexico is stopping them. This is a national security threat and should be dealt with by force if necessary. What else is our military good for if they can’t stop an invading force?”

Note that this message has no toxic words and is weaved into a series of arguments citing evidence, establishing a case of nationwide fear and finally inciting users to take an action. Such views often resonate with the opinions of the “common” audience, and they, in turn, contribute to spreading the message deeper and farther into the network. Our findings on large datasets spanning hundreds of thousands of messages on social media indicate that unlike hate speech, fear speech has very little explicitly toxic content, which makes it look more plausible. Hate speech posters often hurl direct multi-target insults, while fear speech posts more subtly portray a community as a perpetrator using a (fake) chain of argumentation, thus pointing to why general users could be more susceptible to fear speech. The prevalence of fear speech is not just limited to Gab.com; our findings transcend even to other platforms such as Twitter and Facebook.

We also found that users posting a large amount of fear speech accrue more followers and occupy more central positions in the social network compared to users posting a large amount of hate speech. Fear speech posters are able to reach out to benign users (users that might not otherwise post hate/fear messages) far more effectively than hate speech posters through replies, re-posts, and mentions. These findings suggest that fear speech could be more harmful because it normalizes hatred towards other communities. The study points towards the need for more sophisticated moderation policies and mass awareness to combat fear speech.

Graph comparing posters of fear speech vs. hate speech from PNAS paper

Fear speech can be harmful.  Prior research shows that existential fear can bias peaceful people towards extremism or offline action. In a controlled experiment [4], a group of Iranian students were found to support doctrines related to the value of human life as opposed to a jihadist call for suicide bombing. However, when they were frightened about death, they were more likely to support the bomber, even potentially expressing a desire to become a martyr themselves. From time to time, mortality salience polarizes an individual or a group to stick firmly to their own beliefs while demonizing others with opposing beliefs. In real life, elements of fear are often found associated with events of violence. The posts of the alleged attacker who shot worshippers at the Pittsburgh synagogue in October 2018, portrayed the Hebrew Immigrant Aid Society as an organization supporting refugee invasion. Similarly, the shooter of the Christchurch event in 2019 released a manifesto, 'Great Replacement,' containing elements of fear in the form of non-whites replacing whites in the future. Such association of online speech and offline action is also well grounded in the literature of intergroup conflicts [3]. 

Our data indicates widespread, mostly false fear speech about violence by the Muslim community (10% of the fear speech posts), Jews controlling media and culture (10% of the fear speech posts), and white genocide in South Africa (7% of the fear speech posts). Fear speech can be dangerous, and social media provides an opportunity for such fear to be channeled and amplified. Therefore, there should be discussion and thought on how to handle it.

Social media companies use guidelines to appoint manual and automatic moderators to delete hateful posts/suspend hateful users. The research community has started putting consolidated efforts to automate and scale up this moderation creating better datasets and machine learning models to accurately detect hate speech. While these advances are indeed encouraging, newer and more subtle forms of harmful content are inflicting the online world which most often go unnoticed. Fear speech is one such form of malicious content that involves spreading fear about one or more target communities in the online and eventually, the physical world.

Our findings necessitate using sophisticated moderation policies and mass awareness to combat fear speech. This means social media companies need to take a step further to prevent the spread of fear speech by educating the users on its harmful effects. The research community needs to come up with more advanced techniques to detect fear speech, which is often subtle and goes unnoticed. It is our responsibility as individuals to use social media in a responsible way and raise awareness of the prevalence of fear speech.

Moderating such speech may fall in a gray area due to free speech concerns. However, since there is prior evidence that shows how coordinated fear can be dangerous, and social media provides an opportunity for such fear to be channeled and amplified, there should be discussion and thought on how this is handled. We also need to keep in mind the implications of such speech not just from a WEIRD perspective [5] but in fragile societies like India and Myanmar, where there is evidence of massive offline consequences such as riots and communal violence.

While our study provides important insights into the prevalence and characteristics of fear speech on social media platforms, there are some limitations to our methodology and findings that should be acknowledged.

First, our PNAS study focuses on a single platform, Gab.com, which is known for its extremist content and may not be representative of other social media platforms. Therefore, our findings may not generalize to other platforms with different user bases and moderation policies. We do try to use the models built on Gab data to predict the prevalence of fear speech on other, mainstream and moderated platforms like Twitter and find that there are many posts.  We also have previous work examining WhatsApp that complements these findings.

Second, our study only analyzes publicly available posts on Gab.com and may not capture all instances of fear speech or hate speech on the platform. Private messages and posts that are deleted or removed by moderators would not be included in our analysis, which could lead to an underestimation of the prevalence of these types of harmful content. Though this might be mitigated by the lax moderation policies on Gab.

Third, while we used a machine learning approach to identify fear speech and hate speech posts, these types of harmful content are often nuanced and context-dependent, and may not be accurately captured by automated methods. Therefore, some posts may have been misclassified, which could impact our findings. We perform thorough analysis of our models and release them as open source for further development.

Finally, our study is limited by the fact that we only analyzed the prevalence and characteristics of fear speech and hate speech posts, and did not investigate the impact of these types of harmful content on individuals or communities. Further research is needed to understand how exposure to fear speech and hate speech on social media platforms affects individuals' attitudes, beliefs, and behaviors.

References:

[1] Saha, P., Garimella, K., Kalyan, N. K., Pandey, S. K., Meher, P. M., Mathew, B., & Mukherjee, A. (2023). On the rise of fear speech in online social media. Proceedings of the National Academy of Sciences, 120(11), e2212270120.

[2] Saha, P., Mathew, B., Garimella, K., & Mukherjee, A. (2021, April). “Short is the Road that Leads from Fear to Hate”: Fear Speech in Indian WhatsApp Groups. In Proceedings of the Web Conference 2021 (pp. 1110-1121).

[3] Buyse, A. (2014). Words of violence:" Fear speech," or how violent conflict escalation relates to the freedom of expression. Hum. Rts. Q., 36, 779.

[4] Pyszczynski, T., Abdollahi, A., Solomon, S., Greenberg, J., Cohen, F., & Weise, D. (2006). Mortality salience, martyrdom, and military might: The great Satan versus the axis of evil. Personality and social psychology bulletin, 32(4), 525-537.

[5] ​​Henrich, J., Heine, S. J., & Norenzayan, A. (2010). Most people are not WEIRD. Nature, 466(7302), 29-29.

—-

Below are a few announcements from our Psychology of Technology Institute community:

• The ADL is looking for fellows with an application deadline of May 15 for full consideration.

• Matt Katsaros and Julia Kamin, who work on improving technology’s impact on society at the Justice Collaboratory and the Prosocial Design Network, recently shared this paper on A Framework for Digital Interventions for Online Prosocial Behavior on PsyArxiv.

• The Tech Policy Press podcast recently featured a discussion of our advocacy of design over content moderation. It starts at around the 20 minute mark in this episode.

• There are still a couple days left to apply to speak at the Trust and Safety Research Conference, scheduled for September 28-29 this year at Stanford. Applications are due April 30 and the form is here.

• We were recently quoted in Article 14, a publication dedicated to keeping India’s democracy vibrant, on how the design of social media can facilitate hate influencers.

• We continue to be concerned about the potential negative effects of increased AI adoption and were quoted in this Buzzfeed News article about the dangers of using AI for mental health therapy.