Deconstructing the Ecosystem of Online Hate in Social Platforms & Games

I was brought into the Center for Technology and Society at ADL to help the new VP understand the scope and scale of the ecosystem of online hate through a series of diagrams. The project was intended to get the team all on the same page in terms of what we mean when we talk about this ecosystem and to help everyone understand the scope of what we are talking about.

Through several months of research and stakeholder interviews, I developed a series of diagrams and models - some eventually evolved to be released externally as an Online Ecosystem Primer for ADL constituents while others were just used as talking materials within the team. The text for the first three models is text I wrote, that was then edited by our Editorial Director and is on the ADL Primer page.

Perpetrators / Targets / Mitigators / Amplifiers

- a taxonomy of the system

Perpetrator-Target-Amplifier-Mitigator-13

The Perpetrators/Targets/Mitigators/Amplifiers diagram identifies and examines the four primary “players” within the online hate ecosystem and the context in which each acts. The Perpetrators section examines who perpetrates online hate, what types of hate are happening, and what compels people to post hateful content. The Targets section explores why and how people are targeted, as well as the outcomes of being targeted by online hate. These two sections can mirror each other: religious bigotry, for example, can be committed by those of one creed who target those of another, with targeted individuals and communities including Jews, Muslims, Christians, Mormons and other faiths. Outcomes for targets can be subtle (hurt feelings) to devastating (lost jobs, reputations and lives).

The Amplifiers and Mitigators sections explore the people, places and tactics across social spaces through which hate spreads and rhetoric is broadcast and amplified, or, helpfully, can be mitigated and blocked. What are the activities that occur in gaming, on social platforms, and in other media that spread online hate? Who is working to stop online hate? Looking at the Mitigator section, we see system actors—governments, lawmakers, media—as well as individualș—teachers, parents, celebrities and bystanders—all working in various ways to educate, intervene and develop systems for stopping online hate.

The outer ring of this first diagram lists possible reasons why someone might hold a specific ideology or participate in online hate; what might happen as a result of being targeted by online hate; the types of mitigation efforts and policies that can be used to fight online hate; and the areas within social platforms where we most often see online hate amplified.

By then breaking down the higher-level classifications into discrete categories, actions, and outcomes, we can better understand the breadth of each of these classifications and the relevant context, beginning to understand where there are overlaps between areas (e.g., the media can act as a perpetrator, mitigator and amplifier of hate) and strategize solutions based on causalities and other relationships. We also create a shared vocabulary and start to understand the nuances between the contexts.

Download a PDF of the diagram

Relationships Between the Actors

- A simplified concept model

The Relationships Between Actors diagram looks at the problem of online hate from a different lens, examining the deeper relationships between the actors, specific places and opportunities to intervene and how to make changes in the system in the form of a simple concept model.

The diagram walks us through who posts online hate (inputs), where they post and what happens when they participate in hate against their targets (outputs). What are the relationships between the parts, what paths connect them? Where are the perpetrators found, and where the mitigators? It also includes an option to follow along the path that those who fight hate can take and note where it intersects with the inputs and outputs. This helps illustrate possible entry points for those countering hate. For example, one path traces how nonprofits train law enforcement to recognize hate speech – speech that, as we’ve seen so often in recent years, can radicalize and incite offline violence and terror. This is a very high-level visualization of the landscape and is intended to be an overview to prompt discussions and deeper dives into more granular details around each area.

Download a PDF of the diagram

Corporate Value Chain

- Internet and Finance infrastructure supporting bad actors. This diagram helped inform policy colleagues from the agency when giving testimony on capitol hill about some of the issues of tech infrastructure supporting online hate.

The Corporate Value Chain diagram depicts companies that provide infrastructure to online businesses. Who makes money from online hate and by what means? How is online hate supported through site visitors and advertisers? The complexity of the number and types of infrastructure companies -- by which we mean that part of the tech world that provides services such as payment processors, application hosts, and security providers -- is often hidden from the general public. When tech companies choose to not address hate on a platform they have built and managed, it becomes important for infrastructure companies to evaluate whether to provide services or products to that company. These companies have tended to stay neutral when considering online hate content beyond what is strictly prohibited by law, but we are now beginning to see some of them act to deny their services to egregious purveyors of hate. Hopefully through awareness and pressure, more of them can be compelled to reject supporting online businesses that profit off hate.

Download a PDF of the diagram

Deconstructing Online Hate at IXDA Interaction 21

"Desconstructing Online Hate" from Interaction Design Association on Vimeo.
The talk I gave at the IXDA conference Interaction 21 about the work of researching and deconstructing the ecosystem of online hate and reflecting that research back into a series of diagrams to help constituents of ADL as well as stakeholders and legislators understand the scope of what we mean when we say the ecosystem of online hate.