So Powerful?

Russian interference in recent U.S. elections and online radicalization by proponents of violent extremism are just two recent, large-scale examples of the content problems that we reference above.

Following the 2016 election, it was revealed that Russian government actors had attempted to in u ence U.S. election results by promoting false online content (posts and ads alike) and using online messaging to stir tensions between di erent voter factions. These in uence e orts, alongside robust disinformation campaigns run by domestic actors, were able to our ish and reach millions of voters (and perhaps in uence their choices) in an online environment where content-shaping and ad-targeting algorithms play the role of human editors. Indeed, it was not the mere existence of misleading content that interfered with people’s understanding of what was true about each candidate and their positions—it was the reach of these messages, enabled by algorithms that selectively targeted the voters whom they were most likely to in uence, in the platforms’ estimation. 19 20

During the 2016 U.S. presidential election, disinformation circulated widely online. The screenshot above is from a fake story shared on Twitter about then-candidate Hillary Clinton (Twitter, Nov. 21, 2016).

The revelations set in motion a frenzy of content restriction e or ts by major tech companies and fact-checking initiatives by companies and independent groups alike, alongside Congressional investigations into the issue. Policymakers soon demanded that companies rein in online messages from abroad meant to skew a voter’s perspective on a candidate or issue. 21

How would companies achieve this? With more algorithms, they said. Alongside changes in policies regarding disinformation, companies soon adjusted their content moderation algorithms to better identify and weed out harmful election-related posts and ads. But the scale of the problem all but forced them to stick to technical solutions, with the same limitations as those that caused these messages to our ish in the rst place. 22

A screenshot of what YouTube users see when they try to access a video that has been removed by the company for violating its content policy (YouTube, Feb. 26, 2020).

Content moderation algorithms are notoriously di cu lt to implement e ecti vely, and often create new problems. Academic studies published in 2019 found that algorithms trained to identify hate speech for removal were more likely to ag s ocial media content created by African Americans, including posts using slang to discuss contentious events and personal experiences related to racism in America. While companies are free to set their own rules and take down any content that breaks those rules, these kinds of removals are in tension with U.S. free speech values, and have elicited the public blowback to match. 23

Content restriction measures have also in ict ed collateral damage on unsuspecting users outside the United States with no discernible connection to disinformation campaigns. As companies raced to reduce foreign interference on their platforms, social network analyst Lawrence Alexander identi ed several users on Twitter and Reddit whose accounts were suspended simply because they happened to share some of the key characteristics of disinformation purveyors.

A screenshot of the message users see when a tweet has been removed by Twitter for a violation of the “Twitter Rules” (Twitter, Feb. 28, 2020).

One user had even tried to notify Twitter of a pro-Kremlin campaign, but ended up being banned himself. “[This] quick- x approach to bot-hunting seemed to have dragged a number of innocent victims into its nets,” wrote Alexander, in a research piece for Global Voices. For one user who describes himself in his Twitter pro l e as an artist and creator of online comic books, “it appears that the key ‘suspicious’ thing about their account was their location—Russia.” 24

The lack of corporate transparency regarding the full scope of disinformation and malicious behaviors on social media platforms makes it di cult to as sess how e ecti ve these e o rts actually are.

For four years, RDR has tracked whether companies publish key information about how they enforce their content rules. In 2015, none of the companies we evaluated published any data about the content they removed for violating their platform rules. Four years later, we found that Facebook, Google, Microsoft, and Twitter published at least some information about their rules enforcement, including in transparency reports. But this information still doesn’t demonstrate how e ective their content moderation mechanisms have actually 25

26 27

been in enforcing their rules, or how often acceptable content gets caught up in their nets.

How transparent are companies about terms of service enforcement (2015-2019)?

Companies have also deployed automated systems to review election-related ads, in an e or t to better enforce their policies. But these e or ts too have proven problematic. Various entities, ranging from media outlets to LGBTQ-rights groups to Bush’s Baked Beans, have reported having their ads rejected for violating election-ad policies, despite the fact that their ads had nothing to do with elections. Yet companies’ disclosures about how they police ads are even less transparent than those pertaining to user-generated content, and there’s no way to know how e ective these policing mechanisms have been in enforcing the actual rules as intended. 28 29 30 31

32

The issue of online radicalization is another area of concern for U.S. policymakers, in which both types of algorithms have been in play. Videos and social media channels enticing young people to join violent extremist groups or to commit violent acts became remarkably easy to stumble upon online, due in part to what seems to be these groups’ savvy understanding of how to make algorithms that amplify and spread content work to their advantage. Online extremism and radicalization are very real problems that the internet has exacerbated. But e or ts to address this problem have led to unjusti ed censorship. 33

Widespread concern that international terrorist organizations were recruiting new members online has led to the creation of various voluntary initiatives, including the Global Internet Forum to Counter Terrorism (GIFCT), which helps

companies jointly assess content that has been identi ed as promoting or celebrating terrorism.

The GIFCT has built an industry-wide database of digital ngerprints or “hashes” for such content. Companies use these hashes to lter o ending content, often prior to upload. As a result, any errors made in labeling content as terrorist in this database are replicated on all participating platforms, leading to the censorship of photos and videos containing speech that should be protected under international human rights law. 34

Thousands of videos and photos from the Syrian civil war have disappeared in the course of these e orts—videos that one day could be used as evidence against perpetrators of violence. No one knows for sure whether these videos were removed because they matched a hash in the GIFCT database, because they were agged by a content moderation algorithm or human reviewer, or some other reason. The point is that this evidence is often impossible to replace. But little has been done to change the way this type of content is handled, despite its enormous potential evidentiary value. 35

In an ideal world, violent extremist messages would not reach anyone. But the public safety risks that these carry rise dramatically when such messages reach tens of thousands, or even millions, of people.

The same logic applies to disinformation targeted at voters. Scale matters—the societal impact of a single message or video rises exponentially when a powerful algorithm is driving its distribution. Yet the solutions for these problems that we have seen companies, governments, and other stakeholders put forth focus primarily on eliminating content itself, rather than altering the algorithmic engines that drive its distribution.