How the Buffalo shooting exposes shortcomings in Discord’s policing of users

According to screenshots widely shared on social media, a username linked to the shooter posted a “to-do list” on the social network Discord in the weeks leading up to the massacre. Among the items: “continue writing manifesto,” “clean and oil AR,” and “make goodbye list.” (Fortune could not independently verify the authenticity of the screenshots.)

While the posts do not specify a target or timeline, any reasonable person viewing the list could infer that it represented an immediate red flag. Yet to this point, there’s no evidence that anyone contacted Discord or law enforcement to warn about the alleged shooter’s statements, and Discord has provided no evidence to show it knew about the comments. (In a statement to The New York Times, Discord officials said they are investigating the posts and “will do everything we can to assist law enforcement in the investigation.”)

Despite years of intense scrutiny and some well-intentioned content moderation efforts, many platforms remain a largely reactive space. Somebody breaks a rule, somebody else reports the violation, and platform officials remove content.

But as this weekend’s tragedy shows, there aren’t enough good guys with a keyboard. Either nobody saw the ominous writings, or nobody cared enough to warn the appropriate authorities.

For the world’s largest social media outfits, the moral and business costs of reactive policing have forced huge investments in human moderators and artificial intelligence that helps flag troublesome content. Smaller upstarts, however, aren’t nearly as sophisticated with their moderation protocols—either by choice or because of financial constraints. For those like Discord looking to cash in on their audience, the cost could quickly mount. 

At the most-scrutinized social media companies, aggressive moderation efforts have produced remarkably high rates of proactive content removal. For the second half of 2021, Facebook reported that 96.6% of actions taken in response to “violence or incitement content” stemmed from an internal flag, with the rest reported by users. Twitter said 93% of accounts suspended for promotion of terrorism and violent organizations were proactively identified and removed during that same time frame. (Microsoft-owned Twitch, to its credit, said it took down the alleged shooter’s live stream of his attack within two minutes.)

In contrast to its larger peers, Discord reported a proactive removal rate of 32% on servers—the platform’s version of a forum or thread—deemed to contain hateful conduct, violent extremism, or violent and graphic content in the second half of 2021. Reddit said about 20% of content removals due to violent content and 43% of removals due to hateful content were through automation in 2021.

Discord has acknowledged its shortcomings in the past on the content moderation front, particularly as it evolved from a chat hub for gamers into a multipurpose platform preferred by some far-right extremists. The San Francisco-based startup has banned white nationalist users and taken down servers encouraging violence, including a popular forum among organizers of the Jan. 6 attack on the U.S. Capitol.

But as Discord executives consider whether to take the company public following a $15 billion valuation—Bloomberg reported in March that they’re interviewing investment bankers for a potential direct listing or IPO—its content moderation practices will come under deeper scrutiny. As Facebook, Twitter, YouTube, and other huge social media outfits have discovered, advertisers and politicians don’t look kindly on Wild West platforms. That, in turn, could scare off investors.

The possibility of onerous new regulations appears unlikely, even after last weekend’s attack. Outside of a few calls for reforming Section 230, a federal law that declares companies aren’t liable for content posted on their websites, political leaders largely stuck to pleas for help rather than promises of legislation.

“This spreads like a virus, and that’s why I’m calling on the CEOs of all the social media platforms to examine their policies and to be able to look me in the eye and tell me that everything is being done that they can do to make sure this information is not spread,” New York Gov. Kathy Hochul, a Democrat, told CNN on Sunday.

While those appeals don’t carry any legal or financial weight, Discord and its upstart social media peers will eventually face consequences from Madison Avenue and Wall Street if they aren’t heeded.

Want to send thoughts or suggestions for Data Sheet? Drop me a line here.

Correction, May 16, 2022: An earlier version of this story inaccurately stated that the arrested suspect in the Buffalo, N.Y., shooting was influenced by information on the social media site Reddit.

Jacob Carpenter

NEWSWORTHY

Taking on China. The U.S. and European Union planned Monday to announce better collaboration on increasing semiconductor supplies, hoping to avoid a mutual escalation of government subsidies for the industry, Reuters reported. The commitment follows the EU’s decision earlier this year to scrap rules limiting government aid for chip manufacturing, as well as Congress’ preliminary passage of bills designed to funnel tens of billions of dollars to the sector. Both bodies are looking to reduce reliance on chips produced in Asia, where China gains geopolitical power through its chip market share.

Live and well? Netflix is working on live streaming capabilities that could allow the service to show competition series, stand-up specials, and other content in real time, Deadline reported, citing anonymous sources. The addition could help Netflix gain market share from linear networks and Disney, which announced last month that its Dancing with the Stars series will become its first show to broadcast live exclusively on Disney+. Deadline reported that there is no immediate timetable for debuting live streaming on Netflix, which last month reported its first quarterly net subscriber loss in a decade.

Your Uber robot is here. Uber planned Monday to launch two small autonomous food delivery pilots in Southern California, TechCrunch reported. One trial involves self-driving cars engineered by autonomous vehicle technology startup Motional, while the other features sidewalk-bound delivery robots created by Uber spinoff Serve Robotics. Uber officials said the test run will only include delivery from a few merchants, with drivers at the wheel of Motional’s vehicles for now.

A second sale. Rivian shares slipped 5% in mid-day trading Monday following news of Ford selling off another 7 million shares of the electric vehicle startup. Bloomberg reported late Friday that Ford, an early investor in Rivian, unloaded about 7.5% of its shares amid a dramatic downturn in the startup’s share price. Ford sold about 8 million shares earlier last week after a lock-up period ended, a transaction that sent Rivian’s stock price tumbling.

FOOD FOR THOUGHT

One big headache. Uber harbors ambitions for funneling wide swaths of customers into a monthly subscription plan that includes ride-sharing discounts and waived food delivery fees. Executing that vision hasn’t been easy. Current and former employees told Insider that a mix of internal strife, indecision, and a lack of urgency have delayed plans for widespread subscription adoption—with CEO Dara Khosrowshahi chiefly to blame. Uber still has 8 million global subscribers to its Uber One plan, a source told Insider, but some employees describe that total as underwhelming.

From the article:

Proposed at least five years ago, its launch was piecemeal, shepherded by a revolving door of executives and technical teams, and tested at length. Its widest public rollout only happened last November. 

Current and former employees debate why it took this long. Some blame company culture: Uber is fraught with corporate politics and an overreliance on testing above taking action, argued five current and former employees who worked on the product. They ultimately chalk that up to Khosrowshahi's inability to prioritize a strategy and foster enough collaboration among teams to execute it.

Others argue that it took time for Uber's business, specifically on the Eats side, to grow large enough for the economics of a subscription program to make sense.

IN CASE YOU MISSED IT

Sam Bankman-Fried says Bitcoin actually could have a future as money or a payments network—but there’s a catch, by Taylor Locke

SEC filing sheds light on why Trump might be downplaying possible return to Twitter, by Chris Morris

If you don’t like our content, you can quit: Netflix delivers a clear message to employees in new culture guidelines, by Sophie Mellor

Morgan Stanley warns about NFTs after Terra fiasco, by Chris Morris

Luna Foundation Guard has now dumped $2.4 billion from its Bitcoin reserves in failed attempt to defend TerraUSD peg, by Christiaan Hetzner

Ethereum co-founder says every ‘average smallholder’ impacted by Terra’s stablecoin crash should be made whole, cites FDIC’s $250,000 as ‘precedent’, by Taylor Locke

Did Elon Musk break Twitter’s NDA with a tweet?, by Erin Prater

BEFORE YOU GO

Checking you out. There’s no escaping targeted ads. Not even in your Marriott hotel room. The Wall Street Journal reported Monday that the world’s largest hotel chain plans to start using its guest data to refine the ads shown to customers on Marriott-affiliated websites and in-room televisions. The effort, done in partnership with Yahoo, will involve mining customers’ online searches and bookings to give advertisers more specific information about what kind of consumers stay at the hotel’s properties. Several major retail brand names have launched similar projects in recent months, a response to Apple and Alphabet unit Google tightening privacy rules on their products.