Have you ever been attacked by trolls on social media? I have. In December a mocking tweet from white supremacist David Duke led his supporters to turn my Twitter account into an unholy sewer of Nazi ravings and disturbing personal abuse. It went on for days.
We’re losing the Internet war with the trolls. Faced with a torrent of hate and abuse, people are giving up on social media, and websites are removing comment features. Who wants to be part of an online community ruled by creeps and crazies?
Fortunately, this pessimism may be premature. A new strategy promises to tame the trolls and reinvigorate civil discussion on the Internet. Hatched by Jigsaw, an in-house think tank at Google’s parent company, Alphabet (GOOGL), the tool relies on artificial intelligence and could solve the once-impossible task of vetting floods of online comments.
To explain what Jigsaw is up against, chief research scientist Lucas Dixon compares the troll problem to so-called denial-of-service attacks in which attackers flood a website with garbage traffic in order to knock it off-line.
“Instead of flooding your website with traffic, it’s flooding the comment section or your social media or hashtag so that no one else can have a word, and basically control the conversation,” says Dixon.
Such surges of toxic comments are a threat not only to individuals, but also to media companies and retailers—many of whose business models revolve around online communities. As part of its research on trolls, Jigsaw is beginning to quantify the damage they do. In the case of Wikipedia, for instance, Jigsaw can measure the correlation between a personal attack on a Wikipedia editor and the subsequent frequency the editor will contribute to the site in the future.
The solution to today’s derailed online discourse lies in reams of data and deep learning, a fast-evolving subset of artificial intelligence that mimics the neural networks of the brain. Deep learning gave rise to recent and remarkable breakthroughs in Google’s translation tools.
In the case of comments, Jigsaw is using millions of comments from the New York Times and Wikipedia to train machines to recognize traits like aggression and irrelevancy. The implication: A site like the Times, which has the resources to moderate only about 10% of its articles for comments, could soon deploy algorithms to expand those efforts 10-fold.
While the tone and vocabulary on one media outlet comment section may be radically different from another’s, Jigsaw says it will be able to adapt its tools for use across a wide variety of websites. In practice, this means a small blog or online retailer will be able to turn on comments without fear of turning a site into a vortex of trolls.
Technophiles seem keen on what Jigsaw is doing. A recent Wired feature dubbed the unit the “Internet Justice League” and praised its range of do-gooder projects.
But some experts say that the Jigsaw team may be underestimating the challenge.
Recent high-profile machine learning projects focused on identifying images and translating text. But Internet conversations are highly contextual: While it might seem obvious, for example, to train a machine learning program to purge the word “bitch” from any online comment, the same algorithm might also flag posts in which people are using the term more innocuously—as in, “Life’s a bitch,” or “I hate to bitch about my job, but …” Teaching a computer to reliably catch the slur won’t be easy.
“Machine learning can understand style but not context or emotion behind a written statement, especially something as short as a tweet. This is stuff it takes a human a lifetime to learn,” says David Auerbach, a former Google software engineer. He adds that the Jigsaw initiative will lead to better moderation tools for sites like the New York Times but will fall short when it comes to more freewheeling forums like Twitter and Reddit.
Such skepticism doesn’t faze Jigsaw’s Dixon. He points out that, like denial-of-service attacks, trolls are a problem that will never be solved but their effect can be mitigated. Using the recent leaps in machine learning technology, Jigsaw will tame the trolls enough to let civility regain the upper hand, Dixon believes.
Jigsaw researchers also point out that gangs of trolls—the sort that pop up and spew vile comments en masse—are often a single individual or organization deploying bots to imitate a mob. And Jigsaw’s tools are rapidly growing adept at identifying and stifling such tactics.
Dixon also has an answer to the argument that taming trolls won’t work because the trolls will simply adapt their insults whenever a moderating tool catches on to them.
“The more we introduce tools, the more creative the attacks will be,” Dixon says. “The dream is the attacks at some level get so creative no one understands them anymore and they stop being attacks.”
***
Driven from social media by trolls
2015–16
Increasingly, popular media sites and blogs, from NPR to Reuters, are eliminating comments from their pages.
July 2015
Ellen Pao, interim CEO of Reddit, resigns in the wake of what she calls “one of the largest trolling attacks in history.”
July 2016
Movie actress Leslie Jones quits Twitter after trolls send a barrage of racist and sexual images. In one of her final tweets, she writes, “You won’t believe the evil.”
***
A version of this article appears in the February 1, 2017 issue of Fortune with the headline “Troll Hunters.”
