Brash. Controversial. A guard against rising digital threats around the globe. Google’s Project Zero is securing the Internet on its own terms. Is that a problem?
One Friday Afternoon in February, Tavis Ormandy, a virtuosic security researcher with a brown buzz cut and an uneasy smile, was performing some routine “fuzzing,” a common code-testing technique that blasts software with random data to expose faults, at his desk at Google (GOOGL) headquarters in Mountain View, Calif. The process was going as expected when he spotted something amiss in the data set. Weird, he thought. This isn’t typical corrupted data. Instead of the expected output, he saw bizarrely configured anomalies—strange chunks of memory strewn about. So he dug deeper.
After assembling enough information, Ormandy called his fellow security researchers into a huddle to share what he had found. The Google team, which goes by the name Project Zero, soon realized what it was looking at: a wide-ranging data leak spouting from a San Francisco company called Cloudflare. Most of the time, Cloudflare’s content-delivery network processes roughly a tenth of the world’s Internet traffic without a hitch. But Ormandy had discovered that the company’s servers were splattering people’s private data across the web. The information had been leaking for months.
Ormandy didn’t know anyone at Cloudflare, and he was hesitant to cold-call its generic support line so late in the day ahead of a three-day weekend. So he did the next best thing he could think of. Ormandy took to Twitter to appeal to the tens of thousands of people who follow him there.
Could someone from Cloudflare security urgently contact me
The time stamp was 5:11 p.m. Pacific Time.
Ormandy did not bother to alert the company’s Twitter account by tagging its name with an “@” symbol. He didn’t need to. Such is his reputation among the zealous community of information-security professionals that within 15 minutes of Ormandy’s pressing “Send,” everyone in the world who needed to know—and plenty who didn’t—would see the note.
At 1:26 a.m. local time John Graham-Cumming’s phone, plugged into an outlet by his bedside in London, buzzed him awake. The chief technology officer of Cloudflare rubbed his eyes and reached to pick up the rumbling handset. Missed call. A colleague—one of the few whom Graham-Cumming had white-listed to reach him after midnight—had called. The CTO fired off a text message asking what was up.
His colleague responded immediately.
very serious security issue
Graham-Cumming sat up, alarmed, and replied.
I will get online
The CTO rose from bed, went downstairs to the basement, and grabbed the emergency bag—charger, headphones, extra batteries—that he had stowed for such an occasion. He booted up his laptop computer and quickly joined a Google Hangout with his colleagues at Cloudflare’s California headquarters.
The security team briefed him on the unfolding situation. Google’s Project Zero team had found a bug in Cloudflare’s infrastructure—a bad one. The servers that help run more than 6 million customer websites, including those of the FBI, Nasdaq, and Reddit, had sprung a data leak. Anyone could access a Cloudflare-supported site and retrieve in certain circumstances the intimate details—authentication tokens, cookies, private messages—of users of another site on its network, among them Uber, 1Password, OKCupid, and Fitbit.
The information was hidden in plain sight. Worse, search engines and other web crawlers had been storing the leaked data in their caches for months. Plugging the leak would not fully solve the problem.
“I liken it to an oil spill,” Graham-Cumming says. “It’s easy to deal with a hole in the side of a tanker, but then you’ve got a lot of seabeds that need to be cleaned up.”
So Cloudflare’s engineers got to work. Security chief Marc Rogers, who in his spare time serves as a consultant for the USA Network hacker drama Mr. Robot, led the triage effort. In less than an hour the team pushed out an initial mitigating update that plugged the leak worldwide. After several hours the technicians successfully rolled back functions that had contributed to the error. Almost seven hours after Ormandy fired off his tweet, Cloudflare’s engineers managed to enlist the major search engines—Google, Microsoft (MSFT), Yahoo—to clear their historical web page caches.
It was the beginning of a very long weekend. Cloudflare engineers spent the rest of it evaluating how much and what kind of data had leaked as well as how far the mess had spilled.
Google’s Project Zero team was initially impressed with the rapid response of Cloudflare, which has a reputation for transparency when it comes to security matters. But the relationship began to fray as the teams negotiated when they would publicly reveal what had transpired. The companies tentatively agreed to make an announcement as early as Tuesday, Feb. 21. As the day waned, Cloudflare decided it needed more time for cleanup. Tuesday became Wednesday. Wednesday became Thursday. Google put its foot down: Thursday afternoon would be the day the companies published details of the leak, which Ormandy dubbed “Cloudbleed,” whether or not Cloudflare had completed its assessment and ensured that the leaked data was clear from online caches.
Both advisories went up on Feb. 23. A weeklong Internet panic ensued.
You don’t have to be a member of Google’s Project Zero to know that security crises are on the rise around the globe. Every company has become a tech company—and so hacks are increasingly becoming commonplace, draining corporate bank accounts, spying on individuals, and interfering in elections. The headlines are sobering: More than 1 billion Yahoo accounts compromised. Tens of millions of dollars stolen through the SWIFT financial network. Countless private emails from the Democratic National Committee exposed ahead of the 2016 U.S. presidential election. (For more on how business is responding, read “Hacked: How Business Is Fighting Back Against the Explosion of Cybercrime.”)
U.S. companies and government agencies reported 40% more breaches in 2016 than in 2015, and that’s a conservative estimate, according to the Identity Theft Resource Center. At the same time, the average cost of a data breach now runs organizations $3.6 million, according to an IBM-sponsored study conducted by the Ponemon Institute, a research group.
Whether the result of a programmer’s error or hackers working for a nation-state, data leaks are the new norm. So executives are coming to terms with the idea that it might be more economical to nip coding issues in the bud before they lead to bigger—and messier—problems down the road.
But it’s not that simple. Too many organizations either don’t prioritize security or view it as an impediment to meeting product development and delivery deadlines. According to Veracode, an application-security firm acquired by CA Technologies earlier this year, 83% of the 500 IT managers it surveyed admitted that they had released code before testing for bugs or resolving security issues. At the same time, the security industry faces a talent shortage. Cisco (CSCO) estimates that there are 1 million unfilled security jobs worldwide, and Symantec predicts that will increase to 1.5 million by 2019. Some estimates believe that figure will grow to 3.5 million by 2021.
Even if a company has the funds, initiative, and cachet to support a proper security staff, it’s not immune to shipping flawed code. The best quality-assurance programs and agile development practices can’t catch every bug.
So many companies, including Microsoft and Apple (AAPL), have internal security-research teams that investigate their own software. But few have teams that focus on the software made by other companies. That is what makes Google so unusual. To Ormandy and the dozen or so ace computer crackers that make up Google’s Project Zero, there are no boundaries to their jurisdiction—anything that touches the Internet is fair game. Policing cyberspace isn’t just good for humanity. It’s good for business too.
Google officially formed Project Zero in 2014, but the group’s origins stretch back another five years. It often takes an emergency to drive most companies to take security seriously. For Google, that moment was Operation Aurora.
In 2009, a cyberespionage group associated with the Chinese government hacked Google and a number of other tech titans, breaching their servers, stealing their intellectual property, and attempting to spy on their users. The pillaging outraged Google’s top executives—enough so that the company eventually exited China, the world’s biggest market, over the affair.
The event particularly bothered Google co-founder Sergey Brin. Computer-forensics firms and investigators determined that the company had been hacked not through any fault of Google’s own software, but via an unpatched flaw in Microsoft Internet Explorer 6. Why, he wondered, should Google’s security depend on other companies’ products?
In the months that followed, Google began to get more aggressive in demanding that rivals fix flaws in their software’s code. The battles between Google and its peers soon became the stuff of legend. At the center of several of these spats was none other than bug hunter Tavis Ormandy, known for his smashmouth approach to getting flaws fixed. (Ormandy declined to be interviewed for this story.)
For example, not long after Operation Aurora became public, Ormandy disclosed a flaw he found months earlier in Microsoft’s Windows operating system that could allow attackers to commandeer people’s PCs. After waiting seven months for the company to issue a patch, he took matters into his own hands. In January 2010, Ormandy posted details of the flaw on a “full disclosure” mailing list where security researchers notify peers of new vulnerabilities and attack methods. His thinking: If Microsoft wasn’t going to address the problem in a timely manner, people should at least know about the issue so they can develop their own solutions. A few months later, he did the same for a bug affecting Oracle’s Java software as well as for another big Windows flaw, the latter just five days after reporting it to Microsoft.
Critics of the practice censured Ormandy’s behavior, claiming it damaged people’s security. (Apple, Microsoft, and Oracle would not comment for this story.) In a corporate blog post, two Verizon (VZ) security specialists called researchers who choose the full disclosure route “narcissistic vulnerability pimps.” Ormandy ignored the flak. In 2013 he again chose to make a Windows bug public before Microsoft developed a fix for it. Without the threat of a researcher going public, he reasoned, companies have little pressure to fix a flaw in a timely manner. They can sit on bugs indefinitely, putting everyone at risk.
Google quietly began to formalize what became Project Zero in 2014. (The name alludes to “zero-day” vulnerabilities, the term security pros used to describe previously unknown security holes, ones that companies have had no time, or zero days, to prepare for.) The company established a set of protocols and allowed Chris Evans (no relation to Captain America), former head of Google Chrome security, to take the helm. Evans in turn began recruiting Googlers and others to the team.
Security: A Glossary
Bug: An unexpected error in computer code. The ones with security implications are called “vulnerabilities.”
Zero Day: A vulnerability that people and companies have had no time—“zero days”—to fix.
Exploit: A computer program that a hacker crafts to take advantage of a known vulnerability.
He signed on Ian Beer, a British-born security researcher based in Switzerland, who had demonstrated a penchant for sussing out Apple’s coding errors. He brought on Ormandy, a British bruiser known for his highly publicized skirmishes with Microsoft. Evans enlisted Ben Hawkes, a New Zealander known for stomping out Adobe Flash and Microsoft Office bugs. And he invited George Hotz, a precocious teenager who had earned $150,000 after busting open the Google Chrome browser in a hacking competition earlier that year, to be an intern. (Current members of Project Zero declined multiple requests to be interviewed about their work for this story.)
The first sign that Project Zero had arrived came in April 2014 when Apple credited a Google researcher in a brief note for discovering a flaw that would allow a hacker to take control of software running Apple’s Safari web browser. The note thanked “Ian Beer of Google Project Zero.”
On Twitter, the information-security community openly wondered about the secretive group. “What is Google Project Zero?” asked Dan Guido, cofounder and CEO of the New York–based cybersecurity consultancy Trail of Bits, in a tweet posted April 24, 2014. “Employee of mysterious ‘Google Project Zero’ thanked in Apple security update changelog,” noted Chris Soghoian, then the chief technologist at the American Civil Liberties Union.
More credits soon appeared. In May, Apple credited the discovery of several bugs in its OS X operating system to Beer. A month later, Microsoft patched a bug that made it possible to defeat its malware protection, noting the help of “Tavis Ormandy of Google Project Zero” in an advisory.
By then, the team had generated considerable buzz among those who track security issues. Evans finally made its presence officially known in a blog post on the company’s website. “You should be able to use the web without fear that a criminal or state-sponsored actor is exploiting software bugs to infect your computer, steal secrets or monitor your communications,” he wrote, citing recent examples of spies targeting businesses and human-rights activists as unconscionable abuses. “This needs to stop.”
Evans left the team a year later to join Tesla and now serves as an adviser with the bug bounty startup HackerOne. (Hawkes now leads Project Zero.) Today Evans is more circumspect in describing the group’s origins. “The foundations for Project Zero were laid across years of thoughtful lunchtime conversations and years of observing the evolution of attacks,” he says. “We wanted to create jobs focused exclusively on top-tier offensive research, to attract the best in the world to the public research space.”
It’s a more difficult challenge than it seems. Private money soaks up many of the world’s best hackers, luring them to work behind closed doors, where governments and other entities, through brokers, will pay top dollar for their findings. When that research doesn’t see the light of day, Evans says, people suffer.
In the three years since Google’s Project Zero officially came together, the elite hacker squad has built a reputation for being among the most effective computer bug exterminators on the planet. Although an ordinary consumer is unlikely to recognize any one of their names—James Forshaw, Natalie Silvanovich, Gal Beniamini—the world owes them a debt of gratitude for sealing up the devices and services that run our digital lives. The team is responsible for a litany of improvements in other companies’ products, including finding and helping to patch more than a thousand security holes in operating systems, antivirus software, password managers, open-source code libraries, and other software. Project Zero has published more than 70 blog posts about its work to date, some of the best public security research available on the web today.
The team’s work indirectly benefits Google’s primary business: online advertising. Protecting Internet users from threats means protecting the company’s ability to serve those users ads. Project Zero’s effort to hold vendors’ feet to the fire also forces them to fix bugs that cause Google products to crash.
“This is a dorky name for it, but it’s like a sheepdog,” says Dino Dai Zovi, a cybersecurity entrepreneur, noted Apple hacker, and former head of mobile security at Square. “A sheepdog is not a wolf. It’s kind of benevolent, but it still chases the sheep into line to get them back into the pen.”
In April three members of Project Zero traveled to Miami to attend the Infiltrate security conference, a gathering focused entirely on the offensive side of hacking.
In a city built on suntans and sports cars, the computing cohort look somewhat out of place. Hawkes, Ormandy, and Thomas Dullien, a German security researcher and member of the Project Zero team who is better known by the hacker moniker “Halvar Flake,” gather on the lawn of the swanky Fontainebleau hotel to sip mojitos under the rustling palm trees. Seated at a table with a handful of other conference attendees, the Googlers chat about current affairs, favorite sci-fi tales, and how shameful it is that more is not done to preserve hacker history.
At one point Ormandy swipes a pair of gaudy Versace sunglasses left on a table by Morgan Marquis-Boire, a former Google employee, well-known malware researcher, and current head of security at eBay founder Pierre Omidyar’s media venture First Look Media. The Florida sun has subsided, but Ormandy places the shades over his blue eyes and mugs. He looks ridiculous.
Infiltrate organizer Dave Aitel, an ex-NSA hacker who runs Immunity, an offensive hacking shop, whips out his phone to take a photo. His subject contorts his hands into a heavy metal fan’s “sign of the horns.” Behold Tavis Ormandy: online, a quarrelsome critic who suffers no fools; offline, a genial geek who happily horses around.
“People give you a lot of shit, Tavis,” Aitel says, referring to the frustrating battles Ormandy must endure while prodding vendors to fix their code. “You know, you don’t have to deal with that.” With an impish grin, Aitel proceeds with a facetious attempt to persuade Ormandy to join the “dark side” of hacking—researchers who find bugs and then sell them for a profit rather than report them to the affected companies, rendering the bugs kaput.
Ormandy shrugs off Aitel’s offer, laughs, then sets the glasses back on the table. He may be a troublemaker, but his aims are pure. (Ormandy allowed this reporter to hang around, but later declined to comment.)
Despite its hard-edged reputation, Project Zero has had to become more flexible as its high-minded ideals collide with the complexities of the real world. The team initially kept to a strict 90-day disclosure deadline, or just seven days for “actively exploited” bugs, but several instances of disclosure shortly before companies had scheduled to release updates, such as Microsoft and its recurring “Patch Tuesday,” caused the group a lot of backlash. (It has since added a 14-day extension after the 90 days in the event that a vendor has a patch prepared.)
Project Zero has some of the most explicit disclosure policies in the technology industry, says Katie Moussouris, who helped create the disclosure policy at Microsoft and now runs her own bug-bounty consulting firm called Luta Security. That’s a good thing, she says. Many companies fail to establish guidelines on how to report bugs or lack policies on how or when a researcher should expect a bug to go public. Some organizations provide companies with even less time to fix their software. Cert CC, a group run out of Carnegie Mellon University, has a stated 45-day policy—half that of Project Zero, though the group allows for more leeway on individual bases.
Luta Security CEO Katie Moussouis.Photo: Joe Pugliese—August Photo: Joe Pugliese—AugustBug Baroness and Luta Security CEO Katie Moussouris explains the economy of exploits:
There are two markets for bugs: offense and defense. The former is made of nation-states, organized crime groups, and other attackers. The latter consists of bug-bounty programs and companies that sell security products. The offense market pays higher prices and doesn’t have a ceiling. They’re not just buying a vulnerability or an exploit; they’re buying the ability to use it without being detected. They’re buying silence. The defense market can’t pay as much. It’s not like vendors are going to compensate their top developers a million dollars. Even though major companies’ code quality is improving, complexity continues to increase. That means more bugs. What security researchers do with a particular bug may depend on their financial needs, their dispositions about a piece of software or vendor, and their own personal risk. It’s not black-hat sellers vs. white hat.
And Project Zero is as quick to praise a company’s actions to fix a bug as it is to criticize a sluggish response. Earlier this year, Ormandy tweeted that he and colleague Natalie Silvanovich had “discovered the worst windows remote code exec in recent memory,” meaning a way to take over a Windows-based system from afar. “This is crazy bad,” he wrote. The two worked with Microsoft to patch the bug. “Still blown away at how quickly @msftsecurity responded to protect users, can’t give enough kudos. Amazing,” he wrote in a follow-up tweet. Apparently, it’s never too late to improve.
Technology companies may cringe at Project Zero’s audacity, but they should take comfort in the fact that its hackers are willing to resist the urges that drive some researchers to put their findings up for sale. In the years since hacking became professionalized, markets have sprouted for the bugs that Project Zero discloses. Governments, intelligence services, criminals—everyone wants them for themselves and is willing to pay top dollar. The growing adoption of bug bounty programs at software companies is a slight tip of the scale in the other direction, offering compensation to researchers for their time, effort, and expertise. But the payment on the bounty side will never meet the compensation one can get from murkier markets.
Still blown away at how quickly @msftsecurity responded to protect users, can't give enough kudos. Amazing.
— Tavis Ormandy (@taviso) May 9, 2017
“Whatever Google’s bug bounty rewards are, the Chinese government will pay more for it,” says Bruce Schneier, a well-known security guru and executive at IBM.
Back at the Fontainebleau, Dullien tells me he is amazed at how in-demand the skills of hackers have become. What was once a hobby done in dark basements is now a profession at home in the halls of government.
“This was all a ’90s subculture, like hip-hop or break dancing or skateboarding or graffiti,” he says. “It just so happened that the military found it useful.”
According to Matthew Prince, CEO and cofounder of Cloudflare, the leak uncovered by Google’s top bug hunters initially cost his company about a month of growth. (The setback was temporary, he says: Cloudflare’s transparency during the process helped it attract new business.)
If he’s at all sour about the experience, Prince doesn’t let it show. He knows what it’s like to be targeted by truly malicious hackers. A few years ago a hacker group called “UGNazi” broke into Prince’s personal Gmail account, used it to gain control over his corporate email account, then hijacked Cloudflare’s infrastructure. The hooligans could have done significant damage. Instead, they decided to redirect 4chan.org, a common hacker hangout, to their personal Twitter profile for publicity.
Prince still regrets not informing his customers of the full extent of the Cloudbleed issue before Google and Cloudflare published their initial findings. He wishes his company had alerted customers before they read about the leak in news reports. Even so, Prince believes in retrospect that the Project Zero team was right on the timing of when to go live with the disclosure. To his knowledge, no one has uncovered any significant damages related to the leak in the time since. No passwords, credit card numbers, or health records have turned up, despite their initial fears.
Prince says Cloudflare has put new controls in place to prevent such an incident from happening again. The company began a review of all of its code and hired outside testers to do the same. It also instituted a more sophisticated system that identifies common software crashes, which tend to indicate the presence of bugs.
“I have many more gray hairs and will likely live a year less than before as a result of those 14 days,” Prince says about the discovery and the aftermath of the leak. “Thank God it was Tavis and that team who found it and not some crazy hacker.”
Of course, Prince will never be able to rule out the possibility that another person or organization has copies of the leaked data. And that’s just Project Zero’s point. For every one of its team members, there are countless other researchers working in private with less noble goals in mind. It’s the devil you know—or the devil you don’t.
A version of this article appears in the July 1, 2017 issue of Fortune.
