Panera Bread messed up big time.
Even without getting into the technical failures that caused the restaurant giant to leak personal information for what appears to be millions of customers, the company’s handling of the bug reporting and breach disclosure processes alone proved abominable. They represent a masterclass in how not to behave when confronted with a cybersecurity predicament.
It’s worth reviewing what the company got wrong so that other organizations can learn from its mistakes. Fortune has pulled together five lessons that companies can take away from the data-exposing debacle, which left Panera customers’ names, email and street addresses, birthdays, and the last four digits of their payment cards out in the open for months.
The purpose here is not to bash Panera—although such criticism seems to be warranted—but rather to learn from its foul-up. “The story here isn’t the vulnerability, it’s the response,” Mårten Mickos, CEO of HackerOne, a bug bounty reporting firm, told Fortune in an email.
Read more: “Google’s Elite Hacker SWAT Team vs. Everyone”
Moreover, it’s about what other businesses may do when they find themselves in a similar situation. Dylan Houlihan, the security researcher who originally discovered the exposed customer data (including his own) and reported it to Panera in August 2017, found himself ignored by the company for months. Fed up, he posted his findings publicly to force Panera’s hand into fixing the security bug. But as even he put it in a post on Medium: focusing strictly on this one company would be myopic.
“It’s easy to bully Panera Bread for this, but in my opinion we need to take Panera Bread’s actions as symptomatic of a much larger issue with security reporting and compliance,” wrote Houlihan, founder of Breaking Bits, a New York-based digital security firm. “This is not a problem unique to any particular type of company. This has happened before and it will continue to happen.”
The below points lay out where Panera stumbled. (Panera did not reply to Fortune’s request for comment, including one seeking to verify Houlihan’s account of their interactions.)
To avoid the same pitfalls, read on.
1. Post a contact page for bug reports
If a company has no dedicated webpage that clearly details the process for security researchers to submit vulnerability reports, then it is setting itself up to fail from the get-go. This page should ideally be separate from a standard customer support line, where ordinary users might go to report hijacked accounts, and the submissions to it should promptly be reviewed by security pros with the right qualifications. Look to companies such as Google, Microsoft, Facebook, and Apple, for outstanding examples of such contact pages.
When Houlihan sought the proper reporting channel at Panera, he found no such thing. Instead, he took a shot in the dark by guessing at what might be an appropriate e-mail address, security@panerabread.com. When the message he sent there bounced back, Houlihan said he tried reaching out to the company on Twitter and then LinkedIn. Eventually, a mutual connection in the cybersecurity industry provided him an introduction to Panera’s information security director.
Researchers shouldn’t have to jump through so many hoops to help a company out. This doesn’t mean that companies have to offer bug bounties, or rewards for finding security flaws (as much as they’re appreciated); they just need to provide an avenue for researchers to responsibly disclose vulnerabilities. Help them help you.
2. Don’t shoot the messenger
It should go without saying, but you should treat people with courtesy.
When Houlihan heard back from Panera’s security lead, the employee took a defensive stance and seemed to accuse the researcher of being a scammer. In an initial email exchange posted by Houlihan to Medium, the security team leader said his group ignored Houlihan’s pleas because they were “very suspicious and appeared scam in nature.” “If this is a sales tactic,” the director chastised Houlihan in an email reply, then Houlihan’s attempt at an approach “would not be a good way to start off.”
Everyone has a bad day, sure. But if Houlihan’s advances “appeared scam in nature,” it’s likely because the researcher had to dig up, in the absence of a dedicated bug reporting page, alternate means of reaching Panera’s security team, including affiliated social media accounts. This misunderstanding could have been prevented if Panera offered a clear vulnerability reporting policy. In other words, see point No. 1; and if you don’t have such a bug reporting policy in place, at least give researchers the benefit of the doubt when they come knocking.
3. Don’t leave a tipster hanging
Be prompt in your reply.
According to Houlihan, after he persuaded the security director to send him a PGP key—an encryption tool designed to protect communications—and used it to send over his vulnerability report, the security team leader went silent. Houlihan said he repeatedly emailed the manager over the course of several days, as the time stamps on his email messages seem to indicate, to ask for an update. To be fair, one might note that the (mostly one-sided) exchange occurred in the midst of a summer weekend. Still, it took six days for Panera’s security lead finally to reply: “Thank you for the information we are working on a resolution.”
Don’t leave bug reporters dangling, especially when customer data is potentially on the line. Companies should provide clear guidance to researchers, letting them know how long they can expect to wait to hear back as well as any justifications for delay. People tend to be understanding.
4. Fix things. Promptly.
When you know something is broken, fix it.
From the time of Houlihan’s bug submission, Panera allegedly let eight months go by without addressing the vulnerability that exposed people’s information. (Houlihan said in his recap that he “checked on this vulnerability every month or so…. So I personally know for a fact that it was never patched in the interim. And even if it was, that it would be fixed and inadvertently reintroduced is nearly as bad as not fixing it at all.”) This inaction drove Houlihan to post his findings online, and to approach an investigative journalist, Brian Krebs, in the hopes of garnering attention for the issue, escalating its priority, and thereby forcing Panera to patch the hole in its systems.
Casey Ellis, founder and chief technology officer of Bugcrowd, a bug bounty startup, said in an email that its shame when researchers must resort to “full disclosure”—revealing their findings to the public before an organization has addressed the issue—but it is sometimes the only way to get a vulnerability fixed. “Full Disclosure is a necessary but inherently disruptive thing: It’s the last tool available to security researchers when a risk they’ve identified is being ignored,” he wrote to Fortune. “Vendors should work to avoid it, and in an ideal world it is completely unnecessary for a vulnerability.”
the big one: full disclosure still works, is still relevant and still has a place – but it's use is a symptom of process failure on the vendor side, the hacker side, or both.
— cje (@caseyjohnellis) April 3, 2018
5. But don’t rush out a flawed response
Take the time to understand what’s wrong, and to address it.
After Krebs’ story published Tuesday, Panera appeared to attempt to commandeer the narrative by supplying a hasty response to inquiring news outlets, like Fox News, that claimed the problem was less significant than it was. John Meister, Panera’s chief information officer, said in a statement quoted by Fox that “this issue is resolved” and that “our investigation to date indicates that fewer than 10,000 consumers have been potentially affected by this issue.” Krebs followed up by posting tweets that demonstrated how many more people—perhaps as many as 37 million—could have been, and likely still were at that time, affected.
A better reply would have been something along the lines of, “we are working diligently to address these issues and will provide an update when we have more information to share.”
Please, run a proper audit first. Don’t downplay security issues when you don’t yet have the full picture. Dashing off a statement based on a most preliminary understanding, as Panera appears to have done, runs the risk of spreading misinformation, deliberately or not, which will only serve to hurt one’s customers and oneself.
Per my last tweet, Panera issued a statement to Fox News saying the breach only impacted 10,000 customer accounts. Interesting that they had no numbers for me, and yet had this 10k number all ready to go on the same day this was "discovered," eight months after it was reported.
— briankrebs (@briankrebs) April 2, 2018
you know what, let's go for 37M instead of 7M: https://t.co/7DTaherzMi
— briankrebs (@briankrebs) April 2, 2018
If you’ve got a business with a digital component—as just about every company has these days—take heed. Panera is not unique; you can learn from its example. These five bullets are a start.
Katie Moussouris, founder and CEO of Luta Security, a vulnerability disclosure and bug bounty consultancy, told Fortune a Twitter direct message that Panera’s shoddy approach to dealing with cybersecurity issues is, unfortunately, all too common among businesses today. “Panera’s reaction of initial suspicion, followed by silence, hoping the researcher would move on, is sadly still prevalent in the majority of companies & governments,” she wrote.
“Vulnerabilities happen to every organization, without exception,” she said. “Being prepared for the inevitable report is just good business.”
Best to put a plan in place now.
