Above video by Sinelab
A cyber-invasion brought Sony Pictures to its knees and terrified corporate America. The story of what really happened—and why Sony should have seen it coming. A special three-part investigation.
Part 2: The storm builds
We will take “a merciless counter-measure.”
On June 17, leaked emails show, Sony’s appetite for mocking a “real persona” instantly diminished. Days after the film’s first trailer appeared online, Hirai, who had just screened the movie, called Lynton, concerned about roiling already fraught relations between Japan and North Korea. As Hirai saw it, this made The Interview dangerous fare for a Japanese company.
Lynton scrambled, first by yanking the trailer from the Internet for re-editing. As part of Hirai’s One Sony initiative, the studio had just added the Sony logo to the credits for all films released under the studio’s brands, which include Columbia Pictures, TriStar, and Screen Gems. Now orders went out to erase “Sony” from everything associated with The Interview in an attempt to downplay its Japanese parentage. Plans for a limited Asia release were also scrapped. “Have to keep whole interview thing under wraps,” Lynton told Pascal.
That wouldn’t be easy. Within days a North Korean government spokesman warned that The Interview’s release would represent “the most blatant act of terrorism and war,” and threatened “a merciless counter-measure.” (The government later filed formal complaints with the White House and the United Nations Security Council.) North Korea has long been known for its threats against other countries, but most have turned out to be mere bluster.
Still, the Kim regime had been widely blamed for a series of cyberattacks, especially against its archenemy, South Korea, and was believed to employ a cadre of several thousand army hackers. The worst incident had occurred in March 2013. Known as the DarkSeoul attack, it caused $700 million in damage to South Korean banks and broadcasters, freezing ATMs and erasing the hard drives of 30,000 computers. The hackers in this episode, which received considerable press coverage in the U.S., posted a notice featuring images of skulls.
Yet Sony Pictures executives were caught off guard by the growing storm over their film. They tried to assess the danger. Reached in Europe after Hirai’s call, studio executive Doug Belgrad advised Lynton that he was now “doing the homework on whether there is precedent on depicting and/or killing a living leader on film.” Emails show Lynton tapped his personal network, conferring informally with two outside experts. (Rogen brushed it all off with a jocular tweet: “People don’t usually wanna kill me for one of my movies until after they’ve paid 12 bucks for it.”)
In a written statement on behalf of Lynton, Sony spokesman Lawson insists that the “extremely knowledgeable” experts the CEO consulted “gave no hint or warning of the possibility of a cyberattack.” Indeed, one expert Lynton spoke with, Daniel Russel, assistant secretary of state for East Asian and Pacific affairs, made no mention of hacking risk, according to a note Lynton prepared memorializing their conversation.
But Lynton got a different message from the expert he consulted most extensively. Bruce Bennett, a North Korea specialist with the Rand Corp.—where Lynton serves on the board—says he did warn the Sony CEO that a cyberattack was “a possibility.”
After watching The Interview, Bennett sent him a three-page memo assessing the situation even before the Koreans began protesting the film, then had several follow-up exchanges with Lynton. Bennett advised him that the North Koreans frequently made empty threats, and there probably wasn’t much to fear.
Bennett’s memo noted the likelihood that North Korea would probe Sony’s computer systems: “Even if North Korea doesn’t know about the film yet, as soon as they do find out about it, they will likely explore Sony’s computer systems to see if Sony is ready to deal with North Korean criticism.” (The memo was not retrievable from the hacked documents; Bennett read the passage to Fortune.)
In their follow-up conversations, Bennett says, he went further. He says he told the CEO that the Kim regime employed hackers “who could potentially cause damage,” described the DarkSeoul episode, and counseled: “You need to realize something could happen in that area.” Lawson denies this: “If [Lynton] had received any kind of warning, his next call would have been to a cyberexpert to ask about it … In their many phone conversations, Bennett never mentioned the possibility of a cyberattack on the studio … ”)
Rogen and Goldberg also received warnings of a possible cyberattack, according to their spokesman, Matt Labov. Even before they began shooting the film the pair sought the advice of Rich Klein, whose Washington, D.C.–based consulting firm, McLarty, advises Hollywood on sticky geopolitical problems. After reading their script, Klein tells Fortune, he advised the filmmakers to expect North Korean “blowback,” possibly in the form of an electronic assault. He urged them to change their banking and email passwords and closely monitor their Internet accounts, and passed on the name of a cybersecurity adviser.
Klein says he also feared that North Korea might unleash a cyberassault on the studio to try to block The Interview’s release. Rogen and Goldberg relayed that message to Sony executives, according to Labov. “We felt that everybody involved in this had to protect themselves—the studio and the filmmakers,” says Klein. “The North Koreans are pretty aggressive cyberwarriors … It’s just surprising to me that there wasn’t a more robust sense of alarm and caution.” (Sony’s spokesman also denies receiving a warning from Rogen and Goldberg.)
Instead of hardening its network defenses, emails show, Sony focused on somehow trying to offend the North Koreans less. Their actions befit a Hollywood farce. Never mind that Sony was planning to release a film that portrayed the violent death of a real head of state; the company spent $550,000 to digitally alter the movie’s images of Kim family members shown on a wall mural and jacket pins worn by movie characters. It banished marketing materials for The Interview from Sony websites. And the studio prepared a statement insisting the movie was “a fictionalized comedy that is not in any way related to current events.”
In early August, Pascal departed with her family for a long vacation through Asia. Lynton decamped to Martha’s Vineyard. By then they’d decided to postpone The Interview’s release until Christmas Day, buying time to tackle another issue. On Hirai’s orders, studio executives had begun a three-month battle with the filmmakers to soften the movie’s gory climactic scene, in which a tank shell strikes Kim’s helicopter and kills him in a slow-motion, head-popping, flesh-dripping ball of fire.
The Hollywood Reporter caught wind of this, Sony Pictures PR executive Jean Guerin advised Pascal and other executives in an email. It was preparing to report that “Corporate Sony” had asked for “a few key edits,” including “that the melted face of Kim Jong-un will now be taken out of the final cut … ” The studio denied that changes were being made in response to outside pressure. “We are dismissing [the reporter’s] premise and letting her know off the record that this is normal/what happens in the filmmaking process,” Guerin told the executives.
The article, which appeared on Aug. 13, suggested that Sony was changing the movie to appease the North Koreans. Rogen was livid. As he saw it, this painted him as a sellout. Executives convened an emergency call to plan damage control. Asked to join the session, Lynton said he was at a dinner on Martha’s Vineyard and couldn’t get up to leave. “I am sitting between president obama and Hillary Clinton … If we need a change of strategy then I don’t want anything done until we speak.”
Hirai and Lynton, who found the movie more offensive than funny, wanted the “head-popping scene” eliminated entirely. “We cannot be cute here,” Lynton wrote in one email. “What we really want is no melting face and actually not seeing him die.” But Rogen was fiercely resisting major changes in what he called the “awesome” shot.
Sony had final authority over the editing of the movie. But according to interviews and leaked emails, the studio feared that if it imposed its will, Rogen would disassociate himself from the film, creating a box-office and PR debacle. “This movie is supposed to be controversial,” he emailed Pascal. “That was your pitch to us amy. There’s nothing controversial about a movie that has been tempered to appease the very people it’s mocking.” (Rogen barraged Pascal, then vacationing through Vietnam and Bali, with messages, insisting they talk “as soon as humanly possible.” Pascal forwarded one email to a colleague, asking, “Can I be lost in the jungle?”)
On Sept. 25, Pascal, back in L.A. and sitting in temple on the Jewish New Year holiday, emailed Rogen an emotional personal plea to make the scene “a little less gory.” “No one has backed you more than I have,” she wrote. “And this isn’t some flunky it’s the chairman of the entire sony corporation who I am dealing with. You must know there are very few relationships and film makers I would let myself be in this situation for.”
Rogen relented, agreeing to reduce Kim’s “flaming hair by 50%,” cut “three out of four of the face embers,” and alter “the color of the head chunks to try to make them less gross.” On Sept. 28, after viewing the new version, Hirai gave his blessing, according to emails, with the understanding that the scene would be removed entirely from any overseas release. Plans to release and promote The Interview moved forward. “I would love working for you and Sony no matter what you decided,” Pascal emailed Hirai, “but I just needed to tell you how important this was for me and the studio Thank you for being an amazing leader and a very cool boss With much gratitude and devotion Amy.”
“From a single injection, we accessed EVERYTHING.”
Looking back, it’s hard to understand how Sony Pictures could have been so ill-prepared for an electronic invasion. It was part of a tech company that sells digital products—films, TV shows, videogames, and music—readily subject to online theft. Angered by Sony Corp.’s (SNE) heavy-handed tactics to protect intellectual property, hackers have long targeted the company’s various divisions. Says cybersecurity guru Bruce Schneier, a fellow at Harvard’s Berkman Center for Internet and Society: “Sony is a company that hackers have loved to hate for 10 years.”
This dates back to the “rootkit scandal” of 2005, when Sony’s music division, seeking to combat piracy, manufactured millions of CDs that surreptitiously installed software on users’ computers that blocked illegal copying—but also spied on their listening habits, slowed their PCs, and created security vulnerabilities. After a tech blogger exposed this, Sony faced state attorneys-general lawsuits, class-action cases, and Federal Trade Commission charges (later settled). The episode outraged consumers—especially privacy-sensitive hackers—who urged a boycott of Sony products.
In the years that followed, the antagonism only grew. In 2011, Sony launched what became known as its “war on hackers.” Citing copyright and computer fraud laws, the company sued a celebrated 21-year-old hacker named George Hotz (a.k.a. “geohot”) for “jailbreaking” his PlayStation 3 console so it could run pirated games and free software, then posting a video showing how to do it. Sony even subpoenaed server logs showing who had visited Hotz’s website. The company sued a second hacker in Germany; police raided his home and seized his computers.
Blowback swiftly followed. In April hackers, declaring this crackdown “wholly unforgiveable,” breached Sony’s PlayStation Network and exposed personal information for 77 million customers and credit card records for 10 million of them. The episode forced Sony to shut down its network for 24 days and cost it $171 million. In congressional testimony, Tim Schaaff, the chief of Sony’s PlayStation Network, used language that was strikingly similar to what Sony Pictures would employ years later: The company, he insisted, had fallen victim to a “highly sophisticated” breach, “unprecedented in its size and scope,” despite “very, very strong” security.
Outside experts disagreed. They concluded that shoddy IT practices, including a failure to install software security updates, left Sony wide open. British regulators later fined the company the equivalent of $396,100 for failing to protect private information, saying the breach “could have been prevented” and calling Sony’s security measures “simply not good enough.” The company blamed the episode on Anonymous, but the group—which has taken responsibility for other hacks—insisted it had been framed; the guilty party was never determined.
Before 2011 was out, various Sony businesses suffered 20 more breaches, making a mockery of the company’s claims of strong defenses. Sony Pictures fell victim in June, when LulzSec, an Anonymous splinter group, broke into its network using a simple technique and revealed personal information for some customers. LulzSec boasted that it had obtained information on a million Sony customers and had invaded just to show how easy it was: “From a single injection, we accessed EVERYTHING. Why do you put such faith in a company that allows itself to become open to these simple attacks?” Sony’s defenses were viewed as so pitiful by Internet bloggers that they inspired a derisive term: “Sownage,” as in “being totally owned like Sony.”
In the aftermath of the PlayStation attack, Kaz Hirai, then presiding over Sony’s gaming and consumer products businesses, formally bowed in apology at a Tokyo press conference. He vowed new measures to toughen cyberdefenses across the company. In September 2011, Sony Corp. also announced the hiring of its first global chief information security officer, former Department of Homeland Security cybersecurity czar Philip Reitinger.
“I will not invest $10 million to avoid a possible $1 million loss.”
It’s not known precisely what new safeguards Sony Pictures implemented in the wake of Hirai’s promises; the company declined to provide examples. But it’s painfully clear whatever steps it took weren’t enough.
The studio’s email system, for example, didn’t employ a fundamental protection called two-factor authentication, which many companies have used for years. This requires anyone logging in to use two forms of identification—for example, a personal password and a one-time password randomly generated on a mobile phone or electronic key-chain fob—making it far harder for hackers to steal a user’s identity.
Lax email practices weren’t new at Sony. In a 2007 article titled “Your Guide to Good-Enough Compliance” in CIO, a trade publication for IT professionals, studio cybersecurity chief Spaltro told the writer how an auditor, citing Sarbanes-Oxley requirements to protect personal info, had warned him that Sony had multiple security weaknesses, including lax password procedures. “If you were a bank, you’d be out of business,” the auditor told him. Spaltro talked the auditor out of noting the deficiency, according to CIO’s phrasing, by arguing “that if his people had to remember those non-intuitive passwords, they’d most likely write them down on sticky notes and post them on their monitors. And how secure would that be?”
Spaltro seemed more afraid of the costs than the risks. “We literally could go broke trying to cover for everything,” he told CIO. “I will not invest $10 million to avoid a possible $1 million loss,” he reasoned. “It’s a valid business decision to accept the risk.” Although Spaltro’s statements were made eight years ago, before Sony became the prominent repeat target of hackers (and he insisted “Sony is over-compliant in many areas”), there’s little sign the company’s attitude changed.
There was a litany of laxity. Sony’s email-retention policy left up to seven years of old messages on servers, unencrypted and ripe for the taking. The company was essentially using email for long-term storage of business records, contracts, and documents saved in case of litigation. When Sony announced plans, in the fall of 2014, to reduce how long emails would be stored to two years—to make the system run better, according to emails, not because of hacking risk—howls of protest erupted at the studio.
An array of sensitive information—including user names and passwords for IT administrators—was kept in unprotected spreadsheets and Word files with names like “Computer Passwords.” Sony’s IT team had difficulty keeping track of all the hardware in its network, which included 30 data centers. In the fall of 2013, while transferring studio security monitoring from an outside vendor to a corporate Sony team, one firewall and 148 routers, switches, and web servers were left unwatched for months, according to a September 2014 PricewaterhouseCoopers audit included among the hacked documents. “As a result, security incidents impacting these network or infrastructure devices may not be detected or resolved timely,” the report noted. Over a 10-month period, according to the audit, the corporate team had alerted the studio to 193 security “incidents.”
In August 2014, Reitinger, the executive who had been hired to harden Sony Corp.’s defenses, quit as global chief information security officer. A Sony spokesperson says his departure was “long planned and not a surprise.” Emails at the time paint a different picture. “Was this a surprise?” Sony Pictures general counsel Weil wrote security chief Spaltro. “Yes,” he replied.
Reitinger, an advocate for stronger security during his time in government, declined requests for an interview. But his many admirers in the field believe he didn’t accomplish more at Sony because he lacked adequate authority (the company disputes this) and because Tokyo didn’t pay enough attention. Even today, they say, it can be hard for an American to wield significant influence in Sony’s Tokyo headquarters. Says cyber-expert Lewis: “He felt a little frustrated.”
“We are just not making enough money.”
Big Sony’s woes just kept getting more dire. In May 2014, after announcing a projected $489 million loss for the fiscal year ending the following March, Hirai disclosed that he and 40 other top executives would again forgo their annual bonuses. (Lynton wrote the CEO, offering to forfeit his own bonus “to show solidarity.” Much to his relief, Hirai declined the offer. “He gave me the bonus, the bet paid off,” Lynton exulted in an email to a friend.) In September 2014 the company revised its projections dramatically downward. The company now expected a $2.1 billion loss.
Lynton was feeling stress. On Oct. 3 he emailed Pascal: “We are virtually a public company and we have made promises to Sony and the street as to what we will deliver for the next three years. I did not want to be in this situation, but events have overtaken us and so here we are. It is therefore very very important that before we take risky or marginal bets in a given slate we have a rock solid foundation to build on … I am about to go next week and make some big promises to Sony and we have to deliver on them … I am only saying all this so you understand the enormous pressure I am under and why I really don’t have much patience at the moment.” Five days later he emailed Pascal from budget sessions with Hirai in New York: “Meeting pretty rough … We are just not making enough money … Too much overhead. Not enough hits.”
The emails suggest Lynton was going through a trying period. “Work is drudgery,” he wrote a friend in September. He told another, “I haven’t read a script in a month … A weird block that I can’t explain.” Lynton seemed more excited about his personal investment projects, which included Snapchat (he and his wife had provided seed money, and Lynton served on the board) and a scheme to develop a Breathalyzer-type device for detecting marijuana use. Emails show he pitched billionaire Dr. Patrick Soon-Shiong on this “very commercial idea” and wanted to patent the concept. (“Michael Lynton spent minimal time on these outside interests,” maintains Sony’s spokesperson.)
Lynton was also wrestling with chronic insomnia and a bad back, along with the latest in a series of tax audits in California. (Sony’s Lawson says there is “absolutely no connection between work pressures and any assumptions you are making based on stolen emails about Michael’s medical conditions or personal tax matters.”) Unbeknown to anyone at Sony, Lynton was also making plans to move with his family back to New York.
On Oct. 21, Sony finally got some good news. After a rise in Sony stock, Loeb had sold his stake at a profit of nearly 20%. He had remained in friendly mode for months, visiting Culver City for a private lunch with Lynton and other executives, dining with Seligman, and sending his hopes that Hirai had enjoyed his summer vacation in Hawaii. Publicly, Sony had treated Loeb as a respected investor who had raised helpful “concerns.” But when he exited, there was unabashed delight. “Champagne for all!!!!” declared CFO David Hendler.
While Lynton was angling to leave, Pascal grew increasingly anxious over slow-moving negotiations for a new contract to stay. Her agreement was set to expire in March 2015. “You know ml will be as rude as possible and try and make me feel AKWARD instead of loved …,” she complained to Hollywood agent Bryan Lourd. “Tell me how to approach ml differently. Read art of war?”
She was already working with studio deputies and “vision” consultants to develop a plan to upgrade the anemic film pipeline. “I guess I [am] mad at him because this is our company and in some level I still think they are interlopers who are destroying it,” she emailed studio president Belgrad on Nov. 12. “Isn’t that silly? But I can tell you one thing. We will be there one way or another when everyone else has gone. I don’t know if that is good or bad but I know it’s true.”
“There weren’t any extreme hurdles in place.”
FBI director James Comey has said he believes Sony’s cyberattackers first breached the studio’s network in September, gaining access through a common tactic called “spear phishing”—duping an employee into clicking on an email attachment or a web link.
Sony’s traditional virus-detection programs provided little protection from the hackers’ malware, since they block only previously identified attacks, and hackers know to make changes that alter the signature of their code. Indeed, it’s now accepted wisdom in the cyberworld that attackers can penetrate the perimeter defenses of almost any company.
What’s critical is detecting the intruders quickly, before they can do much damage. According to a 2015 report by Mandia’s company, it typically takes a company 205 days to discover it has been penetrated, and less than a third of companies detect the breach themselves. This doesn’t mean it’s impossible to stop attacks; it’s evidence that most companies haven’t embraced the right precautions.
Once inside, the Sony attackers’ next step was to “escalate privileges”—to gain wide access by stealing the credentials of system administrators. For more than two months Sony’s hackers roamed freely, identifying what they wanted to steal. This was possible because the studio, with few exceptions, didn’t segregate or provide extra security for even its most precious secrets. In effect, once the invaders made it past the network gates they could go anywhere they wanted because Sony hadn’t locked any doors—much in the way that the company had left its information security department open and unattended.
It’s “astounding” that the Sony hackers were able to remove so much without being noticed, says J. Alex Halderman, a University of Michigan computer science professor. Most corporate networks employ intrusion-detection software, which is designed to sound alarms about unusual file transfers—big files sent to strange places at unusual times—or odd behavior by system users accessing stuff they don’t usually touch. This has fed suspicions that the Sony attackers had inside help providing access to its system—that someone downloaded its secrets onto portable hard drives (as Edward Snowden did at NSA), rather than sent them through the Internet.
Kevin Mandia, the prominent forensic expert Sony hired to investigate the hack, insists there is no evidence of that. The hackers, he contends, escaped detection by patiently moving data out in chunks over several weeks from different company servers to various attacker-controlled locations around the world. As a media company, Sony routinely transfers giant files, making it harder to spot the theft.
Sony would not permit Mandia, chief operating officer of FireEye, to be interviewed by Fortune, allowing him to provide only a brief written statement. Sony has often cited a note Mandia provided to Lynton that asserts that no company could have been “fully prepared for” the attack. But that note was carefully worded. For example, it notes that “industry standard antivirus software” wouldn’t have detected the malware. That’s not saying much. To a cyberexpert, traditional antivirus protection offers the hacking equivalent of being able to repel a musket ball when today’s villains are firing AK-47s.
Indeed, several cybersecurity vendors—including FireEye—claim their products would have prevented (or at least dramatically reduced) the devastation at Sony. Says FireEye spokesman Vitor De Souza: “If they had our solution, we probably would have spotted the malware used in the attack.” De Souza says two-factor authentication also would have made a big difference at Sony. “It creates a big hurdle of the attackers,” he says. “If you don’t have two-factor authentication and they penetrate your network, you’re in big trouble.” If blocked, De Souza acknowledges, hackers might have employed other methods. “If a state actor wants to get in, he’ll get in,” he says. “The question is, How fast do you respond? Instead of, say, taking 10 terabytes of data, they might have gotten one.”
After pilfering Sony documents, the invaders turned to swiping emails for five top studio executives; the most recent messages are dated just two days before the hackers detonated their attack. By that point they had stolen seven sets of credentials for system administrators and mapped the studio’s entire network. This information was “hard-coded” into the destructive malware, allowing it to infect all the computers those IT managers were authorized to touch.
On Monday, Nov. 24, the attackers unleashed their customized wiper malware—igfxtrayex.exe—into Sony’s network. On each machine the malware reached, it deleted everything on the hard drive while installing the threatening web page, with its skeletal imagery and warning. Anyone already logged in helplessly watched their files disappear. The malware also erased the software instructions that tell the computer how to operate. Two hours later the computer would restart to display another chilling message: “Operating system not found.”
To avoid detection the hackers immediately exited Sony’s network after launching their destruction. The malware reported back to “command and control” servers out in cyberspace, allowing the intruders—wherever they were—to tally up their digital toll. Hackers typically use the simplest means necessary to accomplish their mission, and experts say there was nothing particularly sophisticated about the Sony attack. Ed Skoudis, a “white hat” hacker who teaches cyberdefense testing for corporate IT security professionals at the SANS Institute, says the skill level deployed at Sony looks “pretty average.” He puts its perpetrators on par with students in his mid-level classes. “It shows the defenses of Sony were not particularly good,” says Skoudis. “I didn’t see the bad guys jumping over any extreme hurdles, because there weren’t any extreme hurdles in place.”
What was extreme was the destruction.
A version of this article appears in the July 1, 2015 issue of Fortune magazine.