Ronak Singhal, a senior executive and 20-year veteran of chipmaker Intel, was trying to get to dinner at Helena, his favorite restaurant in Israel, a few weeks ago. But before he could join colleagues celebrating a promotion at the high-end eatery poised on the shores of the Mediterranean Sea south of Haifa, he had to explain to one of the company’s software partners what was going on with Intel’s patches for the notorious Spectre and Meltdown security problems.
The problem that night for Singhal, who oversees the development of the architecture for all of Intel’s processors, was that something was wrong with the patches. Among all the millions and millions of computers in use around the world running Intel CPUs, one of the patches for Spectre was causing some computers to freeze up or spontaneously reboot. Though only affecting a tiny proportion of the market, the problems were widespread enough to spook PC makers and prompt a temporary recall of the updated software. (And even stirred Linux creator Linus Torvalds to publicly proclaim Intel’s work was “pure garbage.”)
Relying on some techniques that Intel had never used previously in its software, “there were cases where the patches didn’t work as intended,” Singhal explained. It took more than an hour to assuage the contractor—Singhal’s co-workers started eating without him. “They thought I’d gotten lost or kidnapped or something,” he jokes recalling the incident. He did get to join the party and eat a dish of Helena’s famed calamari.
A few weeks later, Intel issued corrected patches and the fixes for one of the most serious security incidents in computing history have gone smoothly since then. On Thursday, Intel declared that it had fully deployed patches covering all of the chips it had made in the past five years.
Get Data Sheet, Fortune’s technology newsletter.
Up next for Singhal are fixes that will be embedded directly in the silicon of upcoming products. The revamped chip designs will be ready for 8th generation Core processors released in the second half of the year and a line of Xeon server chips expected in the fourth quarter known by the code name “Cascade Lake.” Building the protections into the hardware eliminates a significant amount of the impact on performance seen with the software patches, Singhal says.
“We’ve made it through the first set of software mitigations,” Intel CEO Brian Krzanich tells Fortune. “We’ve got everything five years and newer completed and we’re now starting to implement hardware mitigations where it’s actually built into our silicon.”
Spectre and Meltdown Variants 1, 2, and 3
The whole mess that revealed such serious security vulnerabilities in nearly every chip made for the past few decades, by Intel and its competitors, started small last summer. Researchers at a special security vulnerability search team at Google reported to Intel’s security section in June that they’d uncovered a problem with a key part of CPU design.
Modern chips typically have so much idle processing power that it makes sense for programs to calculate several options to solve a problem even before earlier steps in the program have completed. Known as speculative execution, the performance enhancing strategy then throws out the answers that don’t match the results of the earlier steps.
But the Google (GOOGL) researchers, followed by several teams in academia, had found ways to trick chips into revealing data like passwords and encryption keys as the secrets were used in the speculative execution calculations. The researchers dubbed two variants of the trick Spectre, after the fictitious evil organization that pursues James Bond, and a third variant was called Meltdown because it effectively melted security barriers. The danger was especially acute for cloud servers, where programs from multiple customers would be running on the same chip, and in web browsers, which can execute code from a web site unknowingly.
By early July, Intel and other chipmakers had realized the vast scope of the problem and convened groups to craft solutions. Singhal held a daily morning conference call, sometimes lasting for two hours, to coordinate Intel’s response across offices in Oregon, California, Texas, and Israel. With people in different time zones working on the problem, the effort could operate around the clock.
All along, the plan was to issue software fixes first and then build the protections into future chip designs. The software patches had a cost in reducing the performance of the affected CPUs. The hit varied widely depending on the type of Intel chip involved and the programs being run. One test on a PC with a Kaby Lake Core i7 processor found most apps slowed less than 10%, which would be barely noticeable in real life usage. But Microsoft (MSFT) warned that PCs running its older Windows 7 or 8 and Intel’s five-year-old Haswell processors would take a big hit.
Intel’s New Security Effort
As a result of the experience, Intel CEO Krzanich set up a new group, dubbed the IPAS or Intel Product Assurance and Security, to not only work on the Spectre and Meltdown fixes but to address future security problems more effectively. Longtime Intel executive Leslie Culbertson, who joined the company in 1979, heads the IPAS group.
“This was going to be a whole new area of research and a whole new area of security understanding that required a long-term investment by Intel,” Krzanich says. The focus will be on uncovering future vulnerabilities, but also thinking about how to make its chips more secure in general. “You’re going to see a constant progression–that’s what this team will be thinking about.”
“We know this isn’t the end of the story,” Singhal adds. “This is going to be an ongoing activity probably for many of us.”
When news of Spectre and Meltdown first leaked out in early January, Intel’s (INTC) stock took a hit, as investors feared the security problems might slow chip sales. More recently, some analysts have argued that Intel’s new chips with built-in protection might spur more rapid sales from companies wanting to upgrade to safer hardware. Intel’s shares are up 12% so far this year, outpacing the 3% gain in the S&P 500 Index.
Krzanich is dismissive of both the positive and negative scenarios. “We’ve said since the beginning of this that we think the impact will be negligible, even on the positive side,” the CEO says. “The analyst community needs to realize that we’re constantly doing these kinds of improvements—improvements in security, improvements in performance, and adding new features to drive refresh cycles.”
(Update: This story was updated on March 15 to clarify that the impact on performance from Intel’s hardware fixes would be “a significant amount.”)
