Prakash Kota, chief information officer at business software firm Autodesk, was fortunate enough to experience an uneventful flight home on Thursday evening to the San Francisco Bay Area after attending a leadership team meeting in Montreal.
By Friday morning, Kota and others awoke to the largest IT outage in history. A software update pushed out by cybersecurity company CrowdStrike caused millions of Windows-based computer systems to crash, upending air travel, banking, retail transactions, hospitals, and railways across the globe. Many CIOs are still dealing with the aftermath.
And while nearly all of Autodesk’s employees were back online by Friday morning, Kota says the episode shows that IT leaders must create more protections in an era when software on corporate devices is often updated by external partners. “I would say it almost gives a wakeup call to some of these vendors that want to be agile, but certain things have to be tested,” Kota said.
One change Kota is strongly considering at Autodesk is more oversight of automatic software updates from vendors before they’re accepted. “Is there a way where we can restrict some of the changes before they get deployed broadly?” Kota asks.
At cloud company Akamai Technologies, the CrowdStrike outage had no direct impact on operations due to a prior decision to prevent vendors from pushing through automatic updates. Instead, the company’s IT department must approve them before they’re downloaded.
“That’s a lesson: Have faith in your providers, but you can’t trust them wholesale,” says Akamai CIO Kate Prouty. “You need to do your own testing.”
While unscathed this time around, Akamai learned a few lessons from the CrowdStrike debacle. It realized that the existing encryption on the company’s devices adds an extra layer of complexity if and when staff needs to decrypt those machines remotely to resolve a problem. Akamai is now considering automating the process to get those devices back online quickly following a mass outage rather than having to unlock them one at a time.
Prouty said she’s also thinking through how to ensure employee communications if the company’s internal messaging system became inaccessible due to a tech disruption.
Peter Mattis, chief technology officer at database startup Cockroach Labs, says CrowdStrike isn’t solely responsible for the outages that impacted so many businesses. Its customers, he argues, also deserve some blame. “Why weren’t they mandating that they had more control over what’s being deployed to their critical infrastructure? They are essentially turning it over into the hands of this vendor,” Mattis says.
When companies sign deals with new vendors, they often require those companies to complete a questionnaire to attest to the security of their systems. Some of these questions focus on “system resiliency,” details that would reveal how CrowdStrike and other vendors think through data protection, disaster recovery, business continuity planning, and how they stage software updates.
“I’m already asking our procurement people to do a little scrutiny to [determine] should we be asking more incisive questions about their resiliency,” says Mattis.
Tom Parker, CTO of security company NetSPI, says the outage exposed significant industrywide “gaps in our ability to react and respond” to CrowdStrike-like threats. But he remains a fan of CrowdStrike and the security industry as a whole. “There’s definitely a tendency to have a knee-jerk reaction,” says Parker.
CrowdStrike customers should perform a deep analysis of what happened inside their companies during the crisis, he adds, and perform tabletop scenarios, or simulated IT emergencies that help train employees and expose weaknesses.
At CNH, a manufacturer of agriculture and construction equipment, 8,500 employees were confronted with the “blue screen of death” on Friday that made their devices unusable. By mid-day Saturday, 100 IT professionals were able to get some operations up and running, and after 72 hours, the company was fully operational.
Marc Kermisch, CNH’s chief digital and information officer, says his optimistic view of the outage is that it gave many companies an opportunity to put their disaster recovery plans to work.
“We really got a chance to exercise that and it was a great learning moment,” he says. And while relieved CNH had a plan to execute against, he adds, “I hope to never have to do that one again.”
John Kell
Send thoughts or suggestions to CIO Intelligence here.
NEWS PACKETS
CrowdStrike outage ripple effects continue. After the largest IT outage in history that impacted around 8.5 million Microsoft Windows devices, CrowdStrike has warned of a hacking threat as the outage continues. A few other key news stories from the past several days include: CrowdStrike’s stock lost nearly one-fourth of its value over two trading days following the outage; CEO George Kurtz has faced some criticism for his initial response, lack of apology, and is being called to testify before Congress; and Delta Air Lines, in particular, continues to struggle with flight cancellations and that it has caught the attention of the federal government. Meanwhile, cyber experts say there are lessons to be learned from the experience, even though the outage wasn’t the result of a hack.
Wiz rejects Google’s $23 billion takeover; will opt for IPO. Cybersecurity startup Wiz said a multibillion-dollar deal with Google that would have been the search giant’s largest acquisition ever was off and that Wiz would instead pursue an initial public offering. Speaking at the Fortune Brainstorm Tech conference last week, CEO Assaf Rappaport said that the cybersecurity industry was ripe for consolidation. But still, he sees IPOs and acquisitions as “milestones” in a longer journey. “The market validation we have experienced following this news only reinforces our goal – creating a platform that both security and development teams love,” Rappaport wrote in the note to employees Monday, reported by Fortune.
OpenAI releases a smaller and cheaper version of ChatGPT. OpenAI has rolled out a new version of its AI model, called GPT-4o mini, that the Wall Street Journal reports is 60% cheaper to use than the model that powered ChatGPT until recently, called GPT-3.5 turbo. It's a pivot that aligns with the evolving thinking of many CIOs: Not all generative AI models need to be so big and so powerful. Smaller models can be more useful for clients that only need AI for select tasks. Google and some AI startups including Anthropic and Mistral have also released smaller models this year.
Cohere raises $500 million as race against OpenAI, Anthropic heats up. Cohere is now one of the most valuable AI startups after a new funding round that values the Toronto-based company at $5.5 billion. With backing from Canadian pension investment manager PSP Investments, along with new investors including Cisco Systems and AMD Ventures (AMD is a sponsor of this newsletter), Cohere is angling to make software to help companies run more efficiently, differing from OpenAI’s mission to build AI that’s capable of performing as well—or better—than humans. A day later, Cohere laid off 20 employees, roughly 5% of the company's workforce, Fortune reports, another indication of the challenges tech firms face as the industry pushes further into generative AI.
ADOPTION CURVE
A cybersecurity disconnect. A pair of studies published by IT software provider Ivanti may indicate that leadership is overconfident in their company’s ability to prevent or stop a cybersecurity incident. Two surveys, conducted by Ivanti in late 2023 and earlier this year of 3,059 leaders and IT professionals, found that 60% of non-IT leaders are “very” or “extremely confident” about cyber safeguards.
But just 46% of IT professionals share the same level of confidence, and Ivanti CIO Robert Grazioli says that with so many serious security events in the news recently, chief information security officers “don’t have to fight too hard” to convince the C-suite about the reputational risks. But what may be harder is converting awareness into strategic buy-in—and getting the budget to match. The latter has been a particularly tough challenge of late as studies consistently highlight spending on cybersecurity has flattened or even declined for most companies in 2024.
JOBS RADAR
Hiring:
- Golin is seeking a VP of AI Technology, based in Chicago. Posted salary range: $150K-$175K/year.
- AlphaSense is seeking a VP of IT and Security, based in New York City. Posted salary range: $250K-$299K/year.
- Bloomberg is seeking a product coach for the CTO Office, based in New York City. Posted salary range: $145K-$210K/year.
Hired:
- Renault Group named Philippe Krief as CTO, taking over the role from Gilles Le Borgne, who has been appointed to a strategic advisory role to the CEO. As CTO at the French automaker, Krief will report to CEO Luca de Meo and manage all engineering activities and resources. He previously served as an executive at Ferrari and Maserati.
- Reed Smith announced that Ryan McEnroe, the law firm’s former global director of IT operations and customer services, has ascended to the role of CIO. He will oversee IT across the firm’s more than 30 offices in the U.S., Europe, the Middle East, and Asia.
- Atlas Van Lines appointed Ryan Parmenter as CIO to oversee IT for Atlas and its 10 subsidiaries. Parmenter has 23 years of IT experience with the moving services provider and most recently served as VP of IT.
- Parts Town Unlimited has named Jamie Head as CIO, bringing over 20 years of experience, including most recently as chief digital and technology officer for cooperative Ocean Spray Cranberries. Prior to that, he held leadership roles at pharmaceutical company GSK and food producer Mars.
- Serco appointed Tom Read as chief digital and technology officer and in this role, he will lead the outsourcing provider’s digital and cybersecurity strategies, IT, infrastructure, and the adoption and scaling of new technologies.
- Cambridge Savings Bank announced that CIO Kevin McGuire had been appointed to the role of chief operating officer and that Christopher Johnson would succeed him as CIO. Johnson will report directly to McGuire and oversee all aspects of the Massachusetts-based mutual bank’s technology, including infrastructure, security, and applications.
- Crossroads Treatment Centers named Frank Elston as CIO, responsible for the company’s IT strategy and helping improve patient care through tech initiatives. Prior to joining Crossroads, Elston was VP of IT at mental health services company Health Connect America.
- Dynasty Financial Partners appointed Leslie Dentinger Norman as CTO, ascending to the role after previously serving as deputy CTO. Prior to joining Dynasty, Norman was a leader in technology product development at financial services firm Raymond James.