Hello and welcome to Eye on AI. In this edition:
- AI’s X-risk breaks into the mainstream
- Anthropic CEO Dario Amodei calls for a coordinated industry safety effort
- Anthropic details attempts to misuse its AI models
- China’s top spy warns AI could pose a risk to the Communist Party
- OpenAI is violating California’s new AI safety law, watch dog group says.
- Half of companies aren’t following their own AI governance policies, E&Y survey says.
In the past few days, I’ve heard a lot of people repeating that old saw—often wrongly attributed to Vladimir Lenin—about there being “weeks when decades happen.” It certainly seemed to be one of those weeks in AI. Concern about existential risk has been a strain of AI discourse for decades. But, despite occasionally making headlines when someone like Elon Musk, Sam Altman, or Geoffrey Hinton would express their fears about AI posing a grave risk to the species, it never really cemented itself in the general public’s consciousness in the way, say, climate change, or the risk of nuclear war, has. If politicians debated AI regulation at all, the discussions centered around data center construction and utility bills, jobs, education, mental health, algorithmic discrimination, and civil liberties, not the risk of rogue AI killing people—maybe even all the people. Until now, that is.
The drumbeat of dire warnings from employees resigning from—or in some cases still working for—Anthropic, OpenAI, and Google DeepMind, all saying that the leading AI companies are developing the technology recklessly and risking human extinction, has dominated the global news cycle for an entire week (which is really saying something in this day and age.) AI company CEOs and politicians have been stirred to respond. After years in which both domestic AI regulation and efforts at some kind of international AI governance regime had mostly stalled, suddenly the air is electric with possibility.
My Fortune colleague Nick Lichtenberg had a good story on why the resignation jeremiad of former Anthropic and OpenAI safety researcher Jacob Coxon had such impact when previous warnings, often from much higher-profile individuals, did not. The short answer is that coverage of the Hugging Face incident and other “rogue AI” episodes as well as people’s own experiences using AI agents seems to have opened the Overton window on discussing “loss of control” dangers. The timing, with Anthropic on the verge of an IPO and OpenAI edging closer to one too, also no doubt played a role.
The question now is what happens next? Fortune editor-in-chief Alyson Shontell sat down with Altman on Friday to ask him those questions for her “Fortune 500: Titans & Disruptors of Industry” vodcast (we just call it “Titans” for short.) Altman said the company was in favor of coordinating an industry-wide slowdown in the pace of AI development with bitter rivals, including Anthropic and SpaceX, as well as Google DeepMind, Meta, and perhaps others. (Altman famously doesn’t get along with either Anthropic CEO Dario Amodei or SpaceX’s Elon Musk.) He hinted that such discussions were already underway and that a coordinated slowdown might be announced soon. He also said that, if necessary, he would have no problem telling investors that OpenAI had taken actions to prioritize safety that had cost them financially—and that OpenAI’s investors were warned of this possibility going in. He also definitively said OpenAI would not go public this year, in part due to the current concerns about the safety of the latest AI models, but also, he hinted, because OpenAI’s business isn’t yet in the right place. You can check out the full vodcast episode here. It’s well worth your time to watch.
A coordinated slowdown?
After Alyson’s interview, Amodei put out a blog post also calling for a coordinated slowdown or pause among frontier labs in democratic countries. He said that in some cases, though, coordinating with other AI labs would require an antitrust exemption from the government. He also said that Anthropic would appoint independent evaluators to be permanently on-site at its offices to review its safety work. (He mentioned the nonprofit AI evaluation company METR as his preferred partner for this.) He also said that the U.S. and other democracies should try to strike some kind of international AI governance agreement with China and authoritarian states, if possible. Altman quickly came out and endorsed most of what Amodei said—in particular saying that OpenAI would also embed outside evaluators alongside its research teams—although he was careful to note that “pacing does not mean stopping.”
In the wake of Coxon’s warnings and Amodei’s call to action, a number of U.S. lawmakers introduced legislation or renewed efforts to push forward existing bills. Some, such as a bill introduced by Vermont independent Sen. Bernie Sanders, call for an outright ban on the development of “artificial superintelligence” and mandate that U.S. AI companies pause current research until safety techniques improve. Others, such as a bipartisan bill from Republican Sen. Ted Cruz, Senate Majority Leader John Thune, and Democratic Sen. Amy Klobuchar, would impose a duty on AI companies to prevent catastrophic harms. There were also calls for Congressional oversight hearings on AI’s catastrophic risks. Former President Barack Obama urged Democrats to put AI governance at the center of their legislative and campaign agenda. Meanwhile, a group of 70 U.K. parliamentarians signed an open letter calling for the British government to ban the creation of artificial superintelligence and work on an international AI treaty.
Trump pushes back
But there was strong pushback from some of the politicians that matter the most. President Trump posted to his Truth Social platform that the only guardrails AI needed “is a STRONG AND SMART (High IQ!) PRESIDENT, and the U.S.A. has that in spades!” He criticized Amodei by name, accusing him of “pretending to be a ‘perfect little angel’” and said his administration had already stopped Anthropic from “doing bad, or potentially bad, ‘things.’” He said the U.S. already had regulatory power and criminal laws that applied to AI companies and that there was “a SICK conspiracy going on against AI and Data Centers, and the only one that is happy about it is China.” He made similar comments in a phone call to Nvidia CEO Jensen Huang that Huang, with Trump’s permission, broadcast to a live audience at an “All in Podcast” summit. This was followed up by the Republican Speaker of the House, Rep. Mike Johnson, saying that fear of AI was drummed up by the media and that “we’re not going to take stupid, knee-jerk reaction prescriptions on this.” Not to be outdone, Chinese state media also criticized Amodei’s proposals, saying they were “self-serving” and “Cold War tactics” designed to hobble China’s technological and economic rise.
With all of that, it seems the prospects for some kind of executive order mandating improved AI safety are poor. The same goes for any actual legislation—at least until after the November midterms. Three points though that have come up in the discussion that are worth addressing.
Are antitrust concerns legit?
One is the debate about whether AI companies need an antitrust waiver to talk to one another about slowing development. Some, such as former Trump administration AI and crypto czar David Sacks, have said the AI companies don’t need such a waiver to coordinate a slowdown. And I agree that we should not grant a broad waiver to these tech giants. But I do think that there are legitimate concerns from the AI companies that any discussion of a pause—or of a coordinated decision not to undertake certain product innovations—could create antitrust issues.
Currently, each new generation of AI models tends to drive down the cost of existing, older models. So limiting the rollout of newer models would potentially keep prices higher for longer for consumers, which would seem to open the AI firms up to antitrust claims. (Matt Levine at Bloomberg had a good column on this.) Also, some of the specific innovations that worry AI safety experts, such as greater use of looped Transformers, also happen to have the benefit of using fewer tokens than forcing a model to spit out its complete reasoning trace in its “chain of thought.” This too has the effect of potentially lowering costs for consumers. So prohibiting this technique on safety grounds would also tend to result in higher prices. Again, that looks problematic from an antitrust perspective. For what it’s worth, Chris Lehane, OpenAI’s chief global affairs officer, has come out and said OpenAI doesn’t think it needs an antitrust waiver to discuss shared safety standards with the other AI companies. He also said that there have already been discussions with Anthropic and Google DeepMind on such standards. But the issue may be that these standards are voluntary, with no mechanism to compel compliance if one company cheats on its commitments. Enforcing the standards would presumably require government action.
Is product liability law enough?
In an example of the strange bedfellows this issue has created, Sacks and former Biden administration FTC head Lina Khan have both said that existing product liability laws could be used to prevent AI companies from releasing unsafe products. Sacks in particular has said that these laws are the reason no new government agency is needed to police AI companies. But there are two problems here. One is that product liability laws generally only apply to products that are sold to customers. Some of the biggest concerns with AI risks lately—as was the case in the Hugging Face incident—have involved unreleased, internal models that were undergoing development or were only deployed inside the AI companies themselves. Product liability law would not cover these internal models.
What’s more, while the fear of liability lawsuits might deter unsafe behavior by AI companies, it might not—and if it doesn’t, suing the companies after the fact is not ideal. This is especially true if the risks are actually existential ones, such as engineering a bioweapon. Suing won’t help us if we’re dead. But even if the risks are merely bad—like hacking into a single financial institution or hospital, manipulating the stock market, or taking out an electrical grid—suing a company after the fact won’t really provide the outcome society wants. Better to prevent these things from happening in the first place. That’s why we do have agencies that police systemically important financial institutions, regulate air travel, ensure power that grids adhere to certain standards, etc.
What about ‘regulatory capture?’
Finally, Sacks and others, including some on the more libertarian left as well as some of the CEOs of AI companies that are slightly behind the frontier, have attacked the proposal for a coordinated pause and agreement on safety standards as an attempt at “regulatory capture.” The claim is that these companies will write the rules in such a way that their leadership position at the front of the AI race gets locked in. I am not denying that this could happen. But it also seems that there are ways to prevent this from happening. Accelerationists like Sacks act as if all regulation results in regulatory capture. But, as I mentioned in a previous newsletter, UC Berkeley AI researcher Stuart Russell likes to quip that there are more mandatory requirements for sandwich shops in San Francisco than there are on OpenAI or Anthropic. And you don’t see too many restaurateurs complaining about regulatory capture. It is simply not the case that mandatory safety rules always result in regulatory capture.
I would also argue that a certain degree of regulatory friction that happens to privilege incumbent players is a price worth paying for a safe industry in cases where failure poses significant risks to human life or physical and financial health. In fact, the industries that pose the greatest potential risks of mass casualty events tend to have fewer players in them, and yes, the burden of regulatory compliance is one of the reasons. But I think this is a tradeoff the public actually thinks is worth the fact that it may also mean they pay slightly more for certain things. There are only a handful of companies around the world that design and build nuclear power plants, for example; only a handful that make commercial aircraft, too. But these also happen to be some of the safest industries out there in terms of their actual operational records. Would there be more players in these industries if there were fewer government safety rules and inspection regimes? Almost certainly. But is the public screaming about regulatory capture and asking for safety standards on nuclear power plants and aircraft to be relaxed?
With that, here’s more AI news.
Jeremy Kahn
jeremy.kahn@fortune.com
@jeremyakahn
Before we get to the news, just a reminder to check out this week’s episode of our new vodcast Fortune AI Weekly. This week, Bea Nolan and I talk to Substack cofounder and CEO Chris Best about his decision to add an AI writing detection feature to the platform. We also talk about AI doomerism going mainstream and the controversy over OpenAI’s Navier-Stokes mathematical breakthrough. You can check out the vod here on YouTube.
Also, come join me at the Fortune AIQ Summit at the New York Stock Exchange on October 1! We’ll join C-suite leaders Bank of America, Booking Holdings, Citi, Ecolab, Elevance Health, United Healthcare, S&P Global, and more to hear about how they are using AI to deliver the growth, innovation, and transformation that is putting them at the top of their respective industries. It promises to be an afternoon of eye-opening insights and inspiration. You can register to attend here.
FORTUNE ON AI
OpenAI may have violated California’s AI safety law with latest model releases, AI watchdog says—by Beatrice Nolan and Emily Forlini
Anthropic CEO calls to slow the race toward AI ‘superintelligence,’ and grants outside evaluators permanent access—by Beatrice Nolan
After ‘tense negotiations’ with OpenAI, California enacts law named for teen who consulted ChatGPT before suicide. It could become a national standard—by Emily Forlini
Google wants Spirit Airlines’ data. Micro1 wants to pay more—by Wen Shao
An ex-Anthropic researcher claims AI could kill us all by 2030. But he fails to answer the most essential question: What are we supposed to do about it?—by Emily Forlini
Exclusive: Manufacturing AI startup CADDi valued at $1.2 billion following $114 million Series D funding round—by Jeremy Kahn
AI IN THE NEWS
China’s spy chief warns of AI risks. Chen Yixin, who heads China’s Ministry of State Security, warned that AI could threaten Communist Party rule by enabling deepfake propaganda, cyberattacks, sensitive-data leaks and more sophisticated military operations. He made the warning in an article in a state-run cybersecurity magazine and called for tighter party control and government oversight of the technology. Chen singled out foreign AI models as security threats and warned they could be used for large-scale espionage or attacks on critical infrastructure. The remarks come ahead of a meeting between Chinese President Xi Jinping and U.S. President Donald Trump in Washington next week that is expected to include talks about AI. Read more from the New York Times here.
Anthropic threat report details efforts to use its AI to run propaganda campaigns and build weapons, including possible bioweapons. Anthropic says criminals, state-sponsored groups, spyware vendors and researchers have used its AI models for malicious activities including cyberattacks, surveillance, propaganda and the development of conventional weapons, and, in at five cases, possible bioweapons research. One of those cases involved a scientist using Claude to help prepare a state-sponsored grant application ways to make the deadly chikungunya virus more potent and transmissible at a military research institute. The company said it banned the accounts involved. The company also detailed efforts by the Houthi rebels in Yemen to use Claude to build and test ballistic missiles. Anthropic warned that such real-world misuse could grow as AI models become more capable and called for AI companies and governments to work together on stronger safeguards and defenses. Read more here in the Guardian.
Anthropic accuses Chinese AI labs of large distillation, secretly routing questions from their users to Claude. Anthropic, in the same threat report, also accused China-based AI labs including Alibaba, Moonshot AI and DeepSeek of secretly using Claude outputs to train their own models through large-scale “distillation,” with the campaigns involving nearly 200 million exchanges in total. Alibaba alone accounted for more than 151 million Claude interactions, while Anthropic said Moonshot and DeepSeek sometimes routed their own users’ queries to Claude without their knowledge, potentially exposing sensitive information. In some cases, Anthropic said its models received prompts and data that seemed to come from Chinese government and national security entities. You can read more here from CNBC.
Anthropic tells investors it was profitable for a second quarter. Anthropic has told investors it expects to post positive adjusted operating income for a second consecutive quarter, strengthening its financial story ahead of a planned Nasdaq IPO that could value the Claude maker at $2 trillion or more, the Financial Times reports, citing unnamed sources that have seen the company’s financials. The company reported $11.5 billion in second-quarter revenue, up 14-fold year over year, and reached a $65 billion annualized revenue run rate by the end of July, with investors forecasting that could hit $120 billion by year-end.
Nvidia, Palantir, and Booz Allen drop OpenAI, Anthropic over data concerns. Palantir, Nvidia and Booz Allen Hamilton are restricting their employees use of frontier AI models or threatening to drop models from Anthropic and OpenAI completely over concerns that the AI companies could retain or derive value from sensitive corporate data. The pushback intensified after Anthropic introduced a 30-day data-retention policy for its Fable 5 model, prompting the company to offer a system that lets enterprise customers store activity data in their own cloud infrastructure under their own encryption keys. The dispute reflects growing demand from enterprises for greater control over proprietary data. Read more from Reuters here.|
Top mathematicians protest AI companies approach to unsolved math challenges. Twenty-four past winners of mathematics highest honor, the Fields Medal, have warned that AI’s growing ability to solve frontier mathematical problems—including OpenAI’s claimed solution to the Navier-Stokes problem (see last week’s “Eye on AI”)—could undermine mathematics by prioritizing answers over the conceptual understanding and insights produced through human problem-solving. They argue that struggling toward proofs often generates new questions, methods and fields of research, benefits that could disappear if AI simply produces proofs that humans struggle to understand. Read more from the Economist here.
EYE ON AI RESEARCH
As companies adopt AI agents, governance lags. That is the finding from a new survey of U.S. senior executives by Ernst & Young. Almost half (47%) reported that in the rush to deploy AI agents, their company has sometimes not adhered to established internal AI governance procedures. About half (49%) also said that their organization’s existing governance frameworks had not been updated to take into account specific risks and requirements related to AI agents, even as 85% of those surveyed said their companies deployed AI agents in at least a few cases where they acted without real-time human oversight. Even more troubling, about a quarter (26%) of those executives who said their organization has AI agents in production deployments also said they lacked systems to reliably detect these agents operating in an unauthorized manner internally.
The survey found that when governance systems and assurance reviews were conducted appropriately, they did catch issues, with many organizations having to pause or stop AI deployments to correct errors or policy violations. “The biggest agentic AI risk is that human oversight hasn’t evolved accordingly,” said John McLain, EY Americas Assurance Technology Risk AI Leader and EY Americas Assurance AI Deputy Leader. You can read more and download the full report here.
AI CALENDAR
|
Oct. 1: Fortune AIQ conference, New York. Apply here to attend.
Oct. 2-4: The Curve, Berkeley, Calif.
Dec. 6-12: Neural Information Processing Systems (Neurips) conference. Sydney, Australia.
|
BRAIN FOOD
What does it take to build a successful AI hub? That’s what a lot of cities have been asking lately. Many would like to capture the startup creation and jobs growth that San Francisco has experienced in the past few years. In the U.K., where I live, there’s now a growing and vibrant hub, mostly around Kings Cross, where Google DeepMind is based and where Anthropic and OpenAI have also taken office space. Lately a crop of startups, many of them founded by DeepMind alums, have sprouted up there too.
Interestingly, Oxford, England, where I live, has not had quite the same success in building an AI startup hub, despite having a world class university that churns out a lot of AI researchers and engineers and roboticists. I recently moderated a breakfast discussion at Oxford North, a new “innovation district” that would like to be hub for tech-forward companies, including AI startups, on the topic of what the city would need to be more like London, or, dare I say it, Cambridge.
Conducted under Chatham House rules, I can’t quote any of the participants by name. But the consensus was that the city has a lot going for it: world class research; access to talent, especially for early career hires; a great global brand; and easy access to London’s venture capital scene. But several ingredients are still missing:
- A more startup-friendly spin-out policy from Oxford University (which takes 10% equity for software spin-outs and a whopping 20% for robotics companies; compare that to the 5% that is more typical of U.S. universities) as well as a much faster pace from the university offices in charge of technology transfer and startup incubation.
•A more entrepreneurial and innovation-minded attitude towards working with startups from across university departments.
•An easier way for investors and corporate partners to learn about and navigate the city’s startup scene. “We need a front door to the ecosystem,” one participant said.
•More presence in the city from larger, established technology companies that can lend partnership, mentorship, and networking opportunities to the hub.
•More networking events and community-building opportunities for those working on AI-related companies.
What do you think it takes to build a successful AI startup hub? Let me know your thoughts.