DeepSeek has tilted the balance towards open source AI, but big security issues remain

yes, we are discussing. i personally think we have been on the wrong side of history here and need to figure out a different open source strategy; not everyone at openai shares this view, and it’s also not our current highest priority.

That’s a big deal coming from the head of a lab that, while originally founded on the principle of open-sourcing all its research and models, has for the past several years been fully invested in a proprietary AI model, where users can only interact with its software through an application programming interface (API).

It’s important to remember that OpenAI adopted a proprietary model for two reasons. The first was commercial—it’s a lot harder to make money from IP you give away, and much harder to maintain any kind of technological advantage over competitors if you let everyone reverse engineer your software. But the second, critically, was about AI Safety. It is a lot harder to prevent people from taking an open source model and using it for harmful purposes. There’s been a lot of discussion in AI policy circles about the critical importance of “securing model weights” as a way to both ensure the U.S. maintains a technological edge over potential adversaries, such as China, and as a way to prevent rogue actors from using AI in dangerous ways.

The business case for open source

The popularity of DeepSeek’s R1 model has altered both aspects of this calculus in some critical ways. First, the marketplace seems to be increasingly voting for open source. Throughout much of 2023, when proprietary models from OpenAI, Anthropic, and Google, were clearly superior in performance, many CTOs and CIOs thought that the only way they could engineer AI systems to achieve business goals was to pay for the more capable proprietary systems.

Since then, however, the free, open weight models have come closer and closer to the performance of the proprietary ones. This was true with Meta’s Llama’s 3 series models, which approach GPT-4o in performance. Now DeepSeek has shown it is also true for the new reasoning models—which are designed to do better at logic problems, math, and coding and which also provide better answers the more time they are given to “think” about a problem. DeepSeek’s R1 comes close to the performance of OpenAI’s o1.

The other thing that has happened is that many companies have discovered that the best way to accomplish business tasks with today’s generative AI systems is to use many models in concert. Engineering this kind of workflow with proprietary models can quickly get expensive. The open source models offer a way for businesses to have more control at potentially lower cost. (Although with open source models, a company still must pay to have the model hosted somewhere.)

Given the narrowing gap in performance between proprietary and open source models and the preference of many companies for open source, it will be difficult for OpenAI, Anthropic, and Google, to hold on to market share from selling foundation models. They may be forced to move “up the stack,” offering their models for free, but then selling ready-made applications and tooling built on top of the models. (This is the traditional business model for open source software providers.) Kevin Weill, OpenAI’s chief product officer, said in the same Reddit AMA in which Altman made his “wrong side of history” remark that OpenAI might open source some of its older models, although whether that would do anything to address the business challenge open source presents OpenAI remains unclear.

The security case against open source

When it comes to safety and security, the growing popularity of powerful open source models also present a dilemma. Open weight models remain fundamentally less secure than proprietary ones. Researchers from the University of Pennsylvania and Cisco published a study last week in which they prodded DeepSeek’s chatbot using 50 common jailbreaking techniques—using prompts that try to trick a model into overcoming its guardrails and outputting potentially harmful content. It failed every single one of them, according to the study. That means all those companies rushing to put DeepSeek into their systems, may be unwittingly incorporating some big security vulnerabilities. It also means that bad actors may be able to easily use DeepSeek’s models to help them create malware and cyberattacks, run phishing scams, or even perhaps, plot a terrorist attack. That’s why more money and more effort should be put into figuring out ways to defend open source models. (Anthropic may have hit upon one idea. See more in the Eye on AI Research section below.)

DeepSeek also shows that it is difficult to prevent competing nations from acquiring cutting edge, or near cutting edge AI capabilities. The question then becomes what to do about it. Some critics of U.S. export controls have argued that DeepSeek’s ability to create models as capable as its V3 LLM and R1, despite having access to fewer cutting edge computer chips due to U.S. export controls, show the export controls don’t work. On the contrary, others, such as former OpenAI policy researcher Miles Brundage, argue that export controls are more necessary than ever. His argument is that while export controls may not be totally effective, people would still rather have more powerful AI chips than not, both for developing leading models, and, critically, for running them. So the restrictions should still make it more difficult for China than if they had access to all the Nvidia’s GPUs they might want.

Squaring these security concerns with the business momentum behind open source models will be difficult. Altman may feel OpenAI’s been “on the wrong side of history”—but then again, is he willing to risk recklessly helping to bring about the end of history just to burnish his company’s popularity with developers?

With that, here’s more AI news.

Jeremy Kahn
jeremy.kahn@fortune.com
@jeremyakahn

AI IN THE NEWS

As mentioned above, OpenAI made so many announcements, we’ll try to round them up in a single long-ish news item here: First, the company debuted o3-mini, a faster version of its most powerful reasoning model o3, tailored specifically for math, coding, and logic use cases. What’s more, it made o3-mini available to all ChatGPT users, including those who use the free tier of the chatbot. In addition, the company launched a product called Deep Research, which uses its o3 model as an agent that can go out and search the web as well as hunt through other data sources to create detailed research reports on almost any subject. Deep Research is initially being made available to subscribers to OpenAI’s $200 per month premium service. The company says Deep Research can match the work of PhD-level researchers and analysts. OpenAI CEO Sam Altman reiterated that the company is working on a new kind of hardware device to replace—or at least compete with—smartphones as the primary way people interact with AI assistants. OpenAI also announced a 50-50 joint venture with Softbank to sell OpenAI’s technology in Japan. This move comes as Softbank joins forces with OpenAI to invest in the $500 billion Stargate plan for massive new data centers for AI and while rumors are circulating that Softbank may lead to a massive new investment round in OpenAI that could value the company at $300 billion.

Meta says it may refrain from releasing future AI systems if they pose too great a risk. The company has belatedly joined other leading AI labs in releasing a framework for how it will handle AI Safety issues. Meta’s Frontier AI Framework says the company may decide to withhold release of any AI models classified as "high-risk" or "critical-risk" based on their potential to enable cybersecurity, chemical, and biological attacks—at least until it can introduce mitigations to reduce those risks. The framework relies on internal and external researcher input rather than empirical tests to assess risk levels. 

Elon Musk ally says key government agency must take an ‘AI-first’ approach. That’s according to a story in Wired that cited anonymous sources the publication said had attended a meeting between GSA staff and Thomas Shedd, the new Technology Transformation Services director. Shedd is a former Tesla engineer who is close to Elon Musk. According to Wired’s reporting, Shedd announced Monday that the U.S. government’s General Services Administration (GSA), which manages federal property and equipment, must pursue an "AI-first strategy.” This includes plans to develop AI coding agents for government-wide use and the automation of various internal tasks, Shedd said. He envisions transforming the GSA to operate like a "startup software company" and plans to create a centralized federal data repository, though cybersecurity experts have raised concerns about the challenges and complexities of automating government operations.

Pentagon employees connected to DeepSeek’s China-hosted service in days before military banned its use. That’s according to a Bloomberg News story that reported that Defense Department employees had connected their work computers to Chinese servers to access the tool for at least two days prior to the government moving to limit defense department access to the service. While the Defense Information Systems Agency began blocking the website on Tuesday, some Pentagon workers could still access DeepSeek on Wednesday, and military personnel had previously been downloading DeepSeek code on their workstations—apparently since the fall 2024, without raising security alarms. The incident has prompted different responses from military branches, with the Navy prohibiting any usage of DeepSeek due to security and ethical concerns, while thousands of Defense Department personnel continue to access DeepSeek through Ask Sage, an authorized software platform that doesn't directly connect to Chinese servers.

Can this tech billionaire save the media from an AI apocalypse? Fortune’s editor-in-chief Alyson Shontell has a must-read piece about Matthew Prince, the cofounder and CEO of Cloudflare, and his plan to use his company’s critical position safeguarding a good chunk of the internet from cyberattacks as a way to help media companies combat unauthorized scraping of their content by AI companies. What’s more, Prince thinks he can transform Cloudflare into a key broker in this data ecosystem, allowing publishers to be justly compensated for the data they contribute to AI models and their outputs. You can read the story in Fortune here.

EYE ON AI RESEARCH

Anthropic says it found a new method to combat jailbreaking. Anthropic said it had created a new system, which it calls Constitutional Classifiers, that was highly effective at preventing users from jailbreaking its Claude AI models. The system builds upon Anthropic's previous work on Constitutional AI—which was a way to create model guardrails by asking the model to apply a set of written principals, a constitution, when it formulated answers to user prompts. The method made Anthropic’s models less susceptible to jailbreaking than some competing models, but it was not entirely effective. The new Constitutional Classifiers are additional models, trained using examples of jailbreaking prompts of various kinds and in various languages, that are designed to detect likely jailbreaking attempts in both inputs, and possible guardrail violating content in model outputs.

Anthropic offered people a “bug bounty” prize of $15,000 if they could find a way to successfully overcome these classifiers and jailbreak Claude. But after 183 people spent more than 3,000 hours collectively attempting to bypass the system, Anthropic found none managed to find a “universal jailbreak” that would successfully overcome Claude’s guardrails across 10 different categories of prohibited response types. (Some of the techniques worked for some content types, but not others.) Overall, the system blocked about 95% of jailbreaking attempts. The classifiers are not a perfect solution though, since using them requires 23.7% more computing power than just running the base Claude model. Using the classifiers also results in Claude refusing to provide a response more often, which may make the model less helpful to legitimate users. The company said it was continuing to run a public test through Feb. 10 specifically looking to see if Claude can be jailbroken around questions involving chemical weapons. But so far, the system looks like a decent defense. It’s not clear yet, however, how well this might work for other models, particularly open weight models like DeepSeek that have been found to be especially easy to jailbreak. You can read Anthropic’s research paper on the topic here and coverage in Ars Technica here.

FORTUNE ON AI

DeepSeek has given open-source AI a big lift. But is it safe for companies to use? —by Christian Vasquez

Americans trust AI more than a CPA to do their taxes, according to new research —by Preston Fore

Exclusive: Ex-Palantir engineers think they can beat Salesforce, Microsoft, and other tech giants at the AI agent game —by Jeremy Kahn

Intel’s AI dreams slip further out of reach as it cancels its big data-center GPU hope, Falcon Shores —by David Meyer

Researchers say DeepSeek left sensitive information exposed, including users’ chat histories —by Bea Nolan

IBM CEO: DeepSeek proved us right—AI is not about big, proprietary systems —by Arvind Krishna (Commentary)

AI CALENDAR

Feb. 10-11: AI Action Summit, Paris, France

March 3-6: MWC, Barcelona

March 7-15: SXSW, Austin

March 10-13: Human [X] conference, Las Vegas

March 17-20: Nvidia GTC, San Jose

April 9-11: Google Cloud Next, Las Vegas

May 6-7: Fortune Brainstorm AI London. Apply to attend here.

BRAIN FOOD

AI may be poised to revolutionize physics. That’s the view of Mark Thomson, who will become the next director general of the advanced physics research laboratory Cern in 2026. In an interview with The Guardian, Thomson predicts that AI will revolutionize fundamental physics just as Google DeepMind’s AlphaFold AI for protein structure prediction is revolutionizing aspects of biology. At the Large Hadron Collider, AI is already being implemented throughout operations, from data collection to data analysis, allowing scientists to achieve results that would have required 20 times more data a decade ago, he said. A major upgrade planned for after 2030 will increase the LHC's beam intensity tenfold, enabling unprecedented observations of two Higgs bosons simultaneously and potentially revealing whether the universe could face a catastrophic collapse in the distant future. The technology could also help in the search for dark matter by allowing scientists to ask more open-ended questions about unexpected patterns in the data, rather than searching for specific signatures, Thomson said. It’s a good example of how some of the most impactful uses of today’s AI may come in the sciences.