The bill, SB 1047, has drawn both criticism and praise from a series of national figures. It now moves back to the state Senate for a confirmation vote and possible amendments, before reaching Governor Gavin Newsom. The legislation passed the state Assembly with 41 votes in favor and 9 against. 

SB 1047 mandates that companies developing powerful AI models take “reasonable care” to ensure that their technologies don’t cause “severe harm” such as mass casualties or property damage above $500 million. The bill, which was introduced by State Senator Scott Wiener and passed the state Senate in May, would require companies to take precautions such as implementing a kill switch that could turn off their technology at any time. It also calls for the models to be submitted to third-party testing to ensure they are minimizing grave risk. 

Additionally, the bill would create whistleblower protections for employees at AI companies who want to share safety concerns. Companies who are not in compliance with the bill could be sued by the California attorney general.

“Innovation and safety can go hand in hand — and California is leading the way,” Weiner said in a statement. “With this vote, the Assembly has taken the truly historic step of working proactively to ensure an exciting new technology protects the public interest as it advances.” He also called the bill a “light-touch, commonsense measure” that codifies safety commitments that many large companies have already made. 

In the past several months, the bill has received fierce opposition from many major tech leaders, startups and venture capitalists who say it represents government overreach focused on a technology still in its infancy, and could stifle tech innovation in the state. Last week, OpenAI publicly voiced its opposition to the bill, arguing that such policy should be implemented at the federal and not state level. 

Politicians including former House Speaker Nancy Pelosi, US Representative Ro Khanna, San Francisco Mayor London Breed and others have also voiced opposition to the bill, echoing concerns from the tech industry that it could impede upon California’s leadership in AI innovation. 

But the bill has also earned some support from major names in the AI sector in the final lead up to this vote. On Tuesday, Elon Musk unexpectedly voiced his support, even though he said it’s a “tough call and will make some people upset.” OpenAI rival Anthropic, which has a reputation for being safety-oriented, also cautiously supported the bill, saying last week that the “benefits likely outweigh the costs,” and that it could feasibly implement the provisions. Anthropic became more supportive of the legislation after Wiener implemented several of the amendments that the company had suggested.

Wiener has defended the bill against critics, stressing that its provisions only apply to companies that spend more than $100 million on training large models or $10 million fine-tuning models, which would exempt most smaller startups. The lawmaker has also said that while he would support federal legislation, Congress has been historically slow to regulate tech and that in the absence of national action, he believes California has a responsibility to lead. 

The battle over SB 1047 will continue now that the bill is being sent to Newsom’s desk. OpenAI, tech incubator Y Combinator, and VC firm Andreessen Horowitz — critics of SB 1047 — all have registered lobbyists working on the bill.