The Safe and Secure Innovation for Frontier Artificial Intelligence Models Act, or SB 1047, was a failed 2024 California bill intended to "mitigate the risk of catastrophic harms from AI models so advanced that they are not yet known to exist". Specifically, the bill would have applied to models which cost more than $100 million to train and were trained using a quantity of computing power greater than 1026 integer or floating-point operations. SB 1047 would have applied to all AI companies doing business in California—the location of the company would not matter. The bill would have created protections for whistleblowers and required developers to perform risk assessments of their models prior to release, with guidance from the Government Operations Agency. It would also have established CalCompute, a University of California public cloud computing cluster for startups, researchers and community groups.
Background The rapid increase in capabilities of AI systems in the 2020s, including the release of ChatGPT in November 2022, caused some researchers and members of the public to become concerned about the existential risks associated with increasingly powerful AI systems. Hundreds of tech executives and AI researchers, including two of the so-called "Godfathers of AI", Geoffrey Hinton and Yoshua Bengio, signed a statement in May 2023 calling for the mitigation of the "risk of extinction from AI" to be a global priority alongside "pandemics and nuclear war". However, the plausibility of these risks is still widely debated. Strong regulation of AI has been criticized for purportedly causing regulatory capture by large AI companies like OpenAI, a phenomenon in which regulation advances the interest of larger companies at the expense of smaller competition and the public in general, although OpenAI ended up opposing the bill. Other advocates of AI regulation aim to prevent bias and privacy violations, rather than existential risks. For example, some experts who view existential concerns as overblown and unrealistic view them as a distraction from near-term harms of AI like discriminatory automated decision making. In the face of existential concerns, technology companies have made voluntary commitments to conduct safety testing, for example at the AI Safety Summit and AI Seoul Summit. In 2023, not long before the bill was proposed, Governor Newsom of California and President Biden issued executive orders on artificial intelligence. State Senator Wiener said SB 1047 draws heavily on the Biden executive order, and is motivated by the absence of unified federal legislation on AI safety. Historically, California has passed regulation on several tech issues itself, including consumer privacy and net neutrality, in the absence of action by Congress.
History
Proposal and voting The bill was authored by State Senator Scott Wiener. Wiener first proposed AI legislation for California through an intent bill called SB 294, the Safety in Artificial Intelligence Act, in September 2023. On February 7, 2024, Wiener introduced SB 1047. On May 21, SB 1047 passed the Senate 32–1. The bill was significantly amended by Wiener on August 15, 2024, in response to industry advice. Amendments included adding clarifications, and removing the creation of a "Frontier Model Division" and the penalty of perjury. On August 28, the bill passed the State Assembly 48–16. Then, due to the amendments, the bill was once again voted on by the Senate, passing 30–9.
Veto by governor On September 29, Governor Gavin Newsom vetoed the bill. The deadline for California lawmakers to overrule Newsom's veto was November 30, 2024. Newsom cited concerns over the bill's regulatory framework targeting only large AI models based on their computational size, while not taking into account whether the models are deployed in high-risk environments. Newsom emphasized that this approach could create a false sense of security, overlooking smaller models that might present equally significant risks. He acknowledged the need for AI safety protocols but stressed the importance of adaptability in regulation as AI technology continues to evolve rapidly. Governor Newsom also committed to working with technology experts, federal partners, and research institutions, including the Carnegie Endowment for International Peace, led by former California Supreme Court Justice Mariano-Florentino Cuéllar; and Stanford University's Human-Centered AI (HAI) Institute, led by Dr. Fei-Fei Li. He announced plans to collaborate with these entities to advance responsible AI development, aiming to protect the public while fostering innovation.
Provisions SB 1047 would have covered AI models with training compute over 1026 integer or floating-point operations and a cost of over $100 million. If a covered model is fine-tuned using more than $10 million, the resulting model would also have been covered. The bill would have defined critical harms with respect to four categories:
Creation or use of a chemical, biological, radiological, or nuclear weapon Cyberattacks on critical infrastructure causing mass casualties or at least $500 million of damage Autonomous crimes causing mass casualties or at least $500 million of damage Other harms of comparable severity Developers would have needed to create a "safety and security protocol" before training covered models. Before deployment, they would have submitted a statement of compliance, confirming they took reasonable care to take measures to prevent covered models that pose an unreasonable risk of critical harms. The statement would have included risk assessments and descriptions of their compliance process. These rules would have applied to both covered models and their derivatives, including post-training modifications, with annual third-party audits required starting in 2026. Safeguards to reduce risk included the ability to shut down the model, which has been variously described as a "kill switch" and "circuit breaker". Whistleblowing provisions would have protected employees who report safety problems and incidents. Additionally, SB 1047 would have created a public cloud computing cluster called CalCompute, associated with the University of California, to support startups, researchers, and community groups that lack large-scale computing resources.
… excerpt ends here. Continue reading the full article.
