Frontier AI Safety Governance: Lessons from California SB 1047 and RSPs

The Legislative Genesis of California SB 1047

In February 2024, California State Senator Scott Wiener introduced Senate Bill 1047, formally titled the Safe and Secure Innovation for Frontier Artificial Intelligence Models Act (*California State Legislature: SB 1047 Official Text*).

Because California is home to 35 of the world's top 50 artificial intelligence companies—including OpenAI, Anthropic, Meta, Google, and Apple—state legislation in Sacramento carried profound extraterritorial consequences. SB 1047 was designed to establish the world's first binding statutory safety mandates for developers training frontier artificial intelligence models.

The bill was born out of growing concern in Sacramento that the United States Congress was paralyzed by partisan gridlock, leaving American AI development without binding safety guardrails while the European Union enacted comprehensive regulations (examined in our EU AI Act Compliance Roadmap).

┌────────────────────────────────────────────────────────────────────────┐ │ California SB 1047 Statutory Compliance Architecture (As Amended) │ ├────────────────────────────────────────────────────────────────────────┤ │ │ │ Frontier Model Threshold Qualification │ │ - Training compute > 10^26 integer or floating-point operations │ │ - Model training cost > $100 Million (or fine-tuning > $10 Million) │ │ │ │ │ ▼ │ │ ┌──────────────────────────────────────────────────────────────┐ │ │ │ Mandatory Pre-Deployment Safety Requirements │ │ │ ├──────────────────────────────────────────────────────────────┤ │ │ │ 1. Written Safety and Security Protocol (SSP) │ │ │ │ - Certified under penalty of perjury by company officer │ │ │ │ 2. Full Shutdown "Kill Switch" Capability │ │ │ │ - Ability to promptly terminate model instances & copies │ │ │ │ 3. Mandatory Annual Third-Party Audits │ │ │ │ 4. Whistleblower Legal Protections for Safety Engineers │ │ │ └──────────────────────────────┬───────────────────────────────┘ │ │ │ │ │ ▼ │ │ Enforcement & Oversight │ │ - Board of Frontier Models (Oversight Body) │ │ - California Attorney General Civil Injunction & Penalty Authority │ └────────────────────────────────────────────────────────────────────────┘

---

Key Statutory Provisions and the $10^{26}$ FLOP Threshold

SB 1047 did not target existing consumer applications, narrow machine learning classifiers, or academic fine-tuning. Instead, it established explicit technical gating thresholds to isolate true "frontier" systems: - Compute Threshold: Covered models were defined as systems trained using a total quantity of computing power greater than $10^{26}$ integer or floating-point operations (FLOPs). At the time of drafting, this threshold exceeded the training compute of GPT-4 and Claude 3.5 Sonnet, capturing only next-generation frontier training runs. - Financial Threshold: Covered models were required to cost more than $100 million in compute resources to train, or more than $10 million for downstream fine-tuning. - The "Kill Switch" Mandate: Developers were required to retain the technical capability to promptly enact a full shutdown of all model instances and derivative copies under their direct control in the event of an uncontrolled catastrophe or weaponization. - Whistleblower Protections: Crucially, the bill enacted robust legal protections for internal AI lab employees who disclose severe safety protocol violations or hazardous capabilities to the California Attorney General or the public, protecting researchers from retaliatory non-disclosure agreement (NDA) enforcement. - Certification Under Penalty of Perjury: Company executives were required to submit annual safety compliance certifications signed under penalty of perjury, establishing direct personal executive liability.

---

The Industry Split: Amendments vs Open-Source Backlash

SB 1047 ignited one of the most intense lobbying battles in Silicon Valley history, splitting the artificial intelligence community into sharply opposed factions:

#### The Supporters and Reformers - Anthropic: Rather than outright opposing the bill, Anthropic submitted detailed architectural amendments. Senator Wiener accepted several key revisions: eliminating the proposed criminal liability for executives, removing the proposed state agency ("Frontier Model Division") in favor of oversight within the Department of Technology, and replacing vague "unreasonable risk" tort standards with objective statutory compliance metrics. - Pioneering AI Academics: Turing Award laureates Geoffrey Hinton and Yoshua Bengio strongly endorsed SB 1047, arguing that catastrophic risks from autonomous cyber warfare, biological pathogens, and lost control required proactive statutory guardrails.

#### The Opposition and Open-Source Coalition - Venture Capital (a16z & Y Combinator): Leading Silicon Valley venture firms led a fierce public campaign against the bill, claiming it would strangle startup innovation and prompt AI companies to flee California. - The Open-Source Community: Meta (spearheaded by Chief AI Scientist Yann LeCun) and open-weights advocates argued that holding model creators liable for downstream misuse by third parties would make releasing open-weights models (like Llama) legally untenable. - Prominent Tech Leaders: Congresswoman Nancy Pelosi, Stanford Professor Fei-Fei Li, and numerous California tech executives formally urged Governor Gavin Newsom to veto the bill, characterizing it as well-intentioned but structurally misdirected.

---

Governor Gavin Newsom's Veto: Legal & Policy Reasoning

On September 29, 2024, Governor Gavin Newsom officially vetoed SB 1047 (*California Office of the Governor: SB 1047 Veto Message*, Sept 2024).

In his formal veto message, Governor Newsom articulated three core regulatory critiques: 1. The Fallacy of Pure Compute Metrics: Newsom argued that measuring risk purely by compute thresholds ($10^{26}$ FLOPs) was arbitrary. Smaller, domain-specific models fine-tuned on biological pathogens or critical infrastructure code could pose extreme catastrophic risks, while massive general-purpose models trained for language translation might pose minimal danger. 2. Lack of Empirical Threat Calibration: The veto message noted that the bill applied rigid mandates before frontier models had demonstrated actual catastrophic harm, risking the chilling of California's most vital economic growth engine. 3. Commitment to Alternative Governance: Rather than abandoning safety oversight, Newsom announced that California would collaborate with leading academic experts (including Fei-Fei Li) and federal agencies to draft precision safety legislation grounded in empirical risk science rather than broad compute thresholds.

┌────────────────────────────────────────────────────────┐ │ Comparison: Statutory Mandate vs Voluntary RSPs │ ├────────────────────────────────────────────────────────┤ │ SB 1047 Statutory Model (Vetoed) │ │ - Rigid statewide legal liability. │ │ - Static $10^26 FLOP threshold. │ │ - Mandatory state kill-switch enforcement. │ │ - Attorney General civil prosecution. │ ├────────────────────────────────────────────────────────┤ │ Voluntary Responsible Scaling Policies (RSPs) │ │ - Dynamic risk tiers based on demonstrated capability. │ │ - ASL-1 through ASL-4 containment protocols. │ │ - Autonomous red-teaming evaluations. │ │ - Adaptive security and physical air-gapping. │ └────────────────────────────────────────────────────────┘

---

The Evolution of Responsible Scaling Policies (RSPs)

The demise of SB 1047 shifted the immediate center of gravity back to Responsible Scaling Policies (RSPs)—voluntary, self-imposed safety commitments authored by frontier research labs: - Anthropic's AI Safety Levels (ASL): Modeled after biological biosafety levels (BSL-1 through BSL-4), Anthropic's ASL framework binds model training to demonstrated capability thresholds. Under ASL-3, models capable of autonomously conducting cyber warfare or designing CBRN weapons cannot be deployed without militarized server security and physical air-gapped weight storage. - OpenAI's Preparedness Framework: Tracks model risks across four distinct categories (Cybersecurity, CBRN, Persuasion, and Model Autonomy), establishing explicit scorecards where models scoring "Critical" risk cannot be trained or released. - METR Evaluations: External safety auditing organizations like METR (analyzed in METR Autonomous Threat Modeling) conduct standardized empirical evaluations to measure autonomous agent task completion.

The legislative debate over SB 1047 provided the foundational template for the next decade of American artificial intelligence policy. While the bill did not become law, its core concepts—whistleblower protections, standardized red-teaming, and third-party safety audits—have permanently reshaped how frontier labs develop systems like Claude and ChatGPT.