On March 18, the Joint California Policy Working Group on AI Frontier Models (the “Working Group”) released its draft report on the regulation of foundation models, with the aim of providing an “evidence-based foundation for AI policy decisions” in California that “ensure[s] these powerful technologies benefit society globally while reasonably managing emerging risks.” The Working Group was established by California Governor Gavin Newsom (D) in September 2024, following his veto of California State Senator Scott Wiener (D-San Francisco)’s Safe & Secure Innovation for Frontier AI Models Act (SB 1047). The Working Group builds on California’s partnership with Stanford University and the University of California, Berkeley, established by Governor Newsom’s 2023 Executive Order on generative AI.
Noting that “foundation model capabilities have rapidly improved” since the veto of SB 1047 and that California’s “unique opportunity” to shape AI governance “may not remain open indefinitely,” the report assesses transparency, third-party risk assessment, and adverse event reporting requirements as key components for foundation model regulation.
Transparency Requirements. The report finds that foundation model transparency requirements are a “necessary foundation” for AI regulation and recommends that policymakers “prioritize public-facing transparency to best advance accountability.” Specifically, the report recommends transparency requirements that focus on five categories of information about foundation models: (1) training data acquisition, (2) developer safety practices, (3) developer security practices, (4) pre-deployment testing by developers and third parties, and (5) downstream impacts, potentially including disclosures from entities that host foundation models for download or use.
Third-Party Risk Assessments. Noting that transparency “is often insufficient and requires supplementary verification mechanisms” for accountability, the report adds that third-party risk assessments are “essential” for “creating incentives for developers to increase the safety of their models.” To support effective third-party AI evaluations, the report calls on policymakers to consider establishing safe harbors that indemnify public interest safety research and “routing mechanisms” to quickly communicate identified vulnerabilities to developers and affected parties.
Whistleblower Protections. Additionally, the report assesses the need for whistleblower protections for employees and contractors of foundation model developers. The report advises policymakers to “consider protections that cover a broader range of [AI developer] activities,” such as failures to follow a company’s AI safety policy, even if reported conduct does not violate existing laws.
Adverse Event Reporting Requirements. The report finds that adverse event reporting, i.e., proactive monitoring systems that collect information about events or incidents from mandated or voluntary reporters, is a “critical first step” in assessing the costs and benefits of AI regulation. The report recommends that foundation model reporting systems (1) provide reports to agencies that have relevant authority and expertise to address identified harms, with discretion to “share anonymized findings from reports with other industry stakeholders,” (2) use “initially narrow adverse event reporting criteria built around a tightly-defined set of harms” that can be revised over time, and (3) adopt a “hybrid approach” that combines mandatory reporting requirements for critical “parts of the AI stack” with voluntary reporting from downstream users.
Foundation Model Regulation Thresholds. Finally, the report assesses various options for defining thresholds that would trigger foundation model requirements, including “developer-level” thresholds (e.g., adeveloper’s employee headcount), “cost-level” thresholds (e.g.,compute-related costs of model training), “model-level” thresholds based on the model’s performance on key benchmarks, and “impact-level” thresholds based on the number of commercial users of the model. The report finds that “compute thresholds,” such as the EU AI Act’s threshold of 1025 floating-point operations per second (“FLOPS”) of computing power for model training, are “currently the most attractive cost-level thresholds” that should be used in combination with other metrics. Notably, the report “caution[s] against the use of customary developer-level metrics,” such as employee headcount, that “do not consider the specifics of the AI industry and its associated technology.”
The report and ongoing public comment process will inform lawmakers as they consider AI legislation during the 2025 legislative session, including SB 53, a foundation model whistleblower bill introduced by Senator Wiener last month. Lawmakers in Colorado, Illinois, Massachusetts, New York, Rhode Island, and Vermont are also considering foundation model legislation. The New York Responsible AI Safety & Education (“RAISE”) Act (A 6453), for example, would impose transparency, disclosure, documentation, and third-party audit requirements on certain developers of AI models that meet its compute and cost thresholds.
The Working Group is seeking public input on the report through its website, with responses due by April 8, 2025. The final version of the report is expected to be released by June 2025, ahead of the California legislature’s adjournment in September.