Anthropic published a framework Wednesday calling on Congress to require mandatory independent safety testing for the most powerful AI models and to give the government legal authority to block or delay deployments that pose catastrophic risks.
The company wants independent evaluations required for models trained above a compute threshold, with government authority to block dangerous deployments
Anthropic CEO Dario Amodei. (Bloomberg / Getty Images)
Anthropic published a framework Wednesday calling on Congress to require mandatory independent safety testing for the most powerful AI models and to give the government legal authority to block or delay deployments that pose catastrophic risks.
The proposal would apply to models trained using more than 10²⁵ floating-point operations and developed by companies earning more than $500 million in AI-related revenue or spending more than $1 billion on AI research and development, the company said. It identifies four categories of catastrophic risk the rules would address: biological weapons, cyberattacks on critical infrastructure, loss of control over AI systems, and AI-accelerated AI development itself.
Under the framework, frontier developers would be required to test their models and publish summaries of the results, engage at least one qualified independent evaluator to review those findings, and maintain a robust security program protecting model weights and training infrastructure, the company said. Civil penalties tied to global annual revenue would escalate with repeated violations.
In an essay published Wednesday, Anthropic CEO Dario Amodei drew on familiar analogies to make the case for stricter oversight, likening AI to automobiles, aviation, and pharmaceuticals — fields where essential technologies are subject to binding safety rules because of their potential to cause mass harm. "It is time to go beyond transparency to more serious and binding regulation of AI."
Wading into the federal-versus-state preemption fight, Anthropic took the position that existing state-level AI rules should remain intact unless Congress enacts legislation that meets or exceeds the rigor of what the company is proposing. Separately, the framework calls on both Congress and state governments to overhaul their unemployment insurance infrastructure in anticipation of workforce disruptions driven by AI, according to Reuters.
The proposal arrives amid a broader standoff over AI oversight. The regulatory landscape has grown increasingly contested, with President Donald Trump signing an executive order that calls for voluntary government access to AI models before release — stopping short of mandatory review — while states including California and New York have moved to fill the gap with their own requirements. Many Democrats criticized the Trump order as insufficient.
Tensions between Anthropic and the Trump administration date to an earlier falling-out in which the company refused military use of its models for domestic surveillance operations and fully autonomous weapons, a stance that prompted the government to add Anthropic to a national security blacklist scheduled to take effect later this year.
Join 500,000+ readers who start their day with Quartz.
By subscribing, you agree to our Terms of Service and Privacy Policy.