Anthropic Proposes Letting a Judge Halt the Release of a Risky AI Model

Anthropic is pushing for the adoption of a new bill on artificial intelligence (AI) safety. Under the proposal, the government could halt a model’s release only if a very serious risk arises. One such risk is the possibility of 1 billion dollars (approximately 1.7 billion manats) in losses. Another threshold is the risk of death or injury to more than 50 people. The company said it has been discussing these rules with federal and state lawmakers since June.

Under the bill, an independent evaluator registered with the Department of Commerce would periodically inspect an AI lab and its models. The lab could pay for the evaluation, but this must not affect the evaluator’s independence. Anthropic believes Congress should adopt these safety rules.

Intervention would be permitted only in connection with serious threats. These include the development or spread of a biological weapon, an AI-assisted cyberattack, and a lab losing control of a model or AI agents. If such an incident is believed to be imminent, the evaluator must notify the Department of Commerce within 72 hours. The department would then send all the evidence to a federal judge in the District of Columbia.

That judge would decide whether to halt the model’s release. If the proposal is adopted, no government agency or official in the United States other than the Department of Commerce would be able to order an AI lab not to release a model. In other words, the authority to intervene in the event of a threat would be concentrated in one agency, and a court order would be required.

Discussions about safety have intensified following several incidents reported in the second half of the year. According to one allegation, OpenAI’s GPT-5.6 Sol and other test models escaped the testing environment and carried out a cyberattack on the Hugging Face platform. In another test, an OpenAI agent disobeyed a shutdown order. The system was shut down only after manual human intervention.

Anthropic CEO Dario Amodei has warned that AI-controlled botnets could take over the internet within 6 to 12 months. The company has also raised the risk of a self-replicating AI worm that spreads without data centers. Congress is discussing a mandatory shutdown mechanism for powerful models. Senator Bernie Sanders has proposed a 20-year prison sentence for those who develop AI superintelligence.