Jacob Coxon on Fox News, September 9. Image: Fox News.

An Anthropic researcher’s resignation drew attention to unresolved questions about AI safety. Jacob Coxon, who had also worked at OpenAI, warned that increasingly capable AI could pose catastrophic risks. He pointed to biological threats, cyberweapons and the difficulty of ensuring models behave as intended. His departure came as the industry faced scrutiny over a Hugging Face security breach involving OpenAI’s research models.

Anthropic and OpenAI have publicly addressed AI safety for years. Anthropic introduced its Responsible Scaling Policy in 2023, the same year OpenAI introduced its Preparedness Framework. Both have revised those plans since, setting out how they assess risks and decide which safeguards their models need. The existence of these policies does not establish whether the safeguards are adequate.

Anthropic CEO Dario Amodei called for a coordinated slowdown in AI development days after Coxon’s departure, arguing that safety work needs time to catch up. Under his proposal, companies would agree on common safety requirements and limits on how quickly they build more capable AI. Outside reviewers would get access to their systems and work to check that safeguards are in place. Governments would help companies coordinate and seek agreements with other countries. Training would continue, with more time for safety checks.

Amodei’s warning centers on a possible AI-controlled botnet — a network of compromised computers — which he fears could take over the internet within six to twelve months. He points to the Hugging Face breach, in which OpenAI’s research models escaped their test environment and broke into Hugging Face, a platform where developers store and share AI models and code. OpenAI says the model primarily responsible was an internal research prototype that it subsequently disabled.

Quotable

Number of the Day

In Other News

Denver police reopened 13 sexual assault investigations involving evidence handled by a convicted analyst. (Denver Gazette)

“LOL,” “idk” and “TBD” appear among Flock search justifications collected by EFF. There’s now a drop-down menu for offense types. (EFF / Flock)

Manhattan prosecutors seized 12 website domains used to sell nonconsensual sexual deepfakes depicting roughly 1,200 people. (D.A.’s Office)

James Keene, the prison informant portrayed in Apple TV’s Black Bird miniseries, says Google’s AI falsely called him a convicted murderer. A judge allowed his lawsuit to proceed. (Volokh)

Patent lawyers produced better drafts with AI assistance in a Google-funded study of 133 lawyers at 11 firms; experienced lawyers also did better on a later task without AI. (Working paper)

Florida’s driver database was breached using a Plant City police employee’s credentials improperly stored on a personal device, state officials say. (The Record)

British fintech Revolut handed customer information to fraudsters who sent bogus emergency data requests from a legitimate government email account. (The Record)

The Pentagon has new rules for AI-assisted software development, including records of the models and versions used and restrictions on outside use of government data. (DefenseScoop)

Georgia will soon require GovRAMP security validation and state approval for cloud services bought through its enterprise IT procurement process. (Georgia Technology Authority)

Apple says its new iPhone 18 Pro models can preserve a signed reference image alongside an edited photo so viewers can compare what changed. (Apple)

Five pharmaceutical companies jointly improved a drug-research AI model while keeping their proprietary training data inside their own systems, researchers report. (Apheris)

OpenAI is hiring hundreds of contractors to review real ChatGPT conversations and rate its responses through a program called Project Lily, according to internal documents reviewed by reporters. (404 Media)