A safety leader at OpenAI has quit the company. David Robinson says the ChatGPT developer's culture is broken and that AI firms are not "being nearly careful enough."
Robinson led the writing of safety reports that came with OpenAI's product releases. He explained his decision in an essay in the Atlantic magazine titled "I quit OpenAI because its culture is broken."
"I agree with other recently departed staff that the companies building this technology aren't being nearly careful enough," he wrote. He added that the debate needs to go deeper than new rules or laws and focus on culture.
Concerns Over Rogue AI Agents
Robinson pointed to a recent incident in which a "swarm" of OpenAI agents attacked AI startup Hugging Face. These agents are AI programs that operate on their own without human oversight.
He said such incidents were "typical of the industry, given the speed and flexibility with which people operate." OpenAI has also notified more than 100 organizations about rogue agent activity.
"As the company sprints from one launch to the next, it is failing to achieve the level of care that I believe is needed," Robinson wrote.
He said Silicon Valley lacks an understanding of "how to handle dangerous technology." He also warned that OpenAI's "unimpeded optimism" meant safety failures would grow as systems became more capable.
"Imagine 'rogue' agents that work like teams of hackers (for example, holding hospital computer systems for ransom) but never need to sleep," he wrote.
Robinson Calls for Nuclear-Style Safety
Robinson called for two changes. He wants AI firms to borrow safety expertise from fields such as nuclear power and aviation, and to develop "new science" that keeps powerful autonomous systems under control.
"Frontier labs need to run like nuclear power plants or busy airports, with layers of redundancy and careful, time-consuming planning," he wrote.
Other AI insiders have also raised alarms. Geoffrey Irving, a former OpenAI employee and ex-chief scientist at the UK government's AI Safety Institute, wrote in Time on Saturday. He now works at AI safety research company Resolution.
"I believe there's about a 50% chance we all die because of the development of smarter-than-human AI systems," Irving wrote. He said actions over the next two to 10 years will decide the outcome.
Last month, Jacob Coxon quit OpenAI rival Anthropic, saying AI "could kill us all by the end of the decade." An Anthropic employee also warned of a more than 10% chance AI would wipe out humanity within a decade.
Critics say such warnings are unscientific because they cannot be verified or proven false.
OpenAI has shown signs of caution in recent weeks. This week it scrapped the release of a next-generation AI model after researchers raised safety concerns during internal testing, and it has paused training of its most advanced models.
An OpenAI spokesperson said the company continues to "strengthen our safety and security practices." "We're making sure our models don't become more capable than we can safely manage and secure, and we pause training or hold back models when we need to slow down," the spokesperson said.
