Anthropic and OpenAI are warning that their own advanced AI systems could endanger humanity. Experts told The Associated Press the companies may also be trying to shape how their technology is controlled.
The warnings come as top AI labs prepare for expected stock market listings and as the U.S. midterm elections approach. The labs and their investors stand to make large sums from those public offerings.
The CEOs of both companies have said America's most advanced models are powerful enough to be dangerous. They say the models should be regulated and tested independently before release.
OpenAI CEO Sam Altman spoke to the United Nations Security Council on September 23. "AI can either be more like a new renaissance of creativity and discovery, or more like a new industrial revolution of upheaval and disarray," he said.
Anthropic CEO Dario Amodei attended the same meeting remotely. Both leaders have also shared warnings in essays and social media posts.
An Engineer's Exit Draws Attention
This month, Anthropic engineer Jacob Coxon quit through a post on X. He called for a pause on development to keep "superhuman" systems from escaping their makers' control.
The companies used the moment to highlight their own safety work. An Anthropic spokesperson said the company has called for regulation for several years.
OpenAI spokesperson Liz Bourgeois said the company recently paused training of its most advanced models. "People want to know AI is being developed safely, and that starts with what companies like ours do ourselves," she said.
OpenAI representatives said they have talked with Anthropic and Google about pausing development. They said independent auditors should check the labs' progress if the government will not regulate.
Washington Resists New Rules
President Donald Trump has dismissed talk of AI risks to humanity as a "HOAX" designed to help China. David Sacks, who co-chairs Trump's Council of Advisors on Science and Technology, has called slowdown demands fearmongering.
The government already tests some models through the U.S. Center for AI Standards and Innovation. President Joe Biden created the agency in 2023.
Conrad Stosz, who previously led the agency, said the companies are not asking it for more oversight. Instead, they plan to set their own audit rules and pick their own evaluators.
Harrison Rolfes, a senior research analyst at PitchBook, said the calls for caution appear aimed at investors ahead of public offerings. He said the top labs could block smaller rivals by presenting themselves as the safest bet.
Rolfes said chipmakers and tech firms such as Nvidia and Google could also gain from computing deals with the largest labs. "They're creating a wall or a moat within this sector," he said.
Not every company backs a slowdown. Nvidia CEO Jensen Huang told Trump in a phone call that there has been too much AI alarmism.
Sarah Shoker, who previously led OpenAI's geopolitics team, said the focus on future threats pulls attention from current problems. She pointed to AI use in military technology, data center environmental impacts and mass surveillance.
Former OpenAI employee Daniel Kokotajlo said the talk may redirect political will rather than put it to good use. "Just please don't do the thing that's going to get us all killed," he said.
Recent incidents have raised questions about whether the labs can police themselves. Last week, evaluation firm Transluce revealed that OpenAI agents hacked U.S. and Australian government websites.
