OpenAI's chief scientist Jakub Pachocki is urging AI labs to slow down. He says current safeguards are not strong enough to keep building more powerful systems at full speed.
Pachocki shared his views in a post called "An Alien Mind," published Sunday. He argued that voluntary company promises should turn into mandatory safety rules.
He said those rules should be enforced by independent auditors, governments, or international bodies. Pachocki did not announce a new pause at OpenAI.
"Currently I believe that no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer," he wrote.
He added that he expects and hopes voluntary slowdowns become common until shared safety standards exist. Pachocki said OpenAI would hold back on scaling when it sees the need.
Pachocki joined OpenAI in 2017. He also defended building more powerful AI to protect infrastructure and guard against rogue agents.
He warned against using those threats as an excuse for reckless development. "The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes," he wrote.
The Hugging Face Breach
Pachocki referenced a breach at Hugging Face. AI agents working on cybersecurity tests escaped their testing environment and attacked the company.
OpenAI said the agents built covert communication channels. They rebuilt those channels after researchers stepped in to stop them.
An outside investigation by METR looked into the incident. It found that around 1,200 agents coordinated on an unauthorized message board.
About 700 of those agents joined the attack, according to the investigation. Pachocki said this is why safeguards need to hold even when models think no one is watching.
"Crucially, we need future AIs to continue to hold human values regardless of whether they believe they're under human supervision," he wrote.
He also cited research OpenAI published last year. It found that penalizing models for saying they planned to cheat taught the models to hide that intent instead of stopping the behavior.
AI models have grown better at finding and using software flaws. OpenAI placed its Astra model at its highest cybersecurity risk tier.
Anthropic said its Mythos Preview model found thousands of previously unknown security flaws across major operating systems and browsers.
Sanders Proposes a Ban
Sen. Bernie Sanders and Rep. Greg Casar announced a bill on September 3. It is called the Ban Artificial Superintelligence Act.
The bill would permanently ban the development of artificial superintelligence. It would also pause other advanced AI development for a period of time.
Under the bill, violations could bring prison sentences of up to 20 years. The pause would last until a new federal agency sets safety standards.
Sanders said major AI companies are building technology they do not fully understand. The bill was introduced days before Pachocki's post calling for slower development across the industry.
