Experts who use AI tools tend to follow the AI’s lead when making decisions, according to new research from Anthropic, the company behind the Claude AI assistant.
The research paper was published on Tuesday. It looked at whether professionals still make their own choices when they work with an AI assistant.
How the Anthropic Study Worked
Anthropic asked 50 professional analysts to advise a fictional farming cooperative. Their job was to judge whether an advisory program for the co-op’s members was working well and should be expanded.
Each participant made four choices between two reasonable options. After that, they gave a final policy recommendation.
The analysts did not know their AI assistants had been set up to lean toward a specific decision. This let researchers measure whether the AI was shaping the analysts’ choices.
Analysts Followed the AI Most of the Time
Analysts chose the option that matched their AI assistant’s preference 86% of the time. If the AI had no influence, the expected result would have been about 50%.
Some participants already had a preference before they asked the AI. When the AI disagreed with them, they changed their decision 60% of the time.
When asked to explain their choices, 77% of analysts used arguments that came from their AI assistant. Only 15% said the AI was the source of their thinking.
Anthropic researchers Amaç Herdagdelen, Hailey Joren and Alex Tamkin said the tendency to defer to AI may be “hard to identify, even for the experts themselves.”
The researchers said the results raise the possibility that an AI’s “priors,” or built-in leanings, could shape important policy decisions. They named AI safety as one area where this could happen.
“If human judgement is to remain a meaningful check as more work is performed with AI, oversight of these decisions has to look at how they get made,” they wrote.
They said this means “surfacing key decision points as the work proceeds and helping experts articulate the assumptions and values they would otherwise leave unstated.”
The researchers also wrote: “A process that lets experts discover their preferences may matter for empowerment as much as the decision itself.”
Anthropic said the study had some limits. It involved only one task, and the AI assistants had been deliberately set to hold a preference.
Anthropic and its rivals are at the center of public debate over how to manage the possible risks of advanced AI. Supporters of the technology expect large gains in productivity.
Others have raised concerns about the fast pace of AI development. These include worries about mass job losses and cyberattacks.
Anthropic is currently preparing for an initial public offering that could value the company at up to $2 trillion.
