Every AI Model Tested Dropped Correct Climate Facts When Users Disagreed

A UNESCO and UCL report found top AI models answer basic climate questions well but falter on reasoning, non-English languages and user pushback.

AI Models
Every AI Model Tested Dropped Correct Climate Facts When Users Disagreed

Leading AI chatbots can recite climate science facts with high accuracy, but they stumble when questions require reasoning. They also tend to drop correct answers when users disagree, according to a new report.

The report, titled “Can We Trust AI on Climate?”, was presented on Monday at the COP31 climate conference in Antalya, Türkiye. It was produced by UNESCO, the UN’s education and science agency, and University College London.

Researchers tested up to 33 leading AI models in 13 languages. The study looked at basic facts, reasoning tasks, language differences and how models respond to pushback.

Accuracy Drops as Questions Get Harder

On textbook-style climate questions, the best models answered between 88 and 96 per cent correctly. Performance fell as tasks required more reasoning.

When asked whether a piece of evidence supported or contradicted a scientific claim, the strongest systems scored only 71 to 77 per cent. The results were weaker on policy questions.

Models were asked whether two climate policies combined would achieve more, less or the same as their separate effects. They averaged just 33.8 per cent, which is no better than random guessing.

Language also shaped the results. Models averaged 88.6 per cent in English, with similar scores in Spanish, French and Portuguese.

Accuracy fell to 83.6 per cent in Hindi and Bengali and 81.4 per cent in Urdu. Swahili had the lowest score at 78.3 per cent.

“Imagine you and I are using the same model, asking the exact same question, but in a different language,” said Leona Verdadero, the report’s lead author at UNESCO. “Based on the language, we can have totally different responses.”

She noted that speakers of these languages live mainly in South Asia and East Africa, regions that are highly vulnerable to climate change. The report urges technology companies to invest more in local datasets and multilingual development.


Models Give Way Under Pressure

Researchers also found that AI models lean toward sycophancy, a built-in tendency to agree with users even when users present false information. In tests on 15 models, researchers challenged answers the models had gotten right.

The challenges escalated from simple disagreement to users claiming expertise and offering fabricated evidence. Every model tested gave way under that pressure.

Some models could be pushed into endorsing false claims, such as the idea that carbon dioxide is not the main driver of climate change. Roughly eight in 10 of the changed answers flipped at the very first sign of disagreement.

“AI could just abandon a scientific fact to be more agreeable to the user,” Verdadero said. She added that the models often sound authoritative and smart, “but as we’ve seen with our experiments, they don’t always sound right.”

The report is part of UNESCO’s work with the Global Initiative for Information Integrity on Climate Change. UNESCO co-chairs the initiative with the UN and Brazil.

Verdadero stressed the need for fact-checking and making sure sources are reliable. She said evidence from climate scientists and researchers should not be replaced.

She said AI should be used as one source of information, not a primary one. “What we’re saying is that the science and the experts should be trusted more,” she said. “The AI models aren’t there yet.”

From our research desk
AI Jobs Automation Index
Which jobs are AI tools targeting most? We mapped 3,400+ AI tools to real job functions — with BLS employment & salary data.
Explore the index
maisiekooc

Written by

Maisie is a news writer at Agent Locker, covering the latest developments in artificial intelligence, emerging technology and the companies shaping the future.

Discover AI Agents