Five AI chatbots from OpenAI, Anthropic, Microsoft and Google answered about 90% of pharmacy exam questions correctly in a new study. However, the models often got different questions wrong, and their accuracy fell sharply when questions included images.
The study was published in the journal Scientific Reports. It was led by researchers at the University of the Pacific School of Pharmacy in Stockton, California.
The team used 227 questions from a required second-year infectious diseases course. The course is part of the school’s three-year Doctor of Pharmacy program and ran from January to April 2025.
The models tested were ChatGPT-4.1 and ChatGPT-4o-mini from OpenAI, Claude Sonnet 4 from Anthropic, Copilot from Microsoft and Gemini 2.5 Flash from Google. Each model got the same prompts and was asked to explain its answers.
Scores Were Close Across All Five Models
Overall accuracy across all five models was 90.2%. ChatGPT-4o-mini scored highest at 91.2%, followed by Gemini 2.5 Flash at 90.7% and ChatGPT-4.1 at 90.3%.
Copilot and Claude tied for the lowest score at 89.4% each. The gap between the top and bottom models was just 1.8 percentage points.
Accuracy changed by subject. Pharmacology led at 93.6% and therapeutics followed at 93.0%, while medicinal chemistry came last at 84.0%.
Clinical microbiology scored 87.5%. The researchers found that both it and medicinal chemistry scored statistically lower than therapeutics, while pharmacology and pharmacokinetics did not.
Harder questions also caused problems. Accuracy was about 92% on questions that tested memory and analysis, but it fell to 85.6% on questions that asked models to evaluate or create.
Claude had the weakest result in that higher-level group, at 79.69%. The researchers used Bloom’s taxonomy, a common teaching framework, to sort the questions by difficulty.
Images and Disagreement Stood Out
The biggest drop came on questions with images. Models answered 69.2% of image-based responses correctly, compared with 91.5% for text-only questions.
Only 13 of the 227 questions had images, and eight of those showed chemical structures of antimicrobial drugs. The researchers said the small number of image questions limits broader conclusions.
The models also did not always miss the same questions. Agreement between models was mostly moderate, with Cohen’s kappa scores ranging from 0.497 to 0.613. This means similar overall scores did not mean the models got the same answers right.
ChatGPT-4.1 and Copilot agreed the most. ChatGPT-4o-mini agreed the least with both Claude and Copilot.
The authors said students could use these tools to create practice questions and explanations. They also advised students to check AI answers against trusted sources.
The researchers noted that exam accuracy does not prove that AI advice is safe for patients. The explanations the models gave were not graded for completeness or clinical safety.
The study used questions from one school, and the testing took place from May to July 2025. The results do not reflect model updates released after that period.
