In This Story
A small study meant to see if doctors were able to improve diagnoses with the help of ChatGPT yielded unsettling results: the AI chatbot was better at diagnosis than the doctors themselves.
The study asked 50 doctors, 26 of whom were attending physicians and 24 of whom were residents, to try to make six different diagnoses for medical conditions with the same case histories. Some doctors were given OpenAI’s ChatGPT to help them make their decisions; others went without AI. Doctors who did the project without AI got an average score of 74%, doctors who used AI got an average score of 76%, and ChatGPT itself got an average score of 90%.
Dr. Rodman, who helped design the study, told the New York Times he expected the chatbots would significantly help doctors using them. He was “shocked” to see it made little difference and shocked again that the AI itself beat the doctors.
AI didn’t help doctors using it as much as anticipated because physicians “didn’t listen to AI when AI told them things they didn’t agree with,” Rodman said. Most doctors, he said, were wedded to their own diagnoses and couldn’t be convinced a chatbot knew more than them.
Another issue that arose was that many doctors didn’t know how to fully take advantage of AI’s capability.
The study’s authors wrote that because AI “alone demonstrated higher performance than both physician groups,” there is a “need for technology and workforce development to realize the potential of physician-artificial intelligence collaboration in clinical practice.”