{"id":217637,"date":"2026-04-16T11:30:23","date_gmt":"2026-04-16T11:30:23","guid":{"rendered":"https:\/\/teknomers.com\/en\/21-popular-ai-chatbots-tested-for-differential-diagnosis-they-perform-worse-than-expected\/"},"modified":"2026-04-16T11:30:24","modified_gmt":"2026-04-16T11:30:24","slug":"21-popular-ai-chatbots-tested-for-differential-diagnosis-they-perform-worse-than-expected","status":"publish","type":"post","link":"https:\/\/teknomers.com\/en\/21-popular-ai-chatbots-tested-for-differential-diagnosis-they-perform-worse-than-expected\/","title":{"rendered":"21 Popular AI Chatbots Tested for Differential Diagnosis: They Perform Worse Than Expected"},"content":{"rendered":"\n
\n

‘House’ is a series that I love. I don’t care about the intrastories in the slightest, but the process of differential diagnosis – despite all the movie stuff – drives me crazy. This ability to rule out diseases that could explain the same symptoms to arrive at the most probable diagnosis seems like witchcraft to me. Well: they have put the 21 most popular AI chatbots to perform this differential diagnosis and the result is clear.<\/p>\n

It fails more than a fairground shotgun.<\/p>\n

AI Chatbots in Differential Diagnosis<\/h2>\n

In short:<\/strong> Mass General Brigham is not ‘anyone’. It is a non-profit network of American doctors and hospitals, including two of the most prestigious medical teaching institutions in the country. From January to December 2025, a group of researchers from the institution conducted<\/a> an evaluation of 21 AI chatbots such as Claude 4.5 Opus, DeepSeek, Gemini 3.0 Pro, GPT-5, and Grok 4 to determine their efficacy in early diagnosis through differential diagnosis.<\/p>\n

Basic Information Challenges<\/h2>\n

The information provided to these models was extremely basic, mirroring what human professionals deal with when forming a differential diagnosis. The ultimate goal was to evaluate the clinical reasoning abilities of these advanced language models. Unfortunately, the answer remains a resounding no. While certain models optimized for reasoning scored higher than simpler ones like Gemini 1.5 Flash, the bottom line reveals that large language models (LLMs) still face significant limitations in this context.<\/p>\n

Evaluation of Efficacy<\/h3>\n

The exam:<\/strong> Each model was given 29 clinical cases, amounting to over 16,200 responses in total. The results showed that these newer, advanced chatbots failed<\/a> to produce an adequate differential diagnosis in about 80% of cases, especially when only basic patient information was available.<\/p>\n

The Limitations of Initial Diagnosis<\/h2>\n

The fundamental data\u2014age, sex, and symptoms\u2014though vague, is what human professionals rely on initially. As tests are conducted and more data accumulated, they refine their findings. It\u2019s that crucial first ‘discard’ step that often makes the difference.<\/p>\n

Performance with More Data<\/h3>\n

As the LLMs were provided with more comprehensive data, their performance greatly improved. When presented with details such as physical analyses, lab results, and diagnostic images, AI could reach a final diagnosis in over 90% of cases. However, this illustrates the gap in their capabilities when tasked with initial filtering.<\/p>\n

Trust and Safety Concerns in AI<\/h2>\n

Don’t trust<\/strong> Google ChatGPT. Researchers emphasize that while commercial LLMs can accurately identify a final diagnosis with complete data, they struggle significantly at the onset of open cases. This raises questions regarding their reliability for home care. Although the AI industry continues to promote these tools in healthcare, the study highlights that \u201cdespite ongoing improvements, commercial LLMs are not ready for unsupervised clinical implementation.\u201d<\/p>\n

The Human Element in Healthcare<\/h3>\n

Human oversight is essential, as \u201cvery close supervision\u201d is required to safely integrate an LLM into healthcare settings. Cases are increasingly emerging of individuals opting for ChatGPT over traditional self-diagnosis methods. The persistence of \u201challucinations\u201d in these advanced models raises alarming concerns regarding patient safety and integrity.<\/p>\n

AI as a Tool Rather Than a Replacement<\/h2>\n

In any case, medical AI serves as just another tool rather than a complete solution. The models tested benefitted from common knowledge but lack specialization. AI can assist with tasks like eliminating possibilities or sorting copious data, but it\u2019s not yet a reliable partner in differential diagnosis.<\/p>\n

El Salvador’s AI Experiment<\/h3>\n

In a bold move, El Salvador\u2019s government, led by President Nayib Bukele, has announced a $500 million initiative to automate healthcare using Gemini. Citizens will access the app Dr.SV<\/a>, which will act as a family doctor, assigning real doctors for diagnosis and monitoring chronic conditions. Despite aspirations for creating an optimal healthcare system, it\u2019s worth noting that over 7,700 employees were laid off from the health sector in 2025.<\/p>\n

For the sake of Salvadorans, let\u2019s hope this experiment does not lead to dire consequences, reminiscent of prior technological misadventures.<\/p>\n<\/div>\n


\n
General News – 2<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"

‘House’ is a series that I love. I don’t care about the intrastories in the slightest, but the process of differential diagnosis – despite all the movie stuff – drives me crazy. This ability to rule out diseases that could explain the same symptoms to arrive at the most probable diagnosis seems like witchcraft to […]<\/p>\n","protected":false},"author":1,"featured_media":217638,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[36399],"tags":[37133,10989,11152,1593,11645,4679,4441,1565],"class_list":["post-217637","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology","tag-chatbots","tag-diagnosis","tag-differential","tag-expected","tag-perform","tag-popular","tag-tested","tag-worse"],"_links":{"self":[{"href":"https:\/\/teknomers.com\/en\/wp-json\/wp\/v2\/posts\/217637","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/teknomers.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/teknomers.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/teknomers.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/teknomers.com\/en\/wp-json\/wp\/v2\/comments?post=217637"}],"version-history":[{"count":1,"href":"https:\/\/teknomers.com\/en\/wp-json\/wp\/v2\/posts\/217637\/revisions"}],"predecessor-version":[{"id":217639,"href":"https:\/\/teknomers.com\/en\/wp-json\/wp\/v2\/posts\/217637\/revisions\/217639"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/teknomers.com\/en\/wp-json\/wp\/v2\/media\/217638"}],"wp:attachment":[{"href":"https:\/\/teknomers.com\/en\/wp-json\/wp\/v2\/media?parent=217637"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/teknomers.com\/en\/wp-json\/wp\/v2\/categories?post=217637"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/teknomers.com\/en\/wp-json\/wp\/v2\/tags?post=217637"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}