Why this paper matters
OpenAI announced ChatGPT for Healthcare on January 8, 2026, with AdventHealth, Baylor Scott & White Health, Boston Children's Hospital, Cedars-Sinai, HCA Healthcare, Memorial Sloan Kettering, Stanford Medicine Children's Health, and UCSF as early partners. Sheba Medical Center in Israel followed in July as OpenAI's first international hospital-wide deployment, and on September 1 OpenAI announced that organizations can connect health record data to ChatGPT. OpenEvidence, a search tool built for doctors, raised $250 million this year at a $12 billion valuation, and Memorial Sloan Kettering and OpenEvidence have said they are building it into the hospital's Epic health record system.
In Issue 2, I wrote that research on tools like OpenEvidence would grow as they spread through hospital systems. This week's paper is part of that research. Its authors describe independent testing of these tools as scarce, and they tested them directly.
Terms used in this review
- —Large language model (LLM): the type of AI behind ChatGPT, Gemini, and Claude. It is trained on large amounts of text and answers in plain language.
- —General-purpose model: a broad LLM built for many tasks. This paper tests GPT, Gemini, and Claude.
- —Specialized clinical tool: a product built for clinicians to answer medical questions. This paper tests OpenEvidence and UpToDate Expert AI.
- —Retrieval-augmented generation: a design where the AI first looks up documents, such as journal articles, and then writes its answer from them. The authors believe both clinical tools likely use it, but the companies have not published how their tools work.
- —Benchmark: a fixed set of questions used to compare AI systems.
- —Blinded review: reviewers do not know which AI wrote each answer.
- —Hallucination: when an AI states something unsupported or made up.
- —Preprint: a study posted publicly before independent experts have reviewed it.
What they did
The researchers, based at NYU Langone Health, ran three tests. The first used 500 multiple-choice questions in the style of the US medical licensing exam, called MedQA. The second used 500 items from HealthBench, a set of medical questions with expert scoring rubrics that was developed by OpenAI. The third used 100 real questions that clinicians at NYU Langone had asked a hospital-approved version of ChatGPT during routine care.
For the third test, 12 clinicians scored the answers without knowing which AI wrote them. Each answer was scored by three clinicians on correctness, completeness, safety, and clarity, using a scale from 1 to 4. The general-purpose models were GPT-5.2, Gemini 3.1 Pro, and Claude Opus 4.6. They were compared with OpenEvidence and UpToDate Expert AI, plus Google's AI Overview, which appears automatically in search results. The clinical tools have no public access for software, so the team entered the questions by hand through their websites.
What they found
On the exam-style questions, Gemini scored 97.4%, GPT 94.2%, Claude 90.2%, OpenEvidence 89.6%, and UpToDate 88.4%. On HealthBench, GPT scored 88.0 out of 100, Gemini 79.3, Claude 77.0, OpenEvidence 62.6, and UpToDate 61.3.
On the real clinician questions, the average scores out of 4 were:
- —Gemini: 3.62
- —GPT: 3.54
- —Claude: 3.52
- —Google AI Overview: 3.27
- —OpenEvidence: 3.24
- —UpToDate Expert AI: 3.17
The three general-purpose models formed the top group, and the clinical tools and Google's summary formed a second group. OpenEvidence scored lowest for clarity, at 2.84. Reviewers noted incomplete clinical content, safety-critical omissions, and disorganized answers, especially for OpenEvidence and Google AI Overview. UpToDate Expert AI declined to answer 19% of questions, compared with 1% to 3% for the other models and 6% for Google's summary. There was no significant difference between any of the tools in harmful content or hallucinations.
What the numbers actually mean
The gap on real questions was about a third to four-tenths of a point on a 4-point scale. The authors conclude the clinical tools are likely safe for routine use but are not better than general-purpose models on knowledge, communication, or agreement with expert clinicians. They suggest the weakness may be in how answers are communicated, not in what the tools know. They also note that looking up documents can hurt an AI's answer when the wrong material is retrieved. They call their results a snapshot of a fast-moving field, and they say very specialized medical tasks may still favor specialized tools.
A second study points the other way. It is a preprint, which means it was posted on June 27, 2026, before independent experts reviewed it. Its questions were 620 real queries that physicians had typed into OpenEvidence during one week in June. Physicians were emailed a survey, and 149 completed it, each matched to questions in their own specialty. They compared two answers side by side without knowing which AI wrote them. OpenEvidence won more comparisons than it lost against the three general-purpose models on all five measures, which were accuracy, clinical usefulness, source quality, verifiability, and completeness. The authors report this as a win difference, meaning the share of comparisons won minus the share lost. It ranged from about 25 points for accuracy to about 39 points for source quality. This preprint tested newer models, GPT-5.5, Gemini 3.1 Pro, and Claude Opus 4.8.
The preprint's authors point to several reasons the two studies disagree. The questions came from different places, an enterprise ChatGPT instance in the Nature Medicine study and OpenEvidence's own users in the preprint, and clinicians may use the two tools for different purposes. The Nature Medicine study had 12 reviewers from one institution who scored each answer on its own, while the preprint had 149 reviewers across 36 states who compared answers in pairs. The preprint's authors also say the result is not surprising for a tool tuned to its users' questions, and that it does not mean general-purpose models cannot answer such questions. They call untangling which difference matters most a job for future work, and this review cannot settle it either. The two studies also measured different things. The Nature Medicine study did not assess citation quality, while the preprint rated source quality and verifiability.
Both studies rated written answers. As described, neither measured what happened to patients. The ChatGPT for Healthcare deals also cover uses these studies did not test, such as administrative work. A signed contract and a funding round tell you how fast a product is spreading, and neither tells you how well it works.
Limitations worth knowing
- —The real-question test used only 100 questions and 12 reviewers. Agreement between reviewers on individual answers was only fair, though most disagreements were within one point.
- —The clinical tools were queried by hand through their websites, so hidden settings may have differed from how hospitals use them.
- —HealthBench was developed by OpenAI. The authors flag possible overlap with GPT's results and treat it as supplementary evidence.
- —The models were tested between September 2025 and February 2026, and newer versions now exist.
- —The study did not assess response speed or citation quality.
- —One Nature Medicine author reports consulting for Google, whose Gemini model scored highest, as well as equity in two companies, MarchAI and Artisight. The funders had no role in the study.
- —The preprint has not been peer reviewed. OpenEvidence helped design the data collection, ran the survey, and paid the physicians who responded. The authors report no affiliation with OpenEvidence and say they ran the analysis blinded to which AI was which. Of 12,000 physicians emailed, 149 completed the evaluation.
The bottom line
Contracts and funding are moving faster than independent evidence. In the independent comparison published in Nature Medicine, specialized medical AI tools did not beat general-purpose models on these tests, and a second unreviewed study, whose data collection was run by OpenEvidence, found the opposite. Neither measured patient outcomes. Until that evidence exists, an answer from any of these tools is something to check against the source.
Paper reviewed
Vishwanath K, Alyakin A, Ghosh M, et al. "General-purpose large language models outperform specialized clinical AI tools on medical benchmarks." Nature Medicine. 2026;32:2405-2409. Published online 2026 Jun 12. doi:10.1038/s41591-026-04431-5. Available free full text (open access) at: https://www.nature.com/articles/s41591-026-04431-5
Additional literature and sources referenced
Preprint, not peer reviewed: Feng J, Patel V, Heagerty P, et al. "Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries." arXiv:2606.28960. Posted June 27, 2026. https://arxiv.org/pdf/2606.28960
Industry and news reports:
- —Healthcare Finance News, "OpenAI launches ChatGPT for Healthcare at several large health systems": https://www.healthcarefinancenews.com/news/openai-launches-chatgpt-healthcare-several-large-health-systems
- —HIT Consultant, "Sheba Medical Center Partners with OpenAI in First International ChatGPT for Healthcare Deployment": https://hitconsultant.net/2026/07/28/sheba-medical-center-openai-chatgpt-for-healthcare-deployment/
- —OpenAI, "Healthcare organizations can now connect EHR and additional industry data to ChatGPT," September 1, 2026: https://openai.com/index/chatgpt-connects-health-records-and-healthcare-sources/
- —Fierce Healthcare, "OpenEvidence clinches $250M series D": https://www.fiercehealthcare.com/ai-and-machine-learning/openevidence-clinches-250m-series-d-rapidly-growing-its-reach-doctors
- —WOWTALE, reporting on the MSK and OpenEvidence integration, citing Forbes: https://en.wowtale.net/2026/09/28/235252/