AI health chatbots can miss medical emergencies, study warns patients
NEW YORK, UNITED STATES — As large language models (LLMs) become a popular source of medical guidance, researchers warn that relying on these tools without professional oversight can put patients in danger.
An ARISE report revealed that more than one third of American adults now turn to artificial intelligence (AI) tools such as ChatGPT Health and Claude for health advice, but new study from the Icahn School of Medicine at Mount Sinai and insights from experts at Harvard Medical School showed that these systems can risk serious conditions, leaving gaps that only human clinicians can safely address.
AI chatbots misjudge medical emergencies and crisis risks
A study published in Nature Medicine found that ChatGPT Health often failed to direct patients to emergency care, especially in nuanced medical situations.
“While the tool generally handled clear-cut emergencies correctly, it under-triaged more than half of cases that physicians determined required emergency care,” said lead author Ashwin Ramaswamy, MD.
Scenarios ranged from asthma attacks to severe allergic reactions, revealing that AI may reassure patients even when danger is present.
The research also flagged inconsistent suicide-risk alerts.
“The system’s alerts were inverted relative to clinical risk, appearing more reliably for lower-risk scenarios than for cases when someone shared how they intended to hurt themselves,” said Dr. Girish Nadkarni of Mount Sinai.
Experts emphasized that patients experiencing worsening symptoms or mental health crises require immediate professional treatment instead of using AI chatbots as their primary solution.
In fact, a report from ECRI, a patient safety organization (PSO), named AI chatbots as the leading health technology hazard for 2026. This warning highlights how digital tools create operational challenges to the United States health systems that extend beyond traditional IT oversight.
Safely integrating AI tools in global healthcare systems
Healthcare providers face challenges as patients increasingly bring AI into consultations, and some patients lack the AI literacy to use it safely.
“One of the challenges with LLMs is they’re potentially incredibly helpful, but they’re not actually intelligent,” said Dr. Adam Rodman, director of AI programs at Beth Israel Deaconess Medical Center.
“If you effectively ask the wrong questions, use them the wrong way, you can get useless or even harmful information,” Rodman added.
For U.S. hospitals, clinics, and health systems, the problem creates a need for human oversight, structured escalation pathways, and operational redesign, areas that outsourced clinical teams and non-clinical teams can address.
Providers can pull in trained staff for AI-assisted patient triage, data management, and follow-up support, ensuring that AI complements with the tasks rather than replaces professional care.
Experts agree that integrating AI safely requires institutional frameworks rather than leaving patients to navigate public chatbots.
“We actually have to embrace the technology and bring it into our health systems, because if we don’t, people are going to use it on their own,” Rodman said.
The fast development of AI health tools requires hospitals and their providers to maintain patient safety while they adopt new technologies through staff expertise and their choice to outsource services while optimizing care workflows.
