According to a new study, artificial intelligence models achieved better results than physicians in medical decision-making in emergency care settings.

Researchers from Harvard Medical School and Beth Israel Deaconess Medical Center compared the performance of artificial intelligence and doctors across a wide range of tasks related to clinical reasoning.

They found that large language models (LLMs) outperformed physicians in several areas, including decision-making in emergency departments based on available data, identifying the most likely diagnoses, and selecting further patient management strategies.

“We tested the AI model across nearly every conceivable benchmark, and it outperformed both previous versions and our physician benchmarks,” said Arjun Manrai, co-lead author of the study and professor at Harvard Medical School.

“However, this does not mean AI will automatically improve the quality of care: how and where it should be applied remains insufficiently studied, and we urgently need rigorous prospective studies to evaluate AI’s impact on clinical practice,” Euronews reports.

Researchers first evaluated o1-preview, a reasoning model developed by OpenAI and released in 2024. The model was presented with a series of clinical cases, including published case reviews and real patient cases from emergency departments.

The AI outperformed physicians in most experiments, particularly in reasoning about treatment strategies, clinical thinking, documentation, and functioning in real-world emergency department conditions with limited information.

“The models are becoming increasingly capable. We used to evaluate them with multiple-choice tests, but now they consistently score close to 100 percent, and we can no longer track progress because we’ve essentially hit the ceiling,” said co-author Peter Brodeur, a clinical research fellow at Harvard Medical School and internal medicine physician at Beth Israel Deaconess Medical Center.

In one of the tests, researchers asked language models — o1 and GPT-4o — to assess patients at different stages of the standard emergency department workflow, from early triage to hospitalization decisions.

At each stage, the models were given only the information available at that specific moment and were asked to identify likely diagnoses and recommend further actions.

The largest gap between AI and physicians was observed during triage, when patient information was the most limited.

Like physicians, the AI models improved their diagnostic accuracy as additional information became available.

“While the use of AI to support clinical decision-making is sometimes viewed as risky, broader adoption of such tools could help reduce the human and financial costs associated with diagnostic errors, delays, and lack of access to care,” the authors wrote.