Today, that benchmark is being challenged in a way few anticipated. A new wave of large language models (LLMs) is not just assisting clinicians but, in some cases, outperforming them in diagnostic reasoning.

The latest Harvard study offers one of the most comprehensive evaluations yet, comparing AI systems directly with hundreds of physicians across multiple real-world and simulated clinical scenarios. The findings are striking and, at times, unsettling.

A turning point in medical intelligence

The research set out to test how well an advanced LLM could handle difficult diagnostic tasks. These included classic clinical case vignettes, structured medical reasoning exercises, and even real emergency room cases drawn from hospital records.

Across six separate experiments, the AI model consistently matched or exceeded physician performance. In controlled diagnostic cases, it achieved near-perfect accuracy, with median scores reaching 97 percent. By comparison, earlier models and human clinicians scored noticeably lower.

This is not a marginal improvement. It signals a shift in how machine intelligence is approaching one of the most complex cognitive tasks in medicine: clinical reasoning.

Real-world performance under pressure

The most compelling part of the study came from its real-world testing. Researchers analysed 76 emergency room cases, evaluating how both physicians and AI generated second-opinion diagnoses at three critical stages: initial triage, physician assessment, and hospital admission.

The results revealed a clear pattern. The AI system outperformed both experienced physicians and earlier AI models at nearly every stage.

At the earliest and most uncertain stage, initial triage, the model correctly identified or closely approximated the diagnosis in 67.1 percent of cases. Physicians, by contrast, achieved between 50 and 55 percent. As more clinical information became available, performance improved across the board, but the AI maintained its lead.

This early-stage advantage is particularly significant. In emergency settings, decisions made with limited information can determine patient outcomes. The ability to reason accurately under uncertainty has long been considered a uniquely human strength. That assumption is now under pressure.

Beyond accuracy: reasoning quality

The study did not focus solely on correct answers. It also examined how well the AI explained its reasoning, prioritised diagnoses, and recommended next steps.

Using validated scoring systems, researchers found that the model demonstrated strong performance in structuring differential diagnoses and suggesting appropriate tests. In many cases, its reasoning was rated as equal to or better than that of clinicians.

Another important measure involved identifying “cannot-miss” diagnoses, conditions where failure to recognise them could lead to severe consequences. Here, the AI performed on par with or slightly better than existing models and showed less variability than human clinicians, whose estimates often varied widely.

Why is AI performing so well?

Several factors help explain this performance.

First, LLMs are trained on vast amounts of medical literature and clinical data, allowing them to draw from a broader knowledge base than any individual physician. Second, they are not subject to cognitive biases, fatigue, or stress, all of which can affect human decision-making, especially in high-pressure environments like emergency departments.

Finally, these models excel at pattern recognition and probabilistic reasoning. When presented with symptoms, test results, and patient history, they can rapidly evaluate multiple possibilities without narrowing prematurely.

But there are limits

Despite the impressive results, the study highlights important limitations.

The AI’s performance was based entirely on text-based data. Real clinical practice involves far more, including visual cues, patient behaviour, and subtle physical signs that cannot easily be captured in written form.

There is also the issue of “clean” data. Many of the test cases were well-structured, whereas real-world clinical information is often incomplete, messy, and ambiguous. This raises questions about how well AI systems will perform outside controlled environments.

Another concern is interpretability. Even when the AI provides correct answers, understanding how it arrives at those conclusions remains a challenge. For healthcare systems, this lack of transparency can be a barrier to trust and adoption.

Implications for the future of healthcare

The findings point to a future where AI is not just a support tool but an active participant in clinical decision-making.

If used correctly, these systems could reduce diagnostic errors, speed up decision-making, and improve access to high-quality care, particularly in resource-limited settings. In countries like Bangladesh, where specialist doctors are scarce, such technology could have a transformative impact.

However, integration will require careful planning. Healthcare systems must invest in infrastructure, develop clear regulatory frameworks, and ensure that clinicians are trained to work alongside AI rather than be replaced by it.

A shift, not a replacement

The study does not suggest that doctors are becoming obsolete. Rather, it signals a shift in the nature of medical expertise.

AI can process information at scale and with consistency. Humans bring judgement, empathy, and the ability to interpret complex, non-verbal signals. The most effective future model of care will likely combine both.

What is clear, however, is that the balance is changing. The long-held assumption that human clinicians are the ultimate authority in diagnostic reasoning is now being challenged by machines that can match, and sometimes exceed, their performance.

The question is no longer whether AI can think like a doctor. It is how doctors will adapt to thinking alongside AI.