Microsoft's MAI-DxO beats doctors on the hardest medical cases in a Science study

Every few months a new headline claims that AI is better than doctors. Most of those claims come from exam style tests, where a model reads a complete case and picks an answer. Real medicine does not work like that. A patient walks in with a complaint, and the doctor has to decide what to ask, which tests to order, how much they cost, and when there is enough evidence to commit to a diagnosis.

On Thursday, 22 October 2026, Microsoft AI published a paper in Science that tries to test exactly that process. The paper, "Sequential diagnosis with language models" by Harsha Nori, Eric Horvitz, Dominic King, Mustafa Suleyman and colleagues, describes two things: a new interactive benchmark called SDBench, and an AI system called the MAI Diagnostic Orchestrator, or MAI-DxO. Microsoft describes the result as a step toward what it calls medical superintelligence. TechCrunch covered the release the same day and added important context about where the gains come from.

The three things you need to know

1. The score. On 304 tough clinicopathological cases from the New England Journal of Medicine, MAI-DxO reached 83.2% accuracy. Generalist physicians who could use textbooks and the internet reached 36.1%, and 22.3% without references. On a second set of emergency department cases from the MIMIC-IV-ED database, the system scored 80.3% against 46.8% for doctors.

2. How it works. MAI-DxO acts like a panel of virtual doctors. The simulated experts ask questions, order tests, challenge each other's leading diagnosis over several rounds, and weigh every test against its estimated cost.

3. The catch. These were controlled, text-only benchmark cases, not real patients. The comparison was with individual generalist physicians, not specialists. And much of the improvement comes from strong underlying models such as OpenAI's GPT-5 and GPT-5.5. A prospective study with Beth Israel Deaconess Medical Center in Boston is the next step.

Why a new kind of benchmark

Language models already pass medical licensing exams and score highly on structured case vignettes. The authors argue that those tests can overstate how good a system really is. When a benchmark bundles the complaint, history, examination and all test results into a single prompt, it cannot measure whether a system asks the right questions, avoids ordering every test in the catalogue, updates its thinking as results arrive, or knows when to stop.

The Science editor's summary puts it plainly: patients "do not come in with a tidy write-up of key symptoms and pertinent laboratory findings." So the team built a test where the AI starts from very little and has to work for every piece of information.

Inside SDBench

SDBench turns 304 case records from the New England Journal of Medicine and 325 emergency department to admission encounters from MIMIC-IV-ED into interactive, step by step challenges. Each case begins with a short clinical vignette. From there, the diagnostician, either an AI agent or a human physician, can ask for history or examination findings, order a test, or commit to a final diagnosis.

Three agents run the game:

• The Gatekeeper answers requests from the underlying case record and only returns information that the specific question or test would really reveal. If a finding is missing from the record, it generates a neutral synthetic result so that missing data does not give away clues.
• The Diagnostic agent is the system or doctor being tested.
• The Judge compares the final answer with the reference diagnosis using a five level rubric written by physicians. Scores of 4 or 5, meaning management would not change, count as correct.

Cost is tracked as well. Each test is priced with CPT based charges from a public hospital chargemaster, and history and examination questions are treated as part of a fixed visit cost. That allows a direct comparison of accuracy and spending for doctors, plain models and the orchestrator.

How MAI-DxO thinks

MAI-DxO is not a new model. It is an orchestrator, a structured workflow that sits on top of existing language models and makes them behave like a small diagnostic team. According to the paper, it simulates a panel of clinical experts with complementary roles. The panel keeps an explicit list of possible diagnoses, challenges the leading hypothesis through critique, and estimates the value of each additional test before ordering it. It also aggregates diagnoses across independent runs.

The authors say this design is meant to reduce classic reasoning failures such as anchoring on the first idea or stopping too early. They also noticed a clear difference in style. Physicians in the study asked an average of 5.6 questions before ordering their first test. MAI-DxO with GPT-5 asked an average of 25.5 questions first. In other words, it does a lot of structured history taking before it spends money on tests.

The approach is model agnostic. Microsoft and TechCrunch say the team tested models from OpenAI, Google, Anthropic, Meta and DeepSeek, and across all models the orchestration raised accuracy on average by 7.9 percentage points on the NEJM cases and 4.4 points on the emergency cases.

The numbers in detail

The headline comparison is on the hard NEJM cases. For the physician study, 56 cases were held out from development. Practising primary care, hospital and acute care physicians completed them under two conditions: closed book and open book with search and reference tools. They spent an average of 11.9 minutes per case closed book and 18.1 minutes open book. Their average accuracy was 22.3% closed book and 36.1% open book. Even the single best physician, at 51.8%, was beaten by the system on both accuracy and cost.

The emergency cases are a useful second check, because they come from routine hospital records rather than cases picked for teaching. Doctors did better here, at 46.8% on average, and the best individual physician reached 62.5%. MAI-DxO without a budget limit reached 80.3%. Interestingly, the system performed about the same on both datasets, while doctors dropped sharply on the rare, complex NEJM cases.

Accuracy versus cost

The paper's main claim is not just higher accuracy, but a better trade off between accuracy and spending. Without budget limits, MAI-DxO with GPT-5 reached 83.2% at an average estimated cost of $5,431 per NEJM case. With budget constraints it reached 80.9% at $2,241 and 79.6% at $1,303. A configuration tuned for maximum accuracy reached 84.5% at $8,945. Physicians spent an average of $3,049 closed book and $4,725 open book.

According to Microsoft, no other approach tested was both more accurate and less costly than the orchestrator. In the paper's terms, MAI-DxO defined the Pareto frontier.

Where the gains really come from

TechCrunch's reporting adds a useful dose of realism. Much of the improvement comes from the underlying models themselves, and the best results came from OpenAI's GPT-5 and GPT-5.5. GPT-5 on its own already scored 78.9% on the NEJM cases, at roughly $8,400 per case according to TechCrunch. The orchestrator lifted that by about four points while cutting roughly $3,000 per case. On the emergency cases, TechCrunch reports that the added gain was small.

With the newer GPT-5.5, the picture changes. TechCrunch reports that the plain model scored 72% on the complex cases, while MAI-DxO on top of it reached nearly 87% at a slightly lower cost. The paper itself notes that the team re-ran the full evaluation with GPT-5.5 before publication without changing prompts or orchestration, and that the advantage over the baseline prompt grew. The authors read this as a sign that better models do not make careful orchestration less useful. Smaller models benefit too: TechCrunch notes that Google's Gemini 2.5 Flash jumped from 46.4% to nearly 69% with the orchestrator.

The limits the authors admit

This is the part that most headlines skip, and the Science paper is unusually direct about it.

• Text only. Doctors and AI interacted with a text based Gatekeeper, not with real patients or electronic health records. There were no images, waveforms or pathology slides.
• Idealised conditions. The cases did not include patients forgetting details, messy documentation or changes over time.
• Unusual case mix. NEJM cases are selected because they are hard and instructive, and both datasets exclude healthy or benign visits.
• No specialists. The comparison was with individual generalist physicians. The authors write that the results "do not establish superiority over specialists, multidisciplinary teams, or real-world clinical practice," and they expect the gap to narrow against specialists.
• Costs are estimates. Prices come from standardised US charges and do not capture invasiveness, patient discomfort or time to result.
• Diagnosis only. The Judge scores the diagnosis, not treatment, safety or follow up.

It is also worth noting that all authors are Microsoft employees who hold equity in the company, as the paper's competing interests statement says.

What comes next

Microsoft says the next step is a prospective study with Beth Israel Deaconess Medical Center in Boston. In that study, patients will interact with the system directly, and its assessments will be compared with physicians' independent assessments. Consenting patients receive standard care in parallel. Microsoft is also working with Mayo Clinic on a healthcare foundation model planned for release next year, and has published a companion paper, "The Building Blocks of Medical Superintelligence," in BMJ Digital Health and AI.

The SDBench code, including the Gatekeeper, Judge and cost estimator, is available under the MIT License on GitHub. The NEJM derived dataset is available from NEJM Group for non commercial research under licence.

Dominic King, vice president of health at Microsoft AI, told TechCrunch: "There is something very substantial and exciting here," calling it "the biggest study of its kind." Suleyman went further and told TechCrunch he expects systems to become "an order of magnitude better in capability within the next 24 to 36 months." Microsoft is not alone in this race. TechCrunch points out that Google published a Lancet study on its AMIE system with Beth Israel this month.

Why it matters

Diagnostic errors are a real problem in medicine, and so is over-testing. A system that both gets more answers right and orders fewer unnecessary tests would help on two fronts at once. The more interesting idea in this paper is not the headline number, but the method: an AI that is forced to ask, test and justify its choices step by step, with every decision carrying a price. That is much closer to how medicine works than a multiple choice exam.

At the same time, a benchmark is still a benchmark. A text interface with a perfect record behind it is very different from a crowded emergency room at 3 a.m. The Boston study will be the first real test of whether these results survive contact with actual patients.

Dany's take

I find this one of the more honest "AI beats doctors" papers I have read. Microsoft built a test that rewards asking the right questions and punishes ordering every test, and that alone is a step forward. The 83% versus 36% number is impressive, but I would not read it as "doctors are obsolete." The doctors were generalists working on some of the hardest teaching cases in medicine, through a text interface, without a team. What I take away is that a structured AI second opinion could be very useful for exactly those rare, confusing cases where a single doctor is most likely to miss something. I also like that the code is open, so others can check the claims. The real verdict comes from the hospital study. Would you want an AI second opinion like this? Tell me in the comments under the video.

What this means for you

For now, nothing changes at your doctor's office. MAI-DxO is a research system, not a product you can use, and it has not been tested on real patients yet. If you already use AI chatbots for health questions, this paper is a reminder of their biggest weakness: a plain chatbot answers what you typed, while good diagnosis depends on asking the right follow up questions. Treat any AI answer as a starting point for a conversation with a doctor, not as a diagnosis. If you work in healthcare or health tech, SDBench is worth a look as a more realistic way to evaluate diagnostic tools.

Sources

• Sequential diagnosis with language models (Nori et al., Science, 22 October 2026)
• Building medical superintelligence (Microsoft AI, 22 October 2026)
• Microsoft's Mustafa Suleyman says its AI outperforms doctors on hard cases as he chases medical superintelligence (TechCrunch, 22 October 2026)
• SDBench source code (Microsoft, GitHub)