Executive Summary

Key numbers

The first two are the values the paper put up. The last two say what conditions those values came out of.

Source: Science, volume 394, issue 6822 (October 22, 2026)

MAI-DxO is not a single model. It hands several virtual physicians different roles so that they put up hypotheses and argue against each other, and while that runs it weighs which test to order next and how far the bill has climbed. The DxO in the name comes from Diagnostic Orchestrator. Rather than producing a diagnosis in one shot, it stacks questions and tests in order and closes in on the answer. The paper's title, sequential diagnosis, names that procedure.

The results came in two sets. One is 304 cases from the clinicopathological conferences published in the New England Journal of Medicine, the other 325 emergency department records. The first group are cases that experts could not answer easily, which is how they ended up in a journal as material for discussion. The second group are ordinary records from patients who walked into an emergency department.

Put up the score of the best-performing physician instead of the average and MAI-DxO still sits above it. On the difficult cases that is 51.8% against 83.2%, and on the emergency department cases 62.5% against 80.3%. There is not much room to argue over the size of that gap itself.

The study also reports that the result holds when the model underneath changes. Gemini 2.5 Flash scored 46.4% working on its own, and once it ran inside the MAI-DxO procedure it climbed to somewhere around 69%. Microsoft leans on this when it describes the study as the performance of a procedure rather than of one particular model.

One place calls for care when citing figures. A preprint released in June 2025 carried 85% against 20% with 21 physicians, and articles that took those numbers over as they stood are still in circulation. Every figure in this article is from the final Science version of October 22, 2026.

So who ended up on the 36.1% side? Fifty-six physicians sat the test and all of them are generalists. No specialists took it, and neither did the multidisciplinary teams in which several departments see a patient together. Reference material was not blocked. The study ran both a condition with textbooks and the internet closed off and a condition with them open, and 36.1% is the average across the two. Take the closed condition on its own and it falls to 22.3%.

The first party to write this limitation down was the authors. The paper states that the result does not show the system to be better than specialists, multidisciplinary teams, or actual clinical practice. The expectation that the performance gap would narrow against specialists also came from the authors. This is not a criticism arriving from outside but a line drawn from the inside first.

There is one more condition to read alongside the numbers. All 15 names on the paper are Microsoft employees. The party that designed the test and the party that built the product scored on that test are the same. Corporate labs publishing their own technology in a journal is common and is not in itself a defect, but it cannot be left out of the question of who picked the comparison group.

An account from the company came out with it. Mustafa Suleyman, chief executive of Microsoft AI, described the result as one step on the road to medical superintelligence, forecasting performance indistinguishable from a person within a few years while adding that the clinician's place stays where it is. This is not a sentence from the paper but an explanation the company attached while announcing the research.

The test is called SDBench and it is made of 629 cases: 304 clinicopathological conference cases from NEJM and 325 emergency department records from a public dataset called MIMIC-IV-ED. Both sets are past cases in which the diagnosis was settled and written into a record.

These records were pieces of writing with the answer in them from the start. The researchers rebuilt them afterward into a step-by-step workup. A few lines on why the patient came in are shown first, and when a question is asked or a test is ordered, the matching information comes back one piece at a time. Every test carries a price and the cost accumulates. This is not a living patient watched in real time but a record whose ending is known, covered over and peeled back one layer at a time.

One condition applied to both sides equally. Neither could use physical examination or medical imaging. Both the AI and the physicians had to answer from written information alone, and on the question of fairness this part falls on the study's side.

Dominic King, vice president for health at Microsoft AI, said that medical AI evaluation has leaned on multiple-choice exams, that AI scores close to 100% there, and that real patients do not arrive carrying a tidy summary. That is the reason SDBench was built. Even so, the material SDBench is made of is also records already tidied up and filed into a journal and a dataset. SDBench did step away from multiple choice, but it did not become the same as the information a clinician meets in an exam room.

The distance between the test and the exam room is noted in the paper as a limitation as well. A controlled simulation, the incompleteness of real clinical records, and the absence of following a patient over time are all listed there. Microsoft has said it will pilot the system with Beth Israel Deaconess Medical Center starting from emergency department triage, and the start date has not been made public. As it stands, this study sits at a stage before any validation in the clinic.

The figure cited about as often as accuracy is testing cost. MAI-DxO is built so that cost accumulates with every test it orders, and it can also be run with a budget ceiling in place. Which setting a figure is taken from turns the conclusion over.

The setting that produced 83.2% is the one with no budget ceiling on it. In that setting MAI-DxO ordered $5,431 worth of tests per case, and for physicians on those same cases the estimate runs $3,049 to $4,725. On the emergency department cases it is $4,979 against $3,400, again higher on the AI side. Where the score came out highest, testing ran heavier than it did for the doctors.

The sentence that AI saved on testing costs too comes out of the setting with a budget ceiling. There the figure drops to between $1,303 and $2,241 per case. Both sentences come out of the same study and both are true. Pull the condition off and both turn false.

The nature of the cost is worth noting as well. These amounts are not actual hospital bills but an estimate built by putting a price tag on the list of tests ordered for each case and adding them up. The share that moves with country and insurance system for the same test is not in there.

What there is to take from this study is not only the medical conclusion. What a single number says and does not say is set by what it was measured against and by where the data came from. Drop those two and there is no way to read the figure, high or low. The same goes for 83.2%. It cannot be read without the fact that the other side was 56 generalists and the fact that the test was built out of 629 cases that had already been diagnosed.

A company that puts performance figures for its own AI in front of outsiders gets the same questions. Three things are worth having settled.

• What the number was set against. If it was people, who they are, how many of them there were, and under what conditions they worked has to sit right next to the number
• Who built the test. If the group that made the evaluation data also made the product being evaluated, saying so first is the better move. It is also the reason handing evaluation to an outside firm works as a business (see the company that grades AI on real work)
• Whether the best number came out of the setting that will actually be run. Accuracy and cost rarely reach their best in the same setting

The place Pebblous keeps checking in its work on data quality is the same one. When a report arrives saying a model score went up, the first questions are what data the score was measured on and whether that data has the same distribution as the data that will come in from the operating environment. Once the source and the makeup of the evaluation data go blurry, every figure stacked on top of them wobbles.

Thanks for reading this far. The figures in this article were checked against the paper in Science volume 394, issue 6822 of October 22, 2026 and the reporting that covered it. If you share AI performance figures inside your own organization, we would be glad to hear what form you use to write the comparison group and the source of the evaluation data alongside them.

Pebblous Data Communication Team
October 24, 2026

Academic Paper

• 1.Nori, H. et al. (2026). "Sequential diagnosis with language models." Science, 394(6822), 423-428.

Industry Coverage

• 2.Unite.AI (2026). "Microsoft Diagnostic AI Outperforms Generalist Physicians in Science Study."
• 3.The Next Web (2026). "Microsoft's MAI-DxO diagnostic AI outperforms doctors in Science study."