The number that traveled was 83%. In a paper published in Science on 22 October, Microsoft's MAI Health team reported that its AI Diagnostic Orchestrator, MAI-DxO, reached 83% diagnostic accuracy on 304 complex case records from the New England Journal of Medicine, against 36% for physicians working with textbooks and internet access and 22% for physicians working without them. On routine emergency-department presentations drawn from MIMIC-IV-ED the gap narrowed but held: 80% for the system, 47% for physicians. A cost-weighted configuration was reported at 81% on the NEJM set for less than half the workup cost of the comparison systems.
Those comparisons will be argued over. What is worth keeping either way is the benchmark underneath them, SDBench, because it changes what a test case is. Instead of handing over every finding at once and asking for a single answer, it opens each case with limited information and makes the model or the physician decide what to ask for next.
What the benchmark actually scores
The mechanics matter for reading the result. A Gatekeeper decides whether a requested piece of evidence — a lab value, an imaging study, a history question — exists in the source record, and reveals it if so. A Judge compares the final diagnosis against the published one. A cost estimator prices what was ordered. Each case yields three numbers rather than one: whether the answer was right, how much evidence was consumed, and what the workup would have cost.
That is a better shape than the licensing-exam items medical AI has leaned on. USMLE-style questions present a complete vignette and ask for one selection, so the skill being measured is retrieval against a knowledge base. Real diagnosis is sequential: form a hypothesis, pick the test with the highest expected value, revise when the result contradicts you, stop when marginal information no longer justifies marginal cost. A benchmark that scores the sequence can separate a system that reasons from one that guesses well.
The baseline is the finding
Which makes the human baseline the most interesting object in the result. Physicians scored 22% and 36% on the same cases, and the two conditions differ only in whether reference resources were available. That is a thin description of a comparison that ordinarily depends on time allowed, team size and access to consultation. The announcement does not say how long each physician had, whether they worked alone, whether they ordered tests through the same menu or free-form, or how a partially correct diagnosis was scored.
Retrospective construction is the other structural feature. A case record is written after the fact by the team that solved it; it is an ordered narrative, not the unordered sequence of requests a diagnostician actually made. Scoring against it rewards arriving at the published diagnosis, which is defensible and narrow at once. It also removes the physical examination, the observation that a patient's story shifts on the second telling, and the informal second opinion — the channels through which clinicians actually accumulate evidence.
Three questions for the paper
The first is contamination, and it is the one that could overturn the headline. NEJM case records are public prose that has very likely been in pretraining corpora. A sequential interface constrains what the model is told, not what it already knows: a model that recognizes a published case from its opening presentation can name the diagnosis on the first turn and then order a plausible workup backwards. The announcement reports no leakage analysis. The clean tests — hold out cases published after the training cutoff, or paraphrase away identifying details, and report the drop — are absent from the post, which is a reason to read the paper before quoting the figure.
The second is the grader. The Judge is itself a model comparing free-text diagnoses against a reference. Free-text equivalence is exactly where automated graders drift: "sepsis from a urinary source" and "pyelonephritis with bacteremia" may or may not be the same answer depending on the rubric. What the result needs is reported agreement between Judge and blinded clinician adjudication on a sample, plus an explicit rule for adjacent and partial diagnoses. Without that, some unknown fraction of 83% is grader variance.
The third is the action space. Evidence is requested from a menu the Gatekeeper can answer. If that menu bundles twelve chemistry values into one orderable panel, a system that orders the panel looks efficient next to one that had to ask for each value. Cost figures inherit the same granularity, because they come from a price table mapped onto menu items rather than from what a hospital bills. That does not invalidate the comparison, but part of the efficiency claim is a property of the interface.
Two cost ledgers, one headline
Cost needs its own paragraph because there are two of them and only one appears in the announcement. The paper's cost is the workup: the summed price of tests ordered. The other is inference. A panel of virtual physicians debating over several rounds, at every step of a sequential case, multiplies model calls per case by roughly an order of magnitude over a single-shot answer. The reported 81% at less than half the cost is a statement about tests, not about compute. They move in opposite directions as orchestration gets heavier, and a deployment decision needs both in one table.
What is reusable
The most portable claim — that gains survived a change of base model without altering the method — is a statement about scaffolding rather than weights, testable the obvious way: same panel, different models, same cases.
What it still lacks is a compute-matched single agent. Multi-agent debate has a long record of beating one-shot prompting and then losing most of that advantage to a single model given the same token budget for longer reasoning or more samples. A panel drawn from instances of one model is also a panel of correlated errors; role prompts change what each instance emphasizes, not what it knows. Until the table includes an equally expensive single-agent baseline, the defensible reading is that orchestration beats one shot, not that it beats a model allowed to think as long.
The genuinely useful deliverable is the harness: releasing Gatekeeper, Judge and cost-estimation code lets someone else put a different model through the same sequence and get a comparable number. Two caveats: the release covers select NEJM cases rather than all 304, and the real test is whether an outsider can run a new model end to end rather than inspect the components.
What would settle it
Four additions would move this from an impressive retrospective result to a durable one: a contamination-controlled split with the accuracy gap reported; a compute-matched single-agent baseline; cost reported in both ledgers; and an error taxonomy rather than an aggregate, because the clinically decisive question is whether the failures are dangerous misses or merely different-but-defensible answers. The announced prospective study, where patients interact with the system and results are scored against independent physician assessments, addresses what no vignette can: a record is not a patient.
The 83% will be quoted for a long time, and under the paper's conditions it is a real result. Benchmarks outlive leaderboards, though, and the durable contribution is the insistence that a diagnostic system be scored on the path it took — what it asked for, in what order, at what price — rather than on a lucky guess from a case it may have read before. The same harness can now answer whether a debating panel is the best way to walk that path. Someone only has to run it against a baseline that spent as much.
Sources
Microsoft AI, "Building Medical Superintelligence", 22 October 2026 — https://microsoft.ai/news/building-medical-superintelligence/
Science, doi:10.1126/science.aeh7930 — https://www.science.org/doi/10.1126/science.aeh7930
SDBench benchmark and evaluation code — https://microsoft.github.io/sdbench
BMJ Digital Health and AI, "The Building Blocks of Medical Superintelligence" — https://bmjdigitalhealth.bmj.com/content/2/1/e000395
MIMIC-IV-ED, PhysioNet — https://physionet.org/content/mimic-iv-ed/