Microsoft’s MAI-DxO Turns “AI Beats Doctors” Into a Cost-Control Claim, Because Plain GPT-5 Already More Than Doubles the Open-Book Doctor Score

Microsoft AI’s diagnostic system, MAI-DxO, scored 83.2% on 304 New England Journal of Medicine case records and 80.3% on 325 emergency department cases in a paper published today in Science, against 36.1% and 46.8% for the generalist physicians it was compared with. Microsoft AI’s announcement post turns that into a system that is “more than twice as accurate as physicians using textbooks and the internet” and “nearly four times as accurate when they diagnose without those resources,” and TechCrunch reports that CEO Mustafa Suleyman called the result a step toward “medical superintelligence.”

The post also embeds the data behind its two charts, one row per system, and that data tells a narrower story than the headline. We pulled it, re-derived every number we use, and read it next to the 2025 preprint that introduced the benchmark. The short version:

• Plain GPT-5, with no orchestrator, scores 78.9% on the NEJM cases, 2.2 times the open-book doctors. Twelve of the 15 off-the-shelf models in that chart beat the doctors’ average.
• What MAI-DxO adds on those cases is mostly cost control. Against GPT-5, the 83.2% configuration (the one with no budget limit) is 4.3 points higher and 35% cheaper, and the $1,000-budget configuration matches GPT-5’s accuracy (79.6% against 78.9%) at about 15% of the spending. On the emergency department cases, no MAI-DxO configuration beats GPT-5 on both accuracy and cost.
• The dollars are simulated spending on visits and tests, not the cost of running the AI, which is how TechCrunch glosses them.
• “Half or even one third of the cost” describes the budgeted configurations. The 83.2% configuration spends 15% more per case than the open-book doctors.
• The doctor baseline is 15 to 18 people per group, each of whom attempted between 6 and 69 cases, and no plotted NEJM doctor attempted more than 56.

One caveat shapes everything below: Science’s site answered our automated fetches, from both a plain HTTP client and a headless browser, with a Cloudflare challenge, so we could not read the paper’s full text. This analysis rests on the abstract (read through Crossref’s API), Microsoft AI’s post and its chart data, the 2025 arXiv preprint, the Microsoft Research page for a companion paper, and TechCrunch’s report. Where a point depends on something we could not see, we say so.

What Microsoft published, and how the benchmark works

The paper, “Sequential diagnosis with language models,” has 15 authors, including Suleyman, Dominic King and Eric Horvitz. Its abstract describes “an interactive benchmark and an artificial intelligence (AI) system that simulates a panel of physicians,” reports the system “outperforming generalist physicians (36.1 and 46.8%, respectively),” and adds that the findings “were obtained in a controlled benchmark setting.” The 36.1% and 46.8% match the open-book doctor rows in the charts on Microsoft AI’s post, so the abstract compares against the open-book doctors, the stronger of the two NEJM human baselines.

The benchmark

The benchmark, SDBench, comes from a June 2025 preprint (arXiv 2506.22405, last revised July 2, 2025). It turns published NEJM clinicopathological conference cases into a conversation. A physician or an AI starts from a short vignette, asks a “gatekeeper” model for history, examination findings and tests, and commits to one final diagnosis. A judge model grades that diagnosis against the published answer, and a cost estimator prices the workup: a flat $300 per visit for questions and examination findings, plus test prices matched through procedure codes to a 2023 price table from a large U.S. health system.

MAI-DxO is an orchestrator rather than a model. It runs a panel of agents with defined physician roles that debate before the panel asks a question, orders a test or commits to a diagnosis, and an optional budget tracker lets the panel cancel tests after seeing their estimated cost. The preprint tests variants from “instant answer” (a diagnosis from the vignette alone) through “questions only” and budgeted runs to an ensemble of several panels, and says MAI-DxO was “primarily developed and optimized using GPT-4.1, but is designed to be model-agnostic.”

The two charts on Microsoft AI’s post are labeled “Models evaluated October 2025.” One covers 304 NEJM cases and the other 325 emergency department cases built from MIMIC-IV-ED. Each plots nine MAI-DxO configurations, 15 (NEJM) or 13 (emergency department) off-the-shelf models, the doctors’ averages and a dot for every individual doctor, and each embeds its data as a table. The script at the end of this article downloads the page, decodes that data and prints every number used below.

Plain GPT-5 already more than doubles the open-book score

Table 1. NEJM cases (304). Selected rows from the chart data on Microsoft AI’s post; the 95% interval and the cost per correct diagnosis are our arithmetic.

Open-book doctors scored 36.1% after spending an average of $4,724 per case on visits and tests, and closed-book doctors scored 22.3% at $3,049. GPT-5, run on its own with what TechCrunch calls “a short, basic prompt,” scored 78.9%: 2.2 times the open-book number and 3.5 times the closed-book one. Twelve of the 15 off-the-shelf models beat the open-book average; the three that did not are GPT-3.5 turbo, GPT-4.1 nano and Gemma 3. The “more than twice as accurate” claim therefore holds for a model Microsoft did not build and did not orchestrate. TechCrunch draws the same conclusion from the paper: “The study’s own data suggests that much of the improvement comes from more powerful underlying AI models, with Microsoft’s orchestration system adding gains on top.”

Table 2. Emergency department cases (325), same layout.

The emergency department chart shows a smaller gap with the same shape. Ten of the 13 off-the-shelf models beat the doctors’ 46.8%, and GPT-5 reaches 76.6%, 1.6 times the doctors, against MAI-DxO’s 80.3%, the 1.7 times in Microsoft AI’s post.

What the orchestrator adds is mostly cost control

The orchestrator’s contribution is easiest to see against GPT-5, the one off-the-shelf model whose numbers TechCrunch pairs with DxO: “On its own with a short, basic prompt, GPT-5 scored 78.9% on complex cases at about $8,400 per case. DxO cut that cost by about $3,000 but raised accuracy by only about four points.” Those numbers match the chart’s GPT-5 row and its “no budget” MAI-DxO row ($8,409 to $5,431, and 78.9% to 83.2%). The chart does not say which model sits behind the MAI-DxO rows, so we treat GPT-5 as the backing model for this comparison and assume the budgeted rows use the same one.

Table 3. Each MAI-DxO configuration against plain GPT-5 on the same cases (our arithmetic from the chart data).

On the NEJM cases, five of the nine MAI-DxO configurations are at least as accurate as GPT-5 and cost less, from the $1,000-budget setting (79.6% at $1,303, 85% cheaper) to the unconstrained one (83.2% at $5,431, 35% cheaper). The accuracy differences are small next to the sampling noise. On 304 cases a 95% interval for a score near 80% runs about 4 to 5 points either side (our arithmetic, treating each case as independent; a paired test needs per-case results, which the chart does not include), and the five budget settings from $1,000 to $16,000 span only 77.3% to 80.9%. Budget barely moved accuracy on these cases. It moved spending.

On the emergency department cases the exercise comes out differently. Accuracy climbs with budget, from 63.4% at $1,000 to 76.0% at $16,000, and no configuration beats GPT-5 on both axes: the $16,000 setting is 0.6 points lower and 13% cheaper, and the unconstrained one is 3.7 points higher and 17% more expensive. TechCrunch’s summary, “little improvement in either accuracy or cost,” fits those rows.

Results outside the chart data

TechCrunch also reports results that are not in the chart data. With GPT-5.5, which it describes as released in April 2026, DxO raised accuracy on the complex cases “from 72% to nearly 87% at a slightly cheaper cost,” while on the emergency cases both versions scored “about 85% at a similar cost.” With Gemini 2.5 Flash the score went from 46.4% on its own to “nearly 69% with DxO,” still below GPT-5 with no orchestrator. In TechCrunch’s words, “the gap between the weaker and stronger models was larger than the boost DxO provided,” and it relays Microsoft’s explanation that accuracy gains naturally shrink as models approach the theoretical maximum. The newer model’s bare 72% sits below GPT-5’s 78.9% in the chart, so its larger boost is measured from a lower starting point. We could not check any of these figures against the paper.

“Half or even one third of the cost” belongs to the budgeted settings

Microsoft AI’s post says MAI-DxO manages its accuracy “at half or even one third of the cost.” The chart data support that for the budgeted configurations. On the NEJM cases the $1,000-budget setting spends $1,303 per case, 28% of the open-book doctors’ $4,724, and the $4,000 setting spends $2,241, 47%. But the 83.2% in the abstract is the unconstrained configuration’s score, and that configuration spends $5,431: 15% more than the open-book doctors and 78% more than the closed-book ones. The highest-scoring configuration, the 84.5% ensemble, spends $8,945, 1.9 times the open-book doctors. On the emergency department chart the unconstrained configuration spends $4,979 against the doctors’ $3,400, and only the $8,000 setting and below (74.5% at $3,128) come in under the doctors’ spend. The post does not say which configuration its sentence refers to. The budgeted ones satisfy both halves (the $4,000 setting scores 80.9% at 47% of the doctors’ spend), but the headline 83.2% belongs to a configuration that spends more than the doctors do, so the 83.2% and the savings cannot be attached to each other.

The dollars measure the workup, not the AI

TechCrunch glosses the cost as “the estimated cost of running the AI on each case.” The first chart is titled “Diagnostic Accuracy vs Care Administration Cost (USD)” and its table column reads “Average cost of diagnostic tests per case (USD),” and the preprint’s cost model is the $300 visit fee plus test prices from a hospital price table. That is why the “instant answer” configuration costs $0 and the “questions only” one costs exactly $300, “the cost of a single physician visit” in the preprint’s words. The dollars measure what the simulated workup would cost a health system. They do not include the cost of running a panel of model calls (the preprint describes the budget tracker as “a separately orchestrated language model call,” and the ensemble runs several panels) or of paying for clinician time. A deployment budget would have to add both.

Dividing spending by accuracy gives a figure a buyer can compare (our arithmetic; the last column of Tables 1 and 2). On the NEJM cases the open-book doctors come to $13,087 per correct diagnosis, GPT-5 to $10,658, the unconstrained MAI-DxO to $6,527 and the $1,000-budget setting to $1,637. On the emergency department cases the doctors come to $7,266, GPT-5 to $5,536, the unconstrained setting to $6,201 and the $4,000 setting to $3,598. That is a real advantage for budgeted MAI-DxO on both charts, measured in simulated test spending.

The doctor baseline is a small, uneven group

Table 4. The human groups as plotted in the two charts.

The preprint says its doctors were 21 physicians practicing in the US or UK, with “a median of 12 years” of experience, of whom “17 were primary care physicians and four were in-hospital generalists,” and that they were “explicitly instructed not to use external resources,” including search engines and language models. The charts on Microsoft AI’s post show three groups of 15 to 18 people. Each person attempted between 6 and 69 cases, and in all three groups the published average tracks the attempt-weighted mean: weighting the 16 plotted NEJM open-book people by their cases gives 36.0%, against 36.1% published, while the plain mean of the 16 is 33.8%. No individual in any group reached the accuracy of GPT-4.1 on the same chart (64.1% on NEJM, 67.4% on the emergency cases); the best open-book doctor scored 51.8% on NEJM and 62.5% on the emergency cases. So how the averages are computed does not change the finding for GPT-4.1 and the stronger models.

Two other things stand out. First, every plotted NEJM doctor attempted at most 56 cases, the size of the preprint’s hidden test set (the 56 most recent cases, from which, per the preprint, the physicians’ cases were drawn). If that holds in the Science paper, the doctors were graded on at most 56 of the 304 cases on which the models were graded. The preprint reports “comparable absolute performance on the test set” for MAI-DxO, next to the 248 validation cases, which would limit the effect, but the charts do not say whether the models were also scored on that subset. Second, the comparison is to individual generalists working alone in a text chat. TechCrunch adds: “Cases this complex are usually handled by specialists or physician teams, who weren’t part of the comparison.”

A 39.5% “instant answer” shows what the NEJM benchmark rewards

The NEJM chart’s cheapest rows are the most striking ones. A MAI-DxO panel that answers from the short vignette alone, with no questions and no tests, scores 39.5% at $0, above the 36.1% that open-book doctors reached after spending $4,724 per case. Let the same panel ask questions without ordering any tests and it scores 62.8% for $300, 1.7 times the open-book doctors’ average. On the emergency department chart the same two rows read 19.4% and 42.2%, both below the doctors’ 46.8%, and the unconstrained panel gains 60.9 points over its instant answer (19.4% to 80.3%), against 43.7 points on NEJM (39.5% to 83.2%).

That says as much about the NEJM benchmark as about the doctors: most of the top score (62.8 of 83.2 points) is available from the vignette plus questions, before any test is ordered. The preprint concedes that “the original NEJM cases are accessible online,” holds out the 56 most recent cases to check for memorization, and reports performance on them comparable to the rest. That is a reasonable check, not proof, and the charts do not split the 39.5% by publication date. The emergency department chart is where the sequential part visibly matters: tests and follow-up questions lift the score from 19.4% to 80.3%.

What the numbers cannot say yet

A model plays the patient and a model grades the answer

In the preprint, the gatekeeper that answers the diagnostic agent’s questions was changed to return “realistic synthetic findings for queries not covered in the original text,” and the judge that decides whether a diagnosis is correct is an o3 model working from a physician-written rubric, with a score of 4 or more on a five-point scale counting as correct. The authors validated both: physicians reviewed 508 gatekeeper responses, and physician grades agreed with the judge’s at a Cohen’s κ of 0.70 on 56 diagnoses from their most accurate system and 0.87 on 56 human-written ones. Those are the preprint’s descriptions; the Science paper may have changed them, and we could not check. They do mean the encounter, the findings and the grading are all produced by models, with physician spot checks.

The models predate the newest ones, and none is a Microsoft model

The charts are labeled “Models evaluated October 2025.” None of the off-the-shelf models in either chart is a Microsoft MAI model, and TechCrunch says DxO’s best results came with OpenAI’s GPT-5 and GPT-5.5. Our earlier post on Microsoft’s MAI models replacing OpenAI in its own products covers the push to put the company’s own models in its products; for this health result, the strongest engine reported is OpenAI’s. TechCrunch also reports that Suleyman told it Microsoft is working with the Mayo Clinic to train a healthcare foundation model, with a release planned for next year.

No real patients yet

The abstract itself calls the findings “obtained in a controlled benchmark setting,” and Microsoft AI’s post says “Benchmarks matter, but the true test is performance in real clinical environments.” The post and TechCrunch describe the next step: a partnership with Beth Israel Deaconess Medical Center in Boston in which, per TechCrunch, patients receive standard care and those who consent also engage with DxO, and Microsoft and its partners compare DxO’s recommendations with the human doctors’. TechCrunch notes that Google published a clinical study with the same hospital in The Lancet this month, evaluating its AMIE diagnostic system in real-world urgent care visits while physicians monitored the interactions in real time.

Microsoft’s own bar for “medical superintelligence” is higher

Microsoft Research’s page for the companion paper that Microsoft AI’s post links, “Medical Superintelligence: Foundations and Aspirations” (the post says it was published in BMJ Digital Health and AI), states: “crossing the threshold to medical superintelligence will require more than superhuman performance on isolated tasks. It will require systems that harness AI, independently and in collaboration with humans, to surpass today’s strongest human-led health systems in integrated medical reasoning across the full arc of care.” By that definition, the Science result is evidence about an isolated task. TechCrunch reports that Suleyman called the result a step toward medical superintelligence, and the evidence fits that wording better than a claim that the threshold has been crossed.

Questions to ask when a result says “beats doctors”

The same checks apply to any vendor claim that an AI system beats experts, and they are cheap when the chart data is public. We have run versions of this audit on Microsoft’s ThinkingBox agent benchmark and on Google’s wet-lab agents, and in both the useful details sat in the supporting figures and tables rather than in the headline.

1. Which row is the base model alone, and how far is it from the headline? Here GPT-5 scores 78.9% against the headline 83.2%.
2. Which configuration produced the accuracy and which produced the savings? The 83.2% belongs to the unconstrained configuration; “half or even one third of the cost” belongs to budgeted ones.
3. What does the cost axis measure? Here, simulated visit and test prices, not inference or clinician time.
4. Who is the human baseline, how many cases did each person attempt, and were people and models scored on the same cases? Here, 15 to 18 people per group, at most 56 NEJM cases each.
5. Is there a per-case table? Without one, nobody outside the lab can run a paired test on a four-point difference.

The bottom line

None of this makes the paper weak. It prices the workup instead of scoring accuracy alone, it adds an open-book baseline (the preprint’s doctors were told not to use external resources), it compares 15 off-the-shelf models, and Microsoft AI’s post embeds the chart data so anyone can check it. Microsoft’s planned prospective study is the test that can show whether an orchestrator changes care. What the abstract and the chart data support today is a narrower sentence than the headline: on these text-based benchmarks, off-the-shelf frontier models already outscore a small group of generalists, and Microsoft’s orchestrator makes that result cheaper to reach in simulated test spending.

Reproduce the numbers

The script below needs Python 3.10 or later and the requests package (pip install requests). It downloads Microsoft AI’s post, decodes the two embedded charts and prints the figures used above. The output shown is from October 22, 2026; if Microsoft changes the page, the output may differ.

"""dxo_check.py: re-derive the numbers used in this article from the figure data embedded in Microsoft AI's post (as of 2026-10-22)."""
import base64
import json
import math
import re

import requests

URL = "https://microsoft.ai/news/building-medical-superintelligence/"
UA = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36"}


def figures():
    page = requests.get(URL, headers=UA, timeout=60).text
    found = {}
    for blob in re.findall(r"PCFET0NUWVBF[A-Za-z0-9+/=]{200,}", page):
        html = base64.b64decode(blob + "=" * (-len(blob) % 4)).decode("utf-8", "replace")
        m = re.search(r"var DATA=", html)
        if m:
            data, _ = json.JSONDecoder().raw_decode(html[m.end():])
            found[data["label"]] = data
    return found


def wilson(k, n, z=1.96):
    p, d = k / n, 1 + z * z / n
    c = p + z * z / (2 * n)
    h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n))
    return (c - h) / d * 100, (c + h) / d * 100


for label, d in figures().items():
    n = d["cases"]
    systems = {"MAI-DxO " + r["n"]: r for r in d["dxo"]}
    systems.update({r["n"]: r for r in d["base"]})
    exact = sum(round(round(r["a"] / 100 * n) / n * 100, 1) == r["a"] for r in systems.values())
    print(f"== {label}: {n} cases, {exact} of {len(systems)} accuracies are exact k/{n} ratios")
    for group, g in d["clin"].items():
        people = g["ind"]
        print(f"   {group}: {len(people)} people, {g['avg']['g']} graded cases, accuracy per person "
              f"{min(a for _, a, _ in people)} to {max(a for _, a, _ in people)}, cases per person "
              f"{min(c for *_, c in people)} to {max(c for *_, c in people)}, average {g['avg']['a']}% at ${g['avg']['c']:,.0f}")
    doc, gpt5 = d["clin"]["openbook"]["avg"], systems["GPT-5"]
    beat = [name for name, r in systems.items() if not name.startswith("MAI-DxO") and r["a"] > doc["a"]]
    print(f"   off-the-shelf models above the open-book average: {len(beat)} of {sum(not s.startswith('MAI-DxO') for s in systems)}")
    wins = [name for name, r in systems.items() if name.startswith("MAI-DxO") and r["a"] >= gpt5["a"] and r["c"] <= gpt5["c"]]
    print(f"   MAI-DxO rows at least as accurate as GPT-5 and no costlier: {len(wins)} {wins}")
    rows = [(name, r["a"], r["c"], wilson(round(r["a"] / 100 * n), n)) for name, r in systems.items()]
    rows += [(f"Doctors, {group.replace('book', ' book')}", g["avg"]["a"], g["avg"]["c"], None) for group, g in d["clin"].items()]
    print(f"   {'system':<26}{'acc':>6}  {'95% interval':<14}{'cost':>7}{'per correct':>13}{'pts vs GPT-5':>14}{'cost vs GPT-5':>15}{'cost vs doctors':>17}")
    for name, acc, cost, ci in sorted(rows, key=lambda row: row[1]):
        span = f"{ci[0]:4.1f} to {ci[1]:4.1f}" if ci else "n/a"
        print(f"   {name:<26}{acc:>5.1f}%  {span:<14}{cost:>7,.0f}{cost / (acc / 100):>13,.0f}"
              f"{acc - gpt5['a']:>+14.1f}{round(100 * (cost / gpt5['c'] - 1)):>+14d}%{round(100 * (cost / doc['c'] - 1)):>+16d}%")

The output. The 95% interval, cost per correct diagnosis and the three comparison columns are our arithmetic, not Microsoft’s; the intervals are Wilson intervals that treat each case as an independent trial.

== CPC: 304 cases, 24 of 24 accuracies are exact k/304 ratios
   closedbook: 18 people, 764 graded cases, accuracy per person 0.0 to 37.3, cases per person 14 to 56, average 22.3% at $3,049
   openbook: 16 people, 662 graded cases, accuracy per person 16.7 to 51.8, cases per person 6 to 56, average 36.1% at $4,724
   off-the-shelf models above the open-book average: 12 of 15
   MAI-DxO rows at least as accurate as GPT-5 and no costlier: 5 ['MAI-DxO $1k budget', 'MAI-DxO $4k', 'MAI-DxO $8k', 'MAI-DxO $16k', 'MAI-DxO No budget']
   system                       acc  95% interval     cost  per correct  pts vs GPT-5  cost vs GPT-5  cost vs doctors
   Doctors, closed book       22.3%  n/a             3,049       13,673         -56.6           -64%             -35%
   GPT-3.5 turbo              22.4%  18.0 to 27.4    1,565        6,985         -56.5           -81%             -67%
   GPT-4.1 nano               25.7%  21.1 to 30.8    2,111        8,214         -53.2           -75%             -55%
   Gemma 3                    30.6%  25.7 to 36.0    3,130       10,227         -48.3           -63%             -34%
   Doctors, open book         36.1%  n/a             4,724       13,087         -42.8           -44%              +0%
   MAI-DxO Instant answer     39.5%  34.1 to 45.1        0            0         -39.4          -100%            -100%
   DeepSeek R1                43.8%  38.3 to 49.4    2,743        6,262         -35.1           -67%             -42%
   Gemini 2.5 Flash           46.4%  40.9 to 52.0    4,792       10,327         -32.5           -43%              +1%
   GPT-4.1 mini               52.3%  46.7 to 57.9    3,064        5,859         -26.6           -64%             -35%
   Llama 4 Maverick           56.6%  51.0 to 62.0    3,418        6,039         -22.3           -59%             -28%
   GPT-OSS 120B               59.9%  54.3 to 65.2    6,274       10,475         -19.0           -25%             +33%
   MAI-DxO Questions Only     62.8%  57.3 to 68.1      300          478         -16.1           -96%             -94%
   GPT-4.1                    64.1%  58.6 to 69.3    4,033        6,292         -14.8           -52%             -15%
   Claude 4.5 Sonnet          69.4%  64.0 to 74.3    7,035       10,137          -9.5           -16%             +49%
   GPT-5 mini                 71.7%  66.4 to 76.5    6,529        9,106          -7.2           -22%             +38%
   Gemini 2.5 Pro             73.0%  67.8 to 77.7    4,518        6,189          -5.9           -46%              -4%
   Claude 4.1 Opus            74.0%  68.8 to 78.6    6,815        9,210          -4.9           -19%             +44%
   Grok 4                     76.6%  71.6 to 81.1    5,045        6,586          -2.3           -40%              +7%
   MAI-DxO $2k                77.3%  72.3 to 81.7    1,659        2,146          -1.6           -80%             -65%
   GPT-5                      78.9%  74.0 to 83.2    8,409       10,658          +0.0            +0%             +78%
   MAI-DxO $8k                79.3%  74.4 to 83.5    2,891        3,646          +0.4           -66%             -39%
   MAI-DxO $1k budget         79.6%  74.7 to 83.8    1,303        1,637          +0.7           -85%             -72%
   MAI-DxO $4k                80.9%  76.1 to 84.9    2,241        2,770          +2.0           -73%             -53%
   MAI-DxO $16k               80.9%  76.1 to 84.9    3,606        4,458          +2.0           -57%             -24%
   MAI-DxO No budget          83.2%  78.6 to 87.0    5,431        6,527          +4.3           -35%             +15%
   MAI-DxO Ensemble           84.5%  80.0 to 88.2    8,945       10,586          +5.6            +6%             +89%
== ED: 325 cases, 22 of 22 accuracies are exact k/325 ratios
   openbook: 15 people, 733 graded cases, accuracy per person 35.5 to 62.5, cases per person 9 to 69, average 46.8% at $3,400
   off-the-shelf models above the open-book average: 10 of 13
   MAI-DxO rows at least as accurate as GPT-5 and no costlier: 0 []
   system                       acc  95% interval     cost  per correct  pts vs GPT-5  cost vs GPT-5  cost vs doctors
   MAI-DxO Instant answer     19.4%  15.5 to 24.0        0            0         -57.2          -100%            -100%
   GPT-3.5 turbo              35.1%  30.1 to 40.4    1,226        3,492         -41.5           -71%             -64%
   GPT-4.1 nano               40.0%  34.8 to 45.4    1,636        4,089         -36.6           -61%             -52%
   MAI-DxO Questions Only     42.2%  36.9 to 47.6      300          711         -34.4           -93%             -91%
   Gemma 3                    43.1%  37.8 to 48.5    1,786        4,144         -33.5           -58%             -47%
   Doctors, open book         46.8%  n/a             3,400        7,266         -29.8           -20%              +0%
   Gemini 2.5 Flash           47.4%  42.0 to 52.8    2,637        5,564         -29.2           -38%             -22%
   Llama 4 Maverick           58.2%  52.7 to 63.4    2,426        4,168         -18.4           -43%             -29%
   GPT-4.1 mini               60.0%  54.6 to 65.2    2,223        3,704         -16.6           -48%             -35%
   MAI-DxO $1k budget         63.4%  58.0 to 68.4    1,477        2,330         -13.2           -65%             -57%
   GPT-OSS 120B               63.7%  58.3 to 68.7    3,771        5,920         -12.9           -11%             +11%
   Claude 4.5 Sonnet          65.5%  60.2 to 70.5    3,753        5,729         -11.1           -12%             +10%
   GPT-4.1                    67.4%  62.1 to 72.3    2,255        3,345          -9.2           -47%             -34%
   Gemini 2.5 Pro             67.7%  62.4 to 72.5    3,294        4,865          -8.9           -22%              -3%
   Claude 4.1 Opus            68.9%  63.7 to 73.7    3,200        4,644          -7.7           -25%              -6%
   MAI-DxO $2k                69.5%  64.3 to 74.3    2,037        2,931          -7.1           -52%             -40%
   MAI-DxO $4k                73.2%  68.2 to 77.8    2,634        3,598          -3.4           -38%             -23%
   MAI-DxO $8k                74.5%  69.5 to 78.9    3,128        4,198          -2.1           -26%              -8%
   GPT-5 mini                 74.8%  69.8 to 79.2    4,225        5,648          -1.8            +0%             +24%
   MAI-DxO $16k               76.0%  71.1 to 80.3    3,686        4,849          -0.6           -13%              +8%
   GPT-5                      76.6%  71.7 to 80.9    4,240        5,536          +0.0            +0%             +25%
   MAI-DxO No budget          80.3%  75.6 to 84.3    4,979        6,201          +3.7           +17%             +46%
   MAI-DxO Ensemble           80.9%  76.3 to 84.8    7,352        9,087          +4.3           +73%            +116%

We did not run MAI-DxO or the benchmark ourselves. Everything above is arithmetic on published numbers and a reading of the preprint, the abstract, Microsoft AI’s post and TechCrunch’s report.