Where frontier models still fail clinicians

What 2,140 physician-graded cases say about multi-step clinical reasoning, the three failure patterns that show up in every model we tested, and what it would take to close the gap.

AO

Written by Dr. Amara Okafor · Head of Clinical Research

  • Updated Nov 13, 2026
  • 4 min read
Cover art: layered charcoal strata on a dark ground, crossed by a thin violet line

Most evaluations of medical AI score the conversation. They ask whether the model's answer sounds like something a doctor would say. Ours score what a physician would actually sign: the plan. Over twelve weeks, 212 board-certified physicians across 18 specialties wrote and graded 2,140 cases drawn from real, de-identified discharge files.

The headline number is simple. The best model we tested handled 64% of cases correctly. The median model managed 52%. That sounds like progress, and it is. But the misses were not random, and that is the more useful finding.

How we built the cases

Every case started from a real patient file, rewritten by a physician to remove anything identifying and to keep everything clinically relevant: the labs, the medication history, the social situation, the reason for admission. Each case ends with the same instruction to the model: write the discharge plan.

A second physician, who had not written the case, then wrote a rubric for it before seeing any model output. Rubrics average twelve lines. Every line is binary: the plan either does the thing or it does not.

  1. A physician writes the case from a real, de-identified file.
  2. A second physician writes the rubric, blind to any model output.
  3. Each model writes a plan; two physicians grade it against the rubric independently.
  4. Disagreements go to a third physician, and the rubric line is clarified if needed.

Three failure patterns

When we sorted the failed cases by why they failed, three patterns accounted for most of them. They appeared in every model, and making a model bigger did not make them go away.

1. Dosing that ignores kidney function

The most common serious error was a medication dose that would have been right for a healthy adult and wrong for this patient. The creatinine was in the note. The model wrote a fluent plan and did not adjust for it.

2. Follow-up that comes too late, or not at all

Plans often named the right test and the wrong time. A potassium check "at the next routine visit" for a patient just started on a drug that can raise it dangerously within days. Graders marked these as failures because, in practice, they are.

3. Plans that are safe on paper and impossible in life

The subtlest failures were plans that were clinically correct and practically unworkable: four new medications for a patient with documented trouble affording one, or daily monitoring for someone who lives two hours from a clinic. The social history was in the case. The plan ignored it.

Abstract violet strata illustration
Rubrics turn "would a physician sign this?" into twelve binary checks that two graders can agree on.

Why the transcript hides it

Read as a conversation, most of the failed cases sound convincing. The reasoning is fluent, the tone is right, the vocabulary is impeccable. If you graded the transcript, you would pass many of them.

The model wrote a confident plan. It would have harmed the patient.

That line came from one of our graders, and it describes roughly a third of the failures. Grading the finished plan, with a second physician checking every score, is what surfaced the pattern. Grading the conversation would have hidden it.

What would close the gap

We don't think these failures need a new architecture. They need data that targets them. In a follow-up pilot with one lab, a small set of physician-written examples focused on renal dosing moved that model's score on the dosing slice by 32 points, without hurting anything else.

  • Write cases around the failure, not around the average patient.
  • Grade the artefact a professional would sign, not the reasoning that led to it.
  • Measure agreement between graders, and throw out rubric lines that don't reach it.

What we're publishing

The full method, every rubric template, the error bars and the per-specialty results are in our clinical reasoning whitepaper. If you are training or deploying a clinical model and want to run your own model against these cases, talk to our research team.

Start with one workflow. Prove it, then add the next.

Tell us the system, the data and the outcome you need. An engineer from SixthEra replies within one working day.

Contact us