Key findings
- 01The best model handled 64% of cases correctly; the median model, 52%.
- 02Renal dosing was missed by 4 of 5 models, regardless of size.
- 03Model size predicted fluency, not correctness.
- 04Independent physicians agreed on 81% of rubric lines (κ 0.81).
- 05Targeted expert data moved one model's dosing score by 32 points.
Executive summary
This report covers twelve weeks of work by 212 board-certified physicians across 18 specialties. Together they wrote and graded 2,140 clinical cases, each built from a real, de-identified discharge file, and used them to evaluate five frontier language models on one task: writing a safe, complete discharge plan.
The best model handled 64% of cases correctly. The failures were concentrated in three areas (renal dosing, follow-up timing and practical feasibility) and did not shrink with model size. A small, targeted set of physician-written examples moved one model's performance on the weakest area by 32 points.
Who this report is for
- Research teams training or post-training clinical models, who need to know where current models fail and what kind of data fixes it.
- Health systems and payers deciding whether, and where, to deploy a model in a clinical workflow.
- Regulators and safety teams looking for an evaluation method that is reproducible and auditable.
Method in brief
Each case passed through four stages. The full protocol, including inclusion criteria and the rubric templates, is in the appendix of the PDF.
- Case writing. A physician rewrote a real discharge file to remove identifying details while keeping everything clinically relevant.
- Rubric writing. A second physician, blind to any model output, wrote a rubric of binary lines (average: 12 per case).
- Grading. Two further physicians graded each model's plan independently against the rubric.
- Adjudication. A third physician resolved disagreements; ambiguous rubric lines were rewritten and the batch re-graded.
Results at a glance

Scores are reported with 95% confidence intervals in the full report. Three findings matter most for anyone deploying these systems:
Fluency is not correctness
Graders rated the readability of every plan separately from its correctness. Readability rose steadily with model size. Correctness did not. A plan that reads well is not evidence that it is safe.
Failures cluster, so they can be fixed
Three categories accounted for 71% of all failed rubric lines. That concentration is good news: it means targeted data can move performance far more efficiently than general-purpose data.
Agreement is measurable, and it should be published
Independent graders agreed on 81% of rubric lines before any discussion. We recommend that every clinical evaluation report its inter-grader agreement alongside its headline score.
Limitations
This study evaluated written discharge plans only. It does not measure performance in live conversation, image interpretation or emergency triage, and its cases come from adult inpatient settings in four countries. We discuss the implications of each limitation, and how to extend the method, in section 6 of the report.
Download the full report
The PDF includes the complete method, per-specialty results, every rubric template, confidence intervals for all scores and a set of recommendations for labs and health systems. If you'd like to run your own model against this case set, contact our research team.



