Surgical Intelligence Leaderboard

Total and modality scores relative to specialised models

SDSC/UChicagoSDSC/UChicago specialist model	1.000	1.000	1.000	1.000	1.000	1.000
LemonFM (linear probe)	0.863	0.813	0.752	0.691	0.980	1.079
GPT-6 Astra	0.566	0.457	0.809	0.511	0.423	0.631
Qwen3.8 Max 0902	0.423	0.404	0.371	0.082	0.532	0.725
Gemini 3.8 Flash	0.395	0.482	0.589	0.038	0.257	0.609
Gemini 3.1 Pro Preview	0.382	0.470	0.528	0.138	0.251	0.524
Gemini 3.7 Flash	0.373	0.480	0.591	0.028	0.219	0.547
Gemini 3 Flash Preview	0.370	0.434	0.409	0.264	0.291	0.452
Claude Fable 5.1	0.363	0.409	0.458	0.297	0.217	0.435
Claude Opus 5	0.346	0.294	0.437	0.138	0.391	0.467
GPT-5.6 Sol	0.312	0.235	0.477	0.111	0.465	0.275
Claude Fable 5	0.310	0.301	0.296	0.212	0.331	0.409
Claude Opus 4.6	0.288	0.252	0.198	0.010	0.449	0.531
Grok 4.6	0.281	0.329	0.033	0.159	0.321	0.565
Kimi K3	0.278	0.219	0.152	0.172	0.319	0.530
GPT-5.6 Terra	0.233	0.220	0.175	0.050	0.381	0.341
GPT-5.4	0.207	0.042	0.036	0.264	0.305	0.386
GPT-5.6 Luna	0.157	0.101	-0.018	0.062	0.357	0.281
GLM-5.3-Flash	0.149	0.244	-0.196	0.039	0.220	0.439
Claude Sonnet 5	0.130	0.234	-0.074	-0.051	0.237	0.306
Claude Sonnet 4.6	0.130	0.116	-0.048	0.035	0.287	0.262
Gemma 3 27B-it	-0.017	-0.183	-0.169	0.120	0.261	-0.116
Qwen3.8 27B	-0.025	0.069	-0.638	0.093	0.125	0.229

The table shows a weighted average performance of models on surgical modalities. 1 means as good as a specialized computer-vision model and 0 means as good as chance. Modalities include Instrument (CholecT50, PitVis-2023, SurgVU); Anatomy (DSAD, CaDIS, Endoscapes); Skill assessment (SAR-RARP50); Context / VQA (PitVQA); Recommendations (CholecT50 verbs, PitVis-2023 steps).

Surgical Intelligence Index: How well do LLMs perform against specialized models across surgical tasks?

Historical performance

Surgical Intelligence Index: How well do LLMs perform against specialized models across surgical tasks?

Performance by modality

About

Kirill Skobelev

CS PhD student at Northwestern

skobelev.me · LinkedIn

Zhuang-Fang (NaNa) Yi, PhD

Lead ML Engineer at SDSC

Eric Fithian

Predoctoral Fellow at Chicago Booth

ericfithian.com · LinkedIn

Sandeep Angara

Senior Machine Learning Engineer at SDSC

If you found this website useful, please cite as:

@misc{skobelev2026comparativestudysurgicalai,
      title={A Comparative Study in Surgical AI: Potential and Limitations of Data, Compute, and Scaling},
      author={Kirill Skobelev and Eric Fithian and Yegor Baranovski and Jack Cook and Sandeep Angara and Shauna Otto and Zhuang-Fang Yi and John Zhu and Neeraj Mainkar and Margaux Masson-Forsythe and Daniel A. Donoho and X. Y. Han},
      year={2026},
      eprint={2603.27341},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2603.27341},
}

Methodology

Dataset score = (model performance − chance performance) ÷ (specialised reference performance − chance performance). A score of 0 matches chance and 1 matches the specialist. Datasets contribute equally to their modality score. The total averages modality scores, so dataset-rich modalities do not dominate. When results are missing, weights are redistributed equally among available datasets and modalities.

Chance is the exact expected score after randomly shuffling the observed label sets across frames, preserving label frequencies and the number of labels per frame across the sample. Each dataset uses one fixed chance baseline from its frozen seed-42 validation sample of up to 1,000 frames. Negative scores and scores above 1 are retained.

Reference models are fixed per dataset, not selected by the highest score. The SDSC/UChicago row combines these specialists; it is not one model. LemonFM uses task-trained linear probes. Scores compare each model’s reported evaluation and may use different sample sizes, as documented on the modality pages. The single-operation Action pilot has no verified chance baseline yet, so its adjusted score is NA and it is excluded from the index; its original results remain on the Action page.