Medical large vision–language models (Med-LVLMs) have recently achieved remarkable progress in vision–language comprehension and medical image segmentation. However, existing models still struggle to unify these two capabilities, which is essential for clinically meaningful reasoning that connects visual findings with semantic interpretation. We present MEDSIGHT, a unified framework that equips Med-LVLMs with structured, pixel-level understanding for grounded visual comprehension. MEDSIGHT introduces a novel Region Perceiver module that produces region-centric tokens, encoding spatial information directly into the representation space of the language model. We further propose a medical region codebook in the LLM vocabulary, allowing the model to generate discrete region codes as symbolic representations of anatomical and pathological regions. These codes are decoded through the Region Perceiver to reconstruct segmentation masks, achieving end-to-end spatial grounding. A progressive training strategy stably aligns these modules. Trained on only 72K multimodal instruction pairs, MEDSIGHT achieves state-of-the-art performance across diverse imaging modalities on both medical comprehension and segmentation tasks.
Recent Med-LVLMs have demonstrated impressive multimodal understanding of medical data, but most remain limited to high-level visual comprehension and lack pixel-level grounding that aligns reasoning with verifiable visual evidence. Two key limitations of prior unified Med-LVLMs:
CLIP patch tokens encode coarse semantic context and discard crucial spatial details — lesion boundaries, organ contours, tissue textures — that medical reasoning depends on.
[SEG] token outputsRepresenting every segmentation region with one token prevents the model from distinguishing diverse anatomical structures and producing region-specific outputs needed for accurate grounding.
MEDSIGHT augments a frozen CLIP image encoder and LLM with two complementary modules — a Region Perceiver that produces pixel-aware region tokens, and a Modality-aware Region Codebook that is embedded into the LLM vocabulary. A progressive training pipeline aligns these components end-to-end.
L learnable region queries Q interact with upsampled
CLIP features through dual cross-attention — region ↔ image
— producing region embeddings Qr that encode both
semantic context and pixel-level spatial detail. The Region Perceiver is
pre-trained on BiomedParse data with
LR = LBCE + LDice + LCE
using Hungarian matching, following DETR-style segmentation supervision.
Region embeddings Qr are discretized into
K × M codes — one bank per imaging modality. Each
region picks its nearest code, which is mapped back into the visual feature space
via Wm. Codes are appended to the LLM vocabulary,
letting the LLM generate region tokens (e.g.,
[C2_16]) that the text→vision projector decodes into
segmentation masks.
Pre-train the Region Perceiver on segmentation data. Align visual features
with the language space via Pv→t.
Learn the modality-aware codebook and integrate it into the LLM vocabulary.
Align text→vision grounding via Pt→v.
Unified grounded instruction tuning: jointly fine-tune LLM, projectors and codebook with both reasoning and grounding data.
On six medical VQA benchmarks (VQA-RAD, SLAKE, PathVQA, MMMU-Med, OmniMedVQA, and the diagnosis subset of DiagSeg), MEDSIGHT achieves an average score of 62.3, outperforming the best baseline HuatuoGPT-Vision (58.3) while being fine-tuned with only 72K instruction-tuning samples (roughly 9× less than HuatuoGPT-Vision). For each VQA dataset we report close-ended (c) accuracy and overall (a) recall. Rows in gray denote models with partial overlap between their training data and the evaluation sets. Bold marks the best result among unified models and overall.
| Type | Model | #Params | #Data | VQA-RAD ↑ | SLAKE ↑ | PathVQA ↑ | MMMU-Med ↑ | OMVQA ↑ | DiagSeg-Diag ↑ | Avg. ↑ | |||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| close | all | close | all | close | all | ||||||||
| Comp. only |
BLIP-2 | 6.7B | — | 43.4 | 36.8 | 41.6 | 35.3 | 48.5 | 28.8 | 27.3 | 26.9 | 22.3 | 34.5 |
| LLaVA-v1.5 | 7B | 158K | 51.8 | 42.8 | 37.1 | 37.7 | 53.5 | 31.4 | 32.7 | 44.7 | 45.3 | 41.9 | |
| InstructBLIP | 7B | 364K | 61.0 | 44.8 | 66.8 | 43.3 | 56.0 | 32.3 | 25.3 | 29.0 | 35.9 | 43.8 | |
| Yi-VL | 6B | 10K | 52.6 | 42.1 | 52.4 | 38.4 | 54.9 | 30.9 | 38.0 | 50.2 | 33.0 | 43.6 | |
| InternVL2 | 8B | 7.3M | 64.9 | 49.0 | 66.6 | 50.1 | 60.0 | 31.9 | 43.3 | 54.5 | 42.3 | 51.4 | |
| Llama-3.2 | 11B | — | 68.9 | 45.5 | 72.4 | 52.1 | 62.8 | 33.6 | 39.3 | 63.2 | 46.4 | 53.8 | |
| Med-Flamingo | 8.3B | 1.3M | 58.6 | 43.0 | 47.0 | 25.5 | 61.9 | 31.3 | 28.7 | 34.9 | 34.6 | 40.6 | |
| LLaVA-Med | 7B | 60K | 60.2 | 48.1 | 58.4 | 44.8 | 62.3 | 35.7 | 30.0 | 41.3 | 35.3 | 46.2 | |
| HuatuoGPT-Vision | 7B | 647K | 69.7 | 60.0 | 69.0 | 60.1 | 63.6 | 41.3 | 43.3 | 66.4 | 51.1 | 58.3 | |
| Qwen3-VL | 8B | — | 72.8 | 53.4 | 76.9 | 66.1 | 66.9 | 36.9 | 46.7 | 75.3 | 44.5 | 59.9 | |
| InternVL3.5 | 8B | 16.3M | 67.7 | 48.9 | 82.1 | 72.2 | 67.0 | 38.6 | 46.0 | 84.5 | 44.6 | 61.3 | |
| HealthGPT-M3 | 3.8B | 1.5M | 73.7 | 55.9 | 74.6 | 56.4 | 78.7 | 39.7 | 43.3 | 68.5 | 44.1 | 59.4 | |
| HealthGPT-L14 | 14B | 1.5M | 77.7 | 58.3 | 76.4 | 64.5 | 85.9 | 44.4 | 49.2 | 74.4 | 51.2 | 64.7 | |
| Lingshu-7B | 7B | 7.1M | 78.3 | 64.5 | 77.2 | 70.8 | 85.0 | 55.5 | 60.7 | 78.4 | 55.4 | 69.5 | |
| Unified | OMG-LLaVA | 7B | 1.2M | 56.3 | 37.6 | 54.6 | 39.9 | 65.2 | 36.2 | 32.7 | 18.3 | 28.0 | 41.0 |
| MIMO* | 7B | — | 58.8 | — | 57.0 | — | 52.4 | — | — | — | — | — | |
| MedPLIB | 14B/7B | 500K | 58.3 | 34.8 | 48.4 | 37.4 | 35.2 | 18.2 | 28.7 | 61.9 | 13.1 | 37.3 | |
| MEDSIGHT (Ours) | 8B | 72K | 79.9 | 61.4 | 70.9 | 60.2 | 66.3 | 42.6 | 51.3 | 68.9 | 58.9 | 62.3 | |
Table 1. Comparison of MEDSIGHT with general/medical LVLMs and unified LVLMs on medical visual comprehension. Comp.-Only denotes models trained solely for visual comprehension; Unified models are trained on both comprehension and segmentation. Gray rows indicate partial overlap between training and evaluation data. *MIMO is evaluated only on the three public VQA splits in the original paper.
On the proposed DiagSeg benchmark covering eight medical imaging modalities, MEDSIGHT attains a mean diagnosis recall of 58.9 and a mean segmentation Dice of 69.9 — substantially ahead of every baseline on both axes.
| Model | #Params | DiagSeg-Diagnosis (Recall) ↑ | DiagSeg-Segmentation (Dice) ↑ | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CT | MRI | X-ray | Path | US | End | Der | OCT | Avg. | CT | MRI | X-ray | Path | US | End | Der | OCT | Avg. | ||
| LISA | 7B | 11.8 | 7.9 | 3.8 | 21.6 | 5.9 | 2.9 | 33.4 | 27.8 | 14.1 | 15.7 | 9.8 | 34.0 | 45.2 | 39.3 | 28.9 | 62.6 | 19.1 | 31.8 |
| LISA++ | 7B | 9.3 | 19.6 | 5.1 | 16.7 | 6.2 | 19.4 | 35.7 | 16.7 | 15.1 | 1.9 | 12.5 | 21.6 | 26.9 | 8.9 | 6.8 | 31.2 | 0.7 | 13.8 |
| LaSagnA | 7B | 3.6 | 4.1 | 1.0 | 8.2 | 5.1 | 0.0 | 14.8 | 2.8 | 4.4 | 9.3 | 8.7 | 18.2 | 30.0 | 14.1 | 18.0 | 34.9 | 12.6 | 18.2 |
| GLaMM | 7B | 11.4 | 20.6 | 6.9 | 16.6 | 38.9 | 10.6 | 22.3 | 3.1 | 16.9 | 1.6 | 4.5 | 16.6 | 22.8 | 9.9 | 5.6 | 26.3 | 0.8 | 11.0 |
| OMG-LLaVA | 7B | 12.7 | 23.0 | 10.7 | 17.0 | 34.6 | 5.5 | 26.9 | 15.7 | 18.3 | 0.9 | 5.1 | 15.5 | 20.7 | 11.7 | 7.7 | 25.9 | 1.0 | 11.1 |
| MedPLIB | 14B/7B | 4.5 | 18.5 | 15.3 | 5.3 | 71.9 | 22.4 | 29.0 | 0.4 | 13.1 | 3.8 | 47.1 | 29.8 | 19.3 | 49.9 | 16.6 | 80.4 | 7.2 | 31.8 |
| MEDSIGHT (Ours) | 8B | 54.6 | 54.4 | 59.1 | 80.1 | 51.7 | 62.4 | 40.2 | 69.3 | 58.9 | 65.8 | 82.3 | 57.7 | 52.9 | 65.8 | 70.0 | 87.7 | 55.6 | 69.9 |
Table 2. Mean Recall and mean Dice on DiagSeg. MEDSIGHT leads on both diagnosis and segmentation across all eight modalities, exceeding the best baselines by +40.6 (Recall) and +38.1 (Dice) on average. Modality abbreviations: CT, MRI, X-ray, Path (pathology), US (ultrasound), End (endoscopy), Der (dermoscopy), OCT.
MEDSIGHT correctly identifies the abnormality,
emits the matching region code (e.g., [C3_11],
[C10_3], [C1_16]), and the text→vision projector
decodes it into an accurate mask. Multi-region segmentation (e.g., liver and
spleen) is captured by separate, semantically distinct codes.
[C1_16] attends to the liver in abdominal CT.
@inproceedings{chang2026medsight,
title = {MEDSIGHT: Towards Grounded Visual Comprehension in
Medical Large Vision-Language Models},
author = {Chang, Aofei and Huang, Le and Boyd, Alex James and
Bhatia, Parminder and Kass-Hout, Taha and
Ma, Fenglong and Xiao, Cao},
booktitle = {Proceedings of the 43rd International Conference on
Machine Learning (ICML)},
year = {2026}
}We thank the anonymous reviewers for their insightful comments. We also thank Dr. Anjali Rajeev from GE HealthCare as one of the experts for participating in the human validation experiments and Dr. Haidong Yi from St. Jude Children's Research Hospital for valuable suggestions on the manuscript. This work was partially supported by the National Science Foundation under Grant No. 223827 (F. Ma).