A Knowledge-Grounded Vision–Language Model with Anatomy-Aware Cross-Modal Fusion for Automated Radiology Report Generation
DOI:
https://doi.org/10.29284/2wq6y779Keywords:
Vision–language models, radiology report generation, chest radiography, multimodal learning, medical image captioning, clinical natural language generation, knowledge grounding, hallucination mitigation, MIMIC-CXR.Abstract
Radiology departments worldwide face a widening gap between imaging volume and reporting capacity, and the drafting of free-text reports remains one of the most time-consuming steps in the diagnostic workflow. Automated radiology report generation (RRG) seeks to close this gap by producing a clinically faithful narrative directly from an imaging study. Although recent vision–language models (VLMs) have substantially improved the fluency of generated reports, they continue to hallucinate findings, omit clinically significant abnormalities, and copy templated language that inflates n-gram overlap scores without improving diagnostic accuracy. This paper presents RadFusion-VL, a knowledge-grounded vision–language architecture for chest radiograph report generation. The model couples an anatomy-aware visual encoder that emits region-level tokens for twenty-nine thoracic anatomical zones with a cross-modal fusion module that is aligned to text through a three-stage curriculum: contrastive image–report pre-training, entity-grounded alignment supervised by RadGraph clinical entities and relations, and instruction tuning of a parameter-efficient large language decoder. A clinical-consistency objective and a factuality-aware re-ranking step at inference explicitly penalise unsupported findings. We evaluate on MIMIC-CXR, IU X-Ray and CheXpert Plus using lexical metrics (BLEU, METEOR, ROUGE-L, CIDEr), clinical-efficacy metrics (CheXbert F1, RadGraph-F1), and composite, expert-aligned metrics (RadCliQ, GREEN, RaTEScore), complemented by a blinded reader study with three board-certified radiologists. RadFusion-VL improves RadGraph-F1 and CheXbert macro-F1 over strong transformer and LLM-based baselines while producing fewer fabricated findings in the reader study. Ablations isolate the contribution of anatomical grounding, entity supervision and factuality re-ranking. We conclude with a discussion of the residual failure modes, evaluation-metric limitations, and the regulatory and ethical prerequisites for clinical deployment.
References
[1] Z. Chen, Y. Song, T.-H. Chang, and X. Wan, “Generating radiology reports via memory-driven transformer,” in Proc. Conf. Empirical Methods in Natural Language Processing (EMNLP), 2020, pp. 1439–1449.
[2] Z. Chen, Y. Shen, Y. Song, and X. Wan, “Cross-modal memory networks for radiology report generation,” in Proc. 59th Annu. Meeting Assoc. Comput. Linguistics (ACL), 2021, pp. 5904–5914.
[3] A. E. W. Johnson et al., “MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports,” Scientific Data, vol. 6, no. 1, p. 317, 2019.
[4] D. Demner-Fushman et al., “Preparing a collection of radiology examinations for distribution and retrieval,” J. Amer. Med. Informat. Assoc., vol. 23, no. 2, pp. 304–310, 2016.
[5] P. Chambon et al., “CheXpert Plus: Augmenting a large chest X-ray dataset with text radiology reports, patient demographics and additional image formats,” arXiv:2405.19538, 2024.
[6] J. Irvin et al., “CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison,” in Proc. AAAI Conf. Artificial Intelligence, 2019, pp. 590–597.
[7] A. Smit, S. Jain, P. Rajpurkar, A. Pareek, A. Y. Ng, and M. P. Lungren, “CheXbert: Combining automatic labelers and expert annotations for accurate radiology report labeling using BERT,” in Proc. EMNLP, 2020, pp. 1500–1519.
[8] S. Jain et al., “RadGraph: Extracting clinical entities and relations from radiology reports,” in Proc. NeurIPS Datasets and Benchmarks Track, 2021.
[9] F. Yu et al., “Evaluating progress in automatic chest X-ray radiology report generation,” Patterns, vol. 4, no. 9, p. 100802, 2023.
[10] S. Ostmeier et al., “GREEN: Generative radiology report evaluation and error notation,” in Findings of EMNLP, 2024. arXiv:2405.03595.
[11] W. Zhao, C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie, “RaTEScore: A metric for radiology report generation,” in Proc. EMNLP, 2024.
[12] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “BLEU: A method for automatic evaluation of machine translation,” in Proc. ACL, 2002, pp. 311–318.
[13] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in Text Summarization Branches Out, 2004, pp. 74–81.
[14] S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in Proc. ACL Workshop on Intrinsic and Extrinsic Evaluation Measures, 2005, pp. 65–72.
[15] R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: Consensus-based image description evaluation,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2015, pp. 4566–4575.
[16] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “BERTScore: Evaluating text generation with BERT,” in Proc. Int. Conf. Learning Representations (ICLR), 2020.
[17] A. Dosovitskiy et al., “An image is worth 16×16 words: Transformers for image recognition at scale,” in Proc. ICLR, 2021.
[18] Z. Liu et al., “Swin Transformer: Hierarchical vision transformer using shifted windows,” in Proc. IEEE Int. Conf. Computer Vision (ICCV), 2021, pp. 10012–10022.
[19] A. Radford et al., “Learning transferable visual models from natural language supervision,” in Proc. Int. Conf. Machine Learning (ICML), 2021, pp. 8748–8763.
[20] Y. Zhang, H. Jiang, Y. Miura, C. D. Manning, and C. P. Langlotz, “Contrastive learning of medical visual representations from paired images and text,” in Proc. Machine Learning for Healthcare (MLHC), 2022, pp. 2–25.
[21] S.-C. Huang, L. Shen, M. P. Lungren, and S. Yeung, “GLoRIA: A multimodal global-local representation learning framework for label-efficient medical image recognition,” in Proc. ICCV, 2021, pp. 3942–3951.
[22] B. Boecking et al., “Making the most of text semantics to improve biomedical vision–language processing,” in Proc. European Conf. Computer Vision (ECCV), 2022, pp. 1–21.
[23] S. Bannur et al., “Learning to exploit temporal structure for biomedical vision–language processing,” in Proc. CVPR, 2023, pp. 15016–15027.
[24] J. Li, D. Li, S. Savarese, and S. Hoi, “BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in Proc. ICML, 2023, pp. 19730–19742.
[25] J.-B. Alayrac et al., “Flamingo: A visual language model for few-shot learning,” in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2022.
[26] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” in Proc. NeurIPS, 2023.
[27] C. Li et al., “LLaVA-Med: Training a large language-and-vision assistant for biomedicine in one day,” in Proc. NeurIPS Datasets and Benchmarks Track, 2023.
[28] S. L. Hyland et al., “MAIRA-1: A specialised large multimodal model for radiology report generation,” arXiv:2311.13668, 2023.
[29] S. Bannur et al., “MAIRA-2: Grounded radiology report generation,” arXiv:2406.04449, 2024.
[30] Z. Chen et al., “CheXagent: Towards a foundation model for chest X-ray interpretation,” arXiv:2401.12208, 2024.
[31] T. Tu et al., “Towards generalist biomedical AI,” NEJM AI, vol. 1, no. 3, 2024.
[32] R. Tanno et al., “Collaboration between clinicians and vision–language models in radiology report generation,” Nature Medicine, vol. 31, pp. 599–608, 2025.
[33] J. P. Zambrano Chaves et al., “Towards a clinically accessible radiology foundation model: Open-access and lightweight, with automated evaluation,” arXiv:2403.08002, 2024.
[34] N. Deperrois et al., “RadVLM: A multitask conversational vision-language model for radiology,” arXiv:2502.03333, 2025.
[35] Z. Wang, L. Liu, L. Wang, and L. Zhou, “R2GenGPT: Radiology report generation with frozen LLMs,” Meta-Radiology, vol. 1, no. 3, p. 100033, 2023.
[36] Z. Wang, L. Liu, L. Wang, and L. Zhou, “METransformer: Radiology report generation by transformer with multiple learnable expert tokens,” in Proc. CVPR, 2023, pp. 11558–11567.
[37] J. Wang, A. Bhalerao, and Y. He, “Cross-modal prototype driven network for radiology report generation,” in Proc. ECCV, 2022, pp. 563–579.
[38] T. Tanida, P. Müller, G. Kaissis, and D. Rueckert, “Interactive and explainable region-guided radiology report generation,” in Proc. CVPR, 2023, pp. 7433–7442.
[39] X. Wang et al., “CXPMRG-Bench: Pre-training and benchmarking for X-ray medical report generation on CheXpert Plus dataset,” arXiv:2410.00379, 2024.
[40] E. J. Hu et al., “LoRA: Low-rank adaptation of large language models,” in Proc. ICLR, 2022.
[41] J. Wu et al., “Chest ImaGenome dataset for clinical reasoning,” in Proc. NeurIPS Datasets and Benchmarks Track, 2021.
[42] H. Q. Nguyen et al., “VinDr-CXR: An open dataset of chest X-rays with radiologist's annotations,” Scientific Data, vol. 9, no. 1, p. 429, 2022.
[43] P. Rajpurkar and M. P. Lungren, “The current and future state of AI interpretation of medical images,” New England J. Medicine, vol. 388, no. 21, pp. 1981–1990, 2023.
[44] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in Proc. ICLR, 2019.
[45] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, “Self-critical sequence training for image captioning,” in Proc. CVPR, 2017, pp. 7008–7024.
[46] Y. Miura, Y. Zhang, E. B. Tsai, C. P. Langlotz, and D. Jurafsky, “Improving factual completeness and consistency of image-to-text radiology report generation,” in Proc. NAACL-HLT, 2021, pp. 5288–5304.
[47] J.-B. Delbrouck et al., “Improving the factual correctness of radiology report generation with semantic rewards,” in Findings of EMNLP, 2022, pp. 4348–4360.
[48] L. Ouyang et al., “Training language models to follow instructions with human feedback,” in Proc. NeurIPS, 2022.
[49] A. Vaswani et al., “Attention is all you need,” in Proc. NeurIPS, 2017, pp. 5998–6008.
[50] C. Wu, X. Zhang, Y. Zhang, Y. Wang, and W. Xie, “Towards generalist foundation model for radiology,” arXiv:2308.02463, 2023.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
