Synonym Based Hierarchical Probability Transformer For Predicting Disease Intensity And Relationships Between Icd Codes
DOI:
https://doi.org/10.29284/q5jpc185Keywords:
Large Language Models, International Classification of Diseases, Synonym-based ICD Code Attention, Integrated Gradients, Hierarchical Conditional Probability, Corrective Attention Transformer.Abstract
Predicting disease intensity levels and identifying relationships between the International Classification of Disease codes utilizing Large Language Models (LLMs) and Transformer Learning investigate and classify immense amounts of clinical data to acquire awareness into disease advancement and possible associations between conditions. Precisely, LLMs explore route of care for a specific patient population on a specific condition represented as series of medical codes paving the way for diagnosis, predict intensity and ascertain possible associations between diseases. Early identification as well as deterrence is pivotal for minimizing hazard of disease evolution. This study intended to introduce a novel method depend on LLM and transformer to predict disease intensity and explore the relationship between ICD codes for predicting disease progression. In this work a method called, Synonym-based Large Language Attention and Hierarchical Probability Transformer (SLLA-HPT) for predicting disease intensity and identifying relationships between ICD codes is proposed. The SLLA-HPT method is split into four sections. First, data collection and pre-processing is performed with the raw MIMIC-III dataset. Next, for predicting disease intensity levels from ICD codes Synonym-based ICD Code Attention Pre-training model is applied. Following which fine-tuning is performed using the Integrated Gradients (IG) function. Next, Hierarchical Conditional Probability and Corrective Attention Transformer are applied to the fine-tuned results for obtaining prediction results. Finally, Correlation-based Relationship Identification is applied to create ranked record of diseases through their provisional probability as well as correlative risk grade. Proposed method is estimated on MIMIC-III database as well as experimental outcomes show effectiveness of our method upon comparison to conventional techniques.
References
[1] Adnan Karamat, Muhammad Imran, Muhammad Usman Yaseen, Rasool Bukhsh, Sheraz Aslam, Nouman Ashraf, “A Hybrid Transformer Architecture for Multiclass Mental Illness Prediction Using Social Media Text”, IEEE Access, Vol. 13, Jan 2025 [Hybrid Transformer Architecture]
[2] Leibo Liu, Oscar Perez-Concha, Anthony Nguyen, Vicki Bennett, Louisa Jorm, “ Automated ICD coding using extreme multi-label long text transformer-based models”, Artificial Intelligence In Medicine, Elsevier, Vol. 144, Oct 2023 [eXtreme Resolution-Long Attention Transformer (XR-LATP)]
[3] Yifan Wu, Min Zeng,Ying Yu,Yaohang Li and Min Li, “A Pseudo Label-wise Attention Network for Automatic ICD Coding”, IEEE Journal of Biomedical and Health Informatics, Volume 26, Issue 10, October 2022, Pages 5201 – 5212 [Label Description-based Pseudo Label-wise Attentional Model (LD-PLAM)]
[4] Lan Wang, Yonghua Yin, Ben Glampson, Robert Peach, Mauricio Barahona, Brendan C. Delaney, and Erik K. Mayer, “Transformer-based deep learning model for the diagnosis of suspected lung cancer in primary care based on electronic health record data”, EBioMedicine, National Library of Medicine, Vol. 110, Dec 2024
[5] Gaspard Hareriman, Jong Wook Kim, Beakcheol Jang, “A deep attention model to forecast the Length Of Stay and the in-hospital mortality right on admission from ICD codes and demographic data”, Journal of Biomedical Informatics, Elsevier, Vol. 118, Jun 2021
[6] Alberto Blancoa, Sonja Remmer, Alicia P´ erez, Hercules Dalianis, Arantza Casillas, “Implementation of specialized attention mechanisms: ICD-10 classification of Gastrointestinal discharge summaries in English, Spanish and Swedish”, Journal of Biomedical Informatics, Elsevier, Vol. 130, Jun 2022
[7] Guangming Huang, Yingya Li, Shoaib Jameel, Yunfei Long, Giorgos Papanastasiou, “From explainable to interpretable deep learning for natural language processing in healthcare: How far from reality?”, Computational and Structural Biotechnology Journal, Elsevier, Vol. 24, Dec 2024
[8] Mustafa Arda Ayden, Mehmet Eren Yuksel, Seniha Esen Yuksel Erdem, “A two-stream deep model for automated ICD-9 code prediction in an intensive care unit”, Heliyon, Elsevier, Feb 2024
[9] Xiangbin Meng, Xiangyu Yan, Kuo Zhang, Da Liu, Xiaojuan Cui, Yaodong Yang, Muhan Zhang, Chunxia Cao, Jingjia Wang, Xuliang Wang, Jun Gao, Yuan-Geng-Shuo Wang, Jia-ming Ji, Zifeng Qiu, “The application of large language models in medicine: A scoping review”, National Library of Medicine, Elsevier, May 2024
[10] Zihao Yang, Chenkang Zhang, Muru Wu, Xujin Chris Liu, Lavender Yao Jiang, Kyunghyun Cho, Eric Karl Oermann, “Intriguing Effect of the Correlation Prior on ICD-9 Code Assignment”, ACL Anthology, Association for Computational Linguistics, Jul 2023
[11] Hanyin Wang, Chufan Gao, Christopher Dantona, Bryan Hull and Jimeng Sun, “DRG-LLaMA : tuning LLaMA model to predict diagnosis related group for hospitalized patients”, Digital Medicine, Jan 2024
[12] Edgar Steiger, DPhil; Lars Eric Kroll, DPhil, “Patient Embeddings from Diagnosis Codes for Health Care Prediction Tasks: Pat2Vec Machine Learning Framework”, Artificial Intelligence, JMIR, Oct 2023
[13] Siva Sai, Aanchal Gaur, Revant Sai, Vinay Chamola Siva Sai Mohsen Guizani, Joel J. P. C. Rodrigues, “Generative AI for Transformative Healthcare: A Comprehensive Study of Emerging Models, Applications, Case Studies, and Limitations”, IEEE Access, Vol. 12, Mar 2024
[14] Gaspard Harereimana Jong Wookkim, Gun Il Kim, Beakcheol Jang, “HSGA: A Hybrid LSTM-CNN Self-Guided Attention to Predict the Future Diagnosis From Discharge Narratives”, IEEE Access, Vol. 11, Oct 2023
[15] Yoon-Sik Cho, Eunsun Kim Younghoon Kwon, “Identifying Disease of Interest With Deep Learning Using Diagnosis Code”, Journal of Korean Medical Science, Mar 2023
[16] Zhichao Yang Hong Yu, Avijit Mitra, Weisong Liu, Dan Berlowitz, “TransformEHR: transformer-based encoder decoder generative model to enhance prediction of disease outcomes using electronic health records”, Nature Communications, Nov 2023
[17] Cathy Shyr, Yan Hu,Lisa Bastarache, Alex Cheng, Rizwan Hamid, Paul Harris, “Identifying and Extracting Rare Diseases and Their Phenotypes with Large Language Models”, Journal of Healthcare Informatics Research, Nov 2024
[18] Fatma M. Talaat, “Revolutionizing cardiovascular health: integrating deep learning techniques for predictive analysis of personal key indicators in heart disease”, Neural Computing and Applications, Springer, Jan 2025
[19] Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, “Large language models encode clinical knowledge”, Nature, Jul 2023
[20] Yunsu Byeon, Yae Won Park, Soohyun Lee, Doohyun Park, Hyung Seob Shin, Kyunghwa Han, Jong Hee Chang, Se Hoon Kim, Seung-Koo Lee, Sung Soo Ahn, DosikHwang, “Interpretable multimodal transformer for prediction of molecular subtypes and grades in adult-type diffuse gliomas”, Digital Medicine, Mar 2025
[21] Quentin Marcou, Laure Berti-Equille, Noël Novelli, “Creating a computer assisted ICD coding system: Performance metric choice and use of the ICD hierarchy”, Journal of Biomedical Informatics, Elsevier, Vol. 151, Apr 2024
[22] Atta Ur Rahman, Yousef Alsenani, Adeel Zafar, Kalim Ullah, Khaled Rabie, Thokozani Shongwe, “Enhancing heart disease prediction using a self attention based transformer model”, Scientific Reports, Nov 2024
[23] Christoph S. Metzner, Shang Gao, Drahomira Herrmannov, Heidi A. Hanson, “Attention Mechanisms in Clinical Text Classification: A Comparative Evaluation”, IEEE Journal of Biomedical and Health Informatics, Vol. 28, No. 4, Apr 2024
[24] Zhangdaihong Liu, Ying Hu, Xuan Wu, Gert Mertes, Yang Yang, David A. Clifton, “Patient Clustering for Vital Organ Failure Using ICD Code With Graph Attention”, IEEE Transactions on Biomedical Engineering, Vol. 70, No. 8, Aug 2023
[25] Subhash Nerella, Sabyasachi Bandyopadhyay, Jiaqing Zhang, Miguel Contreras, Scott Siegela, Aysegul Bumin, Brandon Silva, Jessica Senae, Benjamin Shickelf, Azra Bihoracf, Kia Khezelia, Parisa Rashidi, “Transformers and large language models in healthcare: A review”, Artificial Intelligence in Medicine, Aug 2024
[26] Khalid Nawab, Madalyn Fernbach, Sayuj Atreya, Samina Asfandiyar, Gulalai Khan, Riya Arora, Iqbal Hussain, Shadi Hijjawi, Richard Schreiber, “Fine-tuning for accuracy: evaluation of Generative Pretrained Transformer (GPT) for automatic assignment of International
Classification of Disease (ICD) codes to clinical documentation”, Journal of Medical Artificial Intelligence, Oct 2024
[27] Jakir Hossain Bhuiyan Masud, Chen-Cheng Kuo, Chih-Yang Yeh, Hsuan-Chia Yang, Ming-Chin Lin, “Applying Deep Learning Model to Predict Diagnosis Code of Medical Records”, Diagnostics, National Library of Medicine, Jul 2023
[28] Isabel Coutinho, Bruno Martins, “Transformer-based models for ICD-10 coding of death certificates with Portuguese text”, Journal of Biomedical Informatics, ACM, Vol. 136, Dec 2022
[29] Sander Puts, Catharina M L Zegers, Andre Dekker, Iñigo Bermejo, “Developing an ICD-10 Coding Assistant: Pilot Study Using RoBERTa and GPT-4 for Term Extraction and Description-Based Code Selection”, Advancing Digital Health & Open Science, Vol. 9, Jan 2025
[30] Johnson A., Pollard T., Mark R., “MIMIC-III Clinical Database (version 1.4) PhysioNet”, Oct 2016
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
