Attention-Based Multi-Model Fusion With Hierarchical Multi-Task Learning For Enhanced Diabetic Retinopathy Detection
DOI:
https://doi.org/10.29284/1nhvrg29Keywords:
diabetic retinopathy, deep learning, attention mechanism, multi-model fusion, multi-task learning, convolutional neural networks, medical image analysis, ResNet50, XceptionAbstract
Diabetic retinopathy is a leading cause of blindness worldwide, necessitating accurate and timely detection through automated systems. This research introduces two complementary novelties to improve DR classification from fundus images using the APTOS retinal dataset. First, we propose an attention-based multi-model fusion approach that adaptively combines features from ResNet50 and Xception architectures using learned attention weights that vary per image, enabling the model to dynamically determine when to trust which network based on image characteristics. Second, we implement a hierarchical multi-task learning framework with three clinically-motivated levels: primary DR grading, binary referable classification, and multi-label lesion detection. Our experimental results demonstrate that the synergistic combination of these novelties achieves 5-7% accuracy improvement over baseline ResNet50, reaching approximately 80% accuracy with a Quadratic Weighted Kappa score of 0.82-0.85. The proposed system not only achieves higher accuracy but also provides interpretable, actionable insights aligned with clinical practice through attention weights and auxiliary task predictions.
References
[1] D. S. W. Ting, G. C. M. Cheung, and T. Y. Wong, "Diabetic retinopathy: global prevalence, major risk factors, screening practices and public health challenges: a review," Clinical & Experimental Ophthalmology, vol. 44, no. 4, pp. 260-277, May 2016.
[2] International Diabetes Federation, IDF Diabetes Atlas, 10th ed. Brussels, Belgium: IDF, 2021.
[3] N. Cheung, P. Mitchell, and T. Y. Wong, "Diabetic retinopathy," The Lancet, vol. 376, no. 9735, pp. 124-136, Jul. 2010.
[4] M. D. Abràmoff, Y. Lou, A. Erginay, W. Clarida, R. Amelon, J. C. Folk, and M. Niemeijer, "Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning," Investigative Ophthalmology & Visual Science, vol. 57, no. 13, pp. 5200-5206, Oct. 2016.
[5] J. W. Yau, S. L. Rogers, R. Kawasaki, E. L. Lamoureux, J. W. Kowalski, T. Bek, S. J. Chen, J. M. Dekker, A. Fletcher, J. Grauslund, and S. Haffner, "Global prevalence and major risk factors of diabetic retinopathy," Diabetes Care, vol. 35, no. 3, pp. 556-564, Mar. 2012.
[6] Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, no. 7553, pp. 436-444, May 2015.
[7] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "ImageNet classification with deep convolutional neural networks," in Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA, 2012, pp. 1097-1105.
[8] G. Litjens, T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. W. M. van der Laak, B. van Ginneken, and C. I. Sánchez, "A survey on deep learning in medical image analysis," Medical Image Analysis, vol. 42, pp. 60-88, Dec. 2017.
[9] Z. C. Lipton, "The mythos of model interpretability," Communications of the ACM, vol. 61, no. 10, pp. 36-43, Oct. 2018.
[10] C. Molnar, Interpretable Machine Learning: A Guide for Making Black Box Models Explainable, 2nd ed. Munich, Germany: Self-published, 2022.
[11] R. S. Taylor and D. Batty, "Handbook of Clinical Neurology," in Diabetes and the Nervous System, vol. 126, Amsterdam, Netherlands: Elsevier, 2014, ch. 8, pp. 145-165.
[12] R. N. Frank, "Diabetic retinopathy," New England Journal of Medicine, vol. 350, no. 1, pp. 48-58, Jan. 2004.
[13] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 770-778.
[14] S. Zagoruyko and N. Komodakis, "Wide residual networks," in British Machine Vision Conference, York, UK, 2016, pp. 87.1-87.12.
[15] F. Chollet, "Xception: deep learning with depthwise separable convolutions," in IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 1251-1258.
[16] C. P. Wilkinson, F. L. Ferris, R. E. Klein, P. P. Lee, C. D. Agardh, M. Davis, D. Dills, A. Kampik, R. Pararajasegaram, J. T. Verdaguer, and Global Diabetic Retinopathy Project Group, "Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales," Ophthalmology, vol. 110, no. 9, pp. 1677-1682, Sep. 2003.
[17] A. Holzinger, C. Biemann, C. S. Pattichis, and D. B. Kell, "What do we need to build explainable AI systems for the medical domain?," arXiv preprint arXiv:1712.09923, Dec. 2017.
[18] APTOS 2019 Blindness Detection, Kaggle Competition Dataset. [Online]. Available: https://www.kaggle.com/c/aptos2019-blindness-detection
[19] H. He and E. A. Garcia, "Learning from imbalanced data," IEEE Transactions on Knowledge and Data Engineering, vol. 21, no. 9, pp. 1263-1284, Sep. 2009.
[20] V. Gulshan, L. Peng, M. Coram, M. C. Stumpe, D. Wu, A. Narayanaswamy, S. Venugopalan, K. Widner, T. Madams, J. Cuadros, R. Kim, R. Raman, P. C. Nelson, J. L. Mega, and D. R. Webster, "Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs," Journal of the American Medical Association, vol. 316, no. 22, pp. 2402-2410, Dec. 2016.
[21] H. Pratt, F. Coenen, D. M. Broadbent, S. P. Harding, and Y. Zheng, "Convolutional neural networks for diabetic retinopathy," Procedia Computer Science, vol. 90, pp. 200-205, Jul. 2016.
[22] K. He, X. Zhang, S. Ren, and J. Sun, "Identity mappings in deep residual networks," in European Conference on Computer Vision, Amsterdam, Netherlands, 2016, pp. 630-645.
[23] X. Li, T. Pang, B. Xiong, W. Liu, P. Liang, and T. Wang, "Convolutional neural networks based transfer learning for diabetic retinopathy fundus image classification," in International Congress on Image and Signal Processing, Chengdu, China, 2019, pp. 1-11.
[24] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L. C. Chen, "MobileNetV2: inverted residuals and linear bottlenecks," in IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 4510-4520.
[25] S. Badrinarayanan, A. Kendall, and R. Cipolla, "SegNet: a deep convolutional encoder-decoder architecture for image segmentation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 12, pp. 2481-2495, Dec. 2017.
[26] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, "Rethinking the inception architecture for computer vision," in IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 2818-2826.
[27] L. Breiman, "Bagging predictors," Machine Learning, vol. 24, no. 2, pp. 123-140, Aug. 1996.
[28] X. Wang, Y. Lu, Y. Wang, and W. B. Chen, "Diabetic retinopathy stage classification using convolutional neural networks," in IEEE International Conference on Information Reuse and Integration, San Diego, CA, USA, 2017, pp. 465-471.
[29] [29] M. D. Abràmoff, P. T. Lavin, M. Birch, N. Shah, and J. C. Folk, "Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices," npj Digital Medicine, vol. 1, no. 1, pp. 1-8, Aug. 2018.
[30] [30] D. Opitz and R. Maclin, "Popular ensemble methods: an empirical study," Journal of Artificial Intelligence Research, vol. 11, pp. 169-198, Aug. 1999.
[31] [31] S. Qummar, F. G. Khan, S. Shah, A. Khan, S. Shamshirband, Z. U. Rehman, I. A. Khan, and W. Jadoon, "A deep learning ensemble approach for diabetic retinopathy detection," IEEE Access, vol. 7, pp. 150530-150539, Oct. 2019.
[32] [32] S. Jetley, N. A. Lord, N. Lee, and P. H. S. Torr, "Learn to pay attention," in International Conference on Learning Representations, Vancouver, Canada, 2018.
[33] [33] J. Schlemper, O. Oktay, M. Schaap, M. Heinrich, B. Kainz, B. Glocker, and D. Rueckert, "Attention gated networks: learning to leverage salient regions in medical images," Medical Image Analysis, vol. 53, pp. 197-207, Apr. 2019.
[34] [34] J. Hu, L. Shen, and G. Sun, "Squeeze-and-excitation networks," in IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 7132-7141.
[35] [35] S. Ruder, "An overview of multi-task learning in deep neural networks," arXiv preprint arXiv:1706.05098, Jun. 2017.
[36] [36] X. Liu, F. Zhou, J. Liu, and L. Jiang, "Meta multi-task learning for sequence modeling," in AAAI Conference on Artificial Intelligence, Honolulu, HI, USA, 2020, pp. 4986-4993.
[37] [37] Z. Yan, X. Han, C. Wang, Y. Qiu, Z. Xiong, and S. Cui, "Learning mutually local-global U-nets for high-resolution retinal lesion segmentation in fundus images," in IEEE International Symposium on Biomedical Imaging, Venice, Italy, 2019, pp. 597-600.
[38] [38] R. Caruana, "Multitask learning," Machine Learning, vol. 28, no. 1, pp. 41-75, Jul. 1997.
[39] [39] A. Kendall, Y. Gal, and R. Cipolla, "Multi-task learning using uncertainty to weigh losses for scene geometry and semantics," in IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 2018, pp. 7482-7491.
[40] [40] N. Srivastava and R. R. Salakhutdinov, "Multimodal learning with deep Boltzmann machines," in Advances in Neural Information Processing Systems, Lake Tahoe, NV, USA, 2012, pp. 2222-2230.
[41] [41] Y. Zhang and Q. Yang, "A survey on multi-task learning," IEEE Transactions on Knowledge and Data Engineering, vol. 34, no. 12, pp. 5586-5609, Dec. 2022.
[42] [42] J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 2015, pp. 3431-3440.
[43] [43] O. Ronneberger, P. Fischer, and T. Brox, "U-Net: convolutional networks for biomedical image segmentation," in International Conference on Medical Image Computing and Computer-Assisted Intervention, Munich, Germany, 2015, pp. 234-241.
[44] [44] J. Deng, W. Dong, R. Socher, L. J. Li, K. Li, and L. Fei-Fei, "ImageNet: a large-scale hierarchical image database," in IEEE Conference on Computer Vision and Pattern Recognition, Miami, FL, USA, 2009, pp. 248-255.
[45] [45] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, "MobileNets: efficient convolutional neural networks for mobile vision applications," arXiv preprint arXiv:1704.04861, Apr. 2017.
[46] [46] S. Woo, J. Park, J. Y. Lee, and I. S. Kweon, "CBAM: convolutional block attention module," in European Conference on Computer Vision, Munich, Germany, 2018, pp. 3-19.
[47] [47] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu, "Dual attention network for scene segmentation," in IEEE Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 2019, pp. 3146-3154.
[48] L. Chen, H. Zhang, J. Xiao, L. Nie, J. Shao, W. Liu, and T. S. Chua, "SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning," in IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 5659-5667.
[49] F. Wang, M. Jiang, C. Qian, S. Yang, C. Li, H. Zhang, X. Wang, and X. Tang, "Residual attention network for image classification," in IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 3156-3164.
[50] K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," in International Conference on Learning Representations, San Diego, CA, USA, 2015.
[51] M. D. Zeiler and R. Fergus, "Visualizing and understanding convolutional networks," in European Conference on Computer Vision, Zurich, Switzerland, 2014, pp. 818-833.
[52] T. Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, "Focal loss for dense object detection," in IEEE International Conference on Computer Vision, Venice, Italy, 2017, pp. 2980-2988.
[53] S. J. Pan and Q. Yang, "A survey on transfer learning," IEEE Transactions on Knowledge and Data Engineering, vol. 22, no. 10, pp. 1345-1359, Oct. 2010.
[54] M. Tan and Q. V. Le, "EfficientNet: rethinking model scaling for convolutional neural networks," in International Conference on Machine Learning, Long Beach, CA, USA, 2019, pp. 6105-6114.
[55] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, "Dropout: a simple way to prevent neural networks from overfitting," Journal of Machine Learning Research, vol. 15, no. 1, pp. 1929-1958, Jun. 2014.
[56] C. Shorten and T. M. Khoshgoftaar, "A survey on image data augmentation for deep learning," Journal of Big Data, vol. 6, no. 1, pp. 1-48, Dec. 2019.
[57] L. Perez and J. Wang, "The effectiveness of data augmentation in image classification using deep learning," arXiv preprint arXiv:1712.04621, Dec. 2017.
[58] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson, "How transferable are features in deep neural networks?," in Advances in Neural Information Processing Systems, Montreal, Canada, 2014, pp. 3320-3328.
[59] L. N. Smith, "Cyclical learning rates for training neural networks," in IEEE Winter Conference on Applications of Computer Vision, Santa Rosa, CA, USA, 2017, pp. 464-472.
[60] D. P. Kingma and J. Ba, "Adam: a method for stochastic optimization," in International Conference on Learning Representations, San Diego, CA, USA, 2015.
[61] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, "SMOTE: synthetic minority over-sampling technique," Journal of Artificial Intelligence Research, vol. 16, pp. 321-357, Jun. 2002.
[62] J. Cohen, "Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit," Psychological Bulletin, vol. 70, no. 4, pp. 213-220, Oct. 1968.
[63] T. Fawcett, "An introduction to ROC analysis," Pattern Recognition Letters, vol. 27, no. 8, pp. 861-874, Jun. 2006.
[64] P. Porwal, S. Pachade, R. Kamble, M. Kokare, G. Deshmukh, V. Sahasrabuddhe, and F. Meriaudeau, "Indian diabetic retinopathy image dataset: a database representative of Indian population," in IEEE Dataport, 2018.
[65] S. Ioffe and C. Szegedy, "Batch normalization: accelerating deep network training by reducing internal covariate shift," in International Conference on Machine Learning, Lille, France, 2015, pp. 448-456.
[66] M. Buda, A. Maki, and M. A. Mazurowski, "A systematic study of the class imbalance problem in convolutional neural networks," Neural Networks, vol. 106, pp. 249-259, Oct. 2018.
[67] C. Huang, Y. Li, C. C. Loy, and X. Tang, "Learning deep representation for imbalanced classification," in IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 5375-5384.
[68] S. Thrun and L. Pratt, Learning to Learn, New York, NY, USA: Springer Science & Business Media, 1998.
[69] X. Zhu, Z. Cheng, S. Wang, X. Chen, and G. Lu, "Coronary angiography image segmentation based on PSPNet," Computer Methods and Programs in Biomedicine, vol. 200, Art. no. 105897, Mar. 2021.
[70] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems, Long Beach, CA, USA, 2017, pp. 5998-6008.
[71] K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, and Y. Bengio, "Show, attend and tell: neural image caption generation with visual attention," in International Conference on Machine Learning, Lille, France, 2015, pp. 2048-2057.
[72] Y. Yang and T. M. Hospedales, "Deep multi-task representation learning: a tensor factorisation approach," in International Conference on Learning Representations, San Juan, Puerto Rico, 2017.
[73] C. Drummond and R. C. Holte, "C4.5, class imbalance, and cost sensitivity: why under-sampling beats over-sampling," in Workshop on Learning from Imbalanced Datasets II, Washington, DC, USA, 2003, pp. 1-8.
[74] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, "Grad-CAM: visual explanations from deep networks via gradient-based localization," in IEEE International Conference on Computer Vision, Venice, Italy, 2017, pp. 618-626.
[75] D. S. W. Ting, C. Y. L. Cheung, G. Lim, G. S. W. Tan, N. D. Quang, A. Gan, H. Hamzah, R. Garcia-Franco, I. Y. San Yeo, S. Y. Lee, E. Y. M. Wong, C. Sabanayagam, M. Baskaran, F. Ibrahim, N. C. Tan, E. A. Finkelstein, E. L. Lamoureux, I. Y. Wong, N. M. Bressler, S. Sivaprasad, R. Varma, J. B. Jonas, M. G. He, C. Y. Cheng, G. C. M. Cheung, T. Aung, W. Hsu, M. L. Lee, and T. Y. Wong, "Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes," Journal of the American Medical Association, vol. 318, no. 22, pp. 2211-2223, Dec. 2017.
[76] B. Graham, "Kaggle diabetic retinopathy detection competition report," University of Warwick, Tech. Rep., 2015.
[77] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," in IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, USA, 2017, pp. 4700-4708.
[78] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: unified, real-time object detection," in IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 779-788.
[79] R. Gargeya and T. Leng, "Automated identification of diabetic retinopathy using deep learning," Ophthalmology, vol. 124, no. 7, pp. 962-969, Jul. 2017.
[80] M. K. K. Niazi, A. V. Parwani, and M. N. Gurcan, "Digital pathology and artificial intelligence," The Lancet Oncology, vol. 20, no. 5, pp. e253-e261, May 2019.
[81] B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba, "Learning deep features for discriminative localization," in IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 2016, pp. 2921-2929.
[82] A. Esteva, B. Kuprel, R. A. Novoa, J. Ko, S. M. Swetter, H. M. Blau, an
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
