Ensemble Machine Learning Models For Oral Lesion Classification Using A Clinically Curated Histopathological Dataset

Authors

  • Hemashree H C Department of Information Science and Engineering, Bapuji Institute of Engineering and Technology, Visvesvaraya Technological University, Belagavi – 590018, Karnataka, India.
  • Pradeep N Department of Computer Science and Engineering, Bapuji Institute of Engineering and Technology, Visvesvaraya Technological University, Belagavi – 590018, Karnataka, India.
  • Ahmed Mujib B R Department of Oral Pathology and Microbiology, Bapuji Dental College and Hospital, Davangere, affiliated to Rajiv Gandhi University of Health Sciences, Bengaluru - 560041, Karnataka, India.

DOI:

https://doi.org/10.29284/8as21m60

Keywords:

Oral Lesions, Ensemble Machine Learning, Oral Squamous Cell Carcinoma (OSCC), Hyperplasia, Early Invasive, Carcinoma in situ, Dysplasia Grading, XGBoost, Random Forest, CatBoost.

Abstract

Oral cancer, with the most common type being Oral Squamous Cell Carcinoma (OSCC), is a pervasive world-wide health problem whose successful treatment relies on early detection. In this paper, presents a novel framework of ensemble learning method for oral lesion classification. Starting with a clinically relevant dataset of 2450 histopathological images from six lesion classes, expert-annotated histopathological features were employed to train and refine three ensemble-based models: Random Forest (RF), XGBoost, and CatBoost. The methods show state-of-the-art performance, with CatBoost obtaining the highest test accuracy of 95% compared to XGBoost (94%) and Random Forest (92%). More importantly, it outperforms state-of-the-art methods in oral histopathology categorization and can compare favorably with advanced ensemble method for other cancer diagnostics. This work primary contribution is the integration of clinical domain knowledge with machine learning, taking advantage of explainable histopathological features to provide a solution both highly accurate and feasible in practice. This framework offers the automated detection of oral lesions at scale, using a reliable detection method at a lower cost, and can be particularly beneficial for clinics with fewer resources.

References

[1] Li, L., Yin, Y., Nan, F. & Ma, Z. “Circ_LPAR3 promotes the progression of oral squamous cell carcinoma (OSCC)”. Biochemical and Biophysical Research Communications, PMID: 34922206, 215–222 (2022).

DOI: 10.1016/j.bbrc.2021.12.012

[2] Saba, T. “Recent advancement in cancer detection using machine learning: Systematic survey of decades, comparisons and challenges”. Journal of Infection and Public Health 13 (9), 1274–1289 (2020), PMID: 32758393,

DOI: 10.1016/j.jiph.2020.06.033

[3] World cancer research fund international, https://www.wcrf.org/cancer-trends/mouth-andoral- cancer-statistics/

[4] Deif, M. A. & Hammam, R. E. “Skin lesions classification based on deep learning approach”. Journal of Clinical Engineering 45 (3), 155–161 (2020),

DOI:10.1097/JCE.0000000000000405

[5] Welikala, R. A. et al. “Automated detection and classification of oral lesions using deep learning for early detection of oral cancer”. IEEE Access 8, 132677–132693 (2020), DOI: 10.1109/ACCESS.2020.3010180

[6] Kar, A. et al. “Improvement of oral cancer screening quality and reach: The promise of artificial intelligence”. Oral Pathol. Med. 49, PMID: 32162398, 727–730 (2020), DOI: 10.1111/jop.13013

[7] Das, N., Hussain, E. & Mahanta, L. B. “Automated classification of cells into multiple classes in epithelial tissue of oral squamous cell carcinoma using transfer learning and convolutional neural network”. Neural Netw. 128, PMID: 32416467, 47–60 (2020), DOI: 10.1016/j.neunet.2020.05.003

[8] Shamim, M. Z. M. et al. “Automated detection of oral pre-cancerous tongue lesions using deep learning for early diagnosis of oral cavity cancer”. Comput. J. 65, 18 Sep (2019). DOI:10.13140/RG.2.2.28808.16643

[9] Sunny, S. P. et al. “Oral epithelial cell segmentation from fluorescent multichannel cytology images using deep learning”. Comput. Methods Programs Biomed. 227, PMID: 36384061, 107205 (2022), DOI: 10.1016/j.cmpb.2022.107205

[10] F.Y. Shih, “Image Processing and Pattern Recognition: Fundamentals and Techniques”, John Wiley & Sons, 2010,

https://onlinelibrary.wiley.com/doi/book/10.1002/9780470590416

[11] M.M.R. Krishnan, C. Chakraborty, R.R. Paul, A.K. Ray, “Hybrid segmentation, characterization and classification of basal cell nuclei from histopathological images of normal oral mucosa and oral submucous fibrosis”, Expert Syst. Appl. 39 (1) (2012) 1062–1077, DOI:10.1016/j.eswa.2011.07.107

[12] Jiang, H., Zhang, P., Che, C., Jin, B. & Zhu, Y. “CariesFG: A fine-grained RGB image classification framework with attention mechanism for dental caries”. Eng. Appl. Artif. Intell. 123, 106306 (2023), https://doi.org/10.1016/j.engappai.2023.106306

[13] A. Mikołajczyk, M. Grochowski, “Data augmentation for improving deep learning in image classification problem”, in 2018 International Interdisciplinary PhD Workshop (IIPhDW), IEEE, 2018, pp. 117–122,

DOI:10.1109/IIPHDW.2018.8388338

[14] Goyal, M., Oakley, A., Bansal, P., Dancey, D. & Yap, M. H. “Skin lesion segmentation in dermoscopic images with ensemble deep learning methods”. IEEE Access, 8, 4171–4181 (2020), https://e-space.mmu.ac.uk/625079/7/08936444.pdf

[15] Mathew, T., Johnpaul, C. I., Ajith, B., Kini, J. R. & Rajan, J. “A deep learning based classifier framework for automated nuclear atypia scoring of breast carcinoma”. Eng. Appl. Artif. Intell. 120, 105949 (2023),

https://doi.org/10.1016/j.engappai.2023.105949

[16] Omeroglu, A. N., Mohammed, H. M. A., Oral, E. A. & Aydin, S. “A novel soft attention-based multi-modal deep learning framework for multi-label skin lesion classification”. Eng. Appl. Artif. Intell. 120, 105897 (2023),

DOI:10.1016/j.engappai.2023.105897

[17] M.Q.P. Smith, G.D. Ruxton, “Effective use of the mcnemar test”, Behav. Ecol. Sociobiol. 74 (11) (2020) 1–9, DOI:10.1007/s00265-020-02916-y

[18] Dayananda, C., Yamanakkanavar, N., Nguyen, T. & Lee, B. “AMCC-Net: An asymmetric multi-cross convolution for skin lesion segmentation on dermoscopic images”. Eng. Appl. Artif. Intell. 122, 106154 (2023), https://doi.org/10.1016/j.engappai.2023.106154

[19] Chang, C.-C., Li, Y.-Z., Hui-Ching, W. & Tseng, M.-H. “Melanoma detection using XGB classifier combined with feature extraction and K-means SMOTE techniques”. Diagnostics 12(7), 1747 (2022), https://doi.org/10.3390/diagnostics12071747

[20] Priya Natha and Sivarama Prasad Tera. “Boosting skin cancer diagnosis accuracy with ensemble approach”. Scientific reports, (2025) 15:1290.

https://doi.org/10.1038/s41598-024-84864-5.

[21] Khan, M. A., Muhammad, K., Sharif, M., Akram, T. & Kadry, S. “Intelligent fusion-assisted skin lesion localization and classification for smart healthcare”. Neural Comput. Appl. 36(1), 37–52 (2024). https://doi.org/10.1007/s00521-021-06490-w

Downloads

Published

2026-03-09

Issue

Section

Articles

How to Cite

[1]
H. H C, P. N, and A. M. B R, “Ensemble Machine Learning Models For Oral Lesion Classification Using A Clinically Curated Histopathological Dataset”, IJASIS, pp. 737–755, Mar. 2026, doi: 10.29284/8as21m60.

Share