Multiclass Text Classification Using Token-Level Learning Rate Decay In Bert
DOI:
https://doi.org/10.29284/gkqq0472Keywords:
Bidirectional Encoder Representations from Transformers, contextual embeddings, Layer-wise Learning Rate Decay, Text classification, Token-level Class Importance, social media.Abstract
Text classification is a procedure of categorizing a text into classes or predefined labels, allowing effective analysis and organizing the large volume of textual data. In the growth of online textual data like news, digital platforms, or social media, an effective model for text summarization and classification. However, traditional models of text classification had difficulty capturing deep semantic dependencies and contextual embeddings among texts or words, which led to misclassification and minimized classification accuracy. To overcome these challenges, the research proposed Token Level Class Importance (TLCI) and Layer Wise Learning Rate Decay (LWLRD) named as TLLRD is integrated with Bidirectional Encoder Representations from Transformers (BERT) named as (TLLRD-BERT), which is utilized for multiclass violence text classification and enhanced classification performance. The model integrates LLRD to stabilize fine-tuning across transformer layers and TLCI to emphasize semantically significant words associated with aggressive or violent expressions. The proposed TLLRD-BERT attains an accuracy of 98.24%, 96.54% and 99.71% through YouTube comments dataset, the news headline dataset and the SMS spam detection dataset, respectively.
References
[1] L. Ou, and Z. Li, “Modeling inter-modal incongruous sentiment expressions for multi modal sarcasm detection”, Neurocomputing, Vol. 616, p. 128874, 2025.
[2] S. Tuarob, M. Satravisut, P. Sangtunchai, S. Nunthavanich, and T. Noraset, “FALCoN: Detecting and classifying abusive language in social networks using context features and unlabeled data”, Information Processing & Management, Vol. 60, No. 4, p. 103381, 2023.
[3] F.Z. El-Alami, S.O. El Alaoui, and N.E. Nahnahi, “A multilingual offensive language detection method based on transfer learning from transformer fine-tuning model,” Journal of King Saud University-Computer and Information Sciences, Vol. 34, No. 8, pp. 6048-6056, 2022.
[4] B.R. Chakravarthi, M.B. Jagadeeshan, V. Palanikumar, and R. Priyadharshini, “Offensive language identification in dravidian languages using mpnet and cnn”, International Journal of Information Management Data Insights, Vol. 3, No. 1, p. 100151, 2023.
[5] K.B. Nelatoori and H.B. Kommanti, “Attention-based bi-lstm network for abusive language detection,” IETE Journal of Research, Vol. 69, No. 11, pp. 7884-7892, 2022.
[6] X. Wang, Y. Wang, D. He, Z. Yu, Y. Li, L. Wang, J. Dang, and D. Jin, “Elevating knowledge-enhanced entity and relationship understanding for sarcasm detection”, IEEE Transactions on Knowledge and Data Engineering, Vol. 37, No. 6, pp. 3356-3371, 2025.
[7] S. Khan, I. Qasim, W. Khan, K. Aurangzeb, J. A. Khan, and M. S. Anwar, “A novel transformer attention‐based approach for sarcasm detection”, Expert Systems, Vol. 42, No. 1, p. e13686, 2025.
[8] Y.W. Ma, J.L. Chen, L.D. Chen, and Y.M. Huang, “Intelligent clickbait news detection system based on artificial intelligence and feature engineering”, IEEE Transactions on Engineering Management, Vol. 71, pp. 12509-12518, 2022.
[9] T.T. Aurpa, R. Sadik, and M.S. Ahmed, “Abusive Bangla comments detection on Facebook using transformer-based deep learning models”, Social Network Analysis and Mining, Vol. 12, No. 1, p. 24, 2021.
[10] O. Galal, A.H. Abdel-Gawad, and M. Farouk, “Rethinking of BERT sentence embedding for text classification”, Neural Computing and Applications, Vol. 36, No. 32, pp. 20245-20258, 2024.
[11] S.M.A.H. Shah, S.F.H. Shah, A. Ullah, A. Rizwan, G. Atteia, and M. Alabdulhafith, “Arabic sentiment analysis and sarcasm detection using probabilistic projections-based variational switch transformer”, IEEE Access, Vol. 11, pp. 67865-67881, 2023.
[12] R. Fatima, M.M.S. Fareed, S. Ullah, G. Ahmad, and S. Mahmood, “An optimized approach for detection and classification of spam email’s using ensemble methods”, Wireless Personal Communications, Vol. 139, No. 1, pp. 347-373, 2024.
[13] J. Bacha, F. Ullah, J. Khan, A.W. Sardar, and S. Lee, “A deep learning-based framework for offensive text detection in unstructured data for heterogeneous social media”, IEEE Access, Vol. 11, pp. 124484-124498, 2023.
[14] M. Anand, K.B. Sahay, M.A. Ahmed, D. Sultan, R.R. Chandan, and B. Singh, “Deep learning and natural language processing in computation for offensive language detection in online social networks by feature selection and ensemble classification techniques”, Theoretical Computer Science, Vol. 943, pp. 203-218, 2023.
[15] T. Yue, R. Mao, H. Wang, Z. Hu, and E. Cambria, “KnowleNet: Knowledge fusion network for multimodal sarcasm detection”, Information Fusion, Vol. 100, p. 101921, 2023.
[16] R. Ghanem and H. Erbay, “Spam detection on social networks using deep contextualized word representation”, Multimedia Tools and Applications, Vol. 82, No. 3, pp. 3697-3712, 2022.
[17] D. Vinoth and P. Prabhavathy, “An intelligent machine learning-based sarcasm detection and classification model on social networks”, The Journal of Supercomputing, Vol. 78, No. 8, pp. 10575-10594, 2022.
[18] R.T. Gedela, U. Baruah, and B. Soni, “Deep contextualised text representation and learning for sarcasm detection”, Arabian Journal for Science and Engineering, Vol. 49, No. 3, pp. 3719-3734, 2024.
[19] H.A. Al-Kabbi, M.R. Feizi-Derakhshi, and S. Pashazadeh, “Multi-type feature extraction and early fusion framework for sms spam detection”, IEEE Access, Vol. 11, pp. 123756-123765, 2023.
[20] C. Thaokar, J.K. Rout, M. Rout, and N.K. Ray, “N-Gram based sarcasm detection for news and social media text using hybrid deep learning models”, SN Computer Science, Vol. 5, No. 1, p. 163, 2024.
[21] K. Arunthavachelvan, S. Raza, C. Ding, and E. Hasan, “PL-NCC: a novel approach for fake news detection through data augmentation”, Social Network Analysis and Mining, Vol. 14, No. 1, p. 217, 2024.
[22] E.Y. Daraghmi, S. Qadan, Y.A. Daraghmi, R. Yousuf, O. Cheikhrouhou, and M. Baz, “From text to insight: An integrated cnn-bilstm-gru model for arabic cyberbullying detection”, IEEE Access, Vol. 12, pp. 103504-103519, 2024.
[23] Ç.O. Aliyeva and M. Yağanoğlu, “Deep learning approach to detect cyberbullying on twitter”, Multimedia Tools and Applications, Vol. 84, No. 19, pp. 20497-20520, 2024.
[24] Clickbait dataset access link: https://www.kaggle.com/datasets/amananandrai/clickbait-dataset/code (Accessed on October 2025).
[25] Hate speech and abusive language dataset access link: https://www.kaggle.com/datasets/mrmorj/hate-speech-and-offensive-language-dataset (Accessed on October 2025).
[26] Sarcasm detection dataset access link: https://www.kaggle.com/datasets/saurabhbagchi/sarcasm-detection-through-nlp (Accessed on October 2025).
[27] News headline dataset access link: https://www.kaggle.com/datasets/rmisra/news-headlines-dataset-for-sarcasm-detection (Accessed on October 2025).
[28] YouTube comments dataset access link: https://www.kaggle.com/datasets/atifaliak/youtube-comments-dataset (Accessed on October 2025).
[29] SMS spam detection dataset access link: https://www.kaggle.com/datasets/uciml/sms-spam-collection-dataset (Accessed on October 2025).
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
