Self Attention K Folded With Xgboost
DOI:
https://doi.org/10.29284/9wd74n30Keywords:
Sarcasm detection, mBERT, multilingual NLP, English, Hindi, Bengali, PunjabiAbstract
Sarcasm detection is a challenging task in Natural Language Processing (NLP) due to its reliance on contextual, contrastive, and figurative cues. While deep learning models such as BERT have advanced contextual representation learning, they often rely on computationally expensive end-to-end fine-tuning. This paper proposes a hybrid framework that integrates BERT embeddings with a custom self-attention mechanism for feature extraction, combined with XGBoost classification for efficient decision making. The self-attention layer highlights sarcasm-relevant tokens, producing contextualized, weighted sentence representations, which are then fed into XGBoost to establish robust decision boundaries without neural classification overhead. We further employ Stratified 6-fold cross validation to ensure balanced evaluation and reliable performance reporting. Experiments on benchmark sarcasm datasets demonstrate that the proposed approach achieves superior F1-scores compared to traditional baselines, validating the effectiveness of combining contextual embeddings with gradient-boosted classification. This work highlights the potential of attention-augmented embeddings with ensemble learning for advancing sarcasm detection in social media text.
References
[1] Joseph, Thomas. "Natural language processing (NLP) for sentiment analysis in social media." International Journal of Computing and Engineering 6.2 (2024): 35-48.
[2] Chen, Shijie, Yu Zhang, and Qiang Yang. "Multi-task learning in natural language processing: An overview." ACM Computing Surveys 56.12 (2024): 1-32.
[3] Gardazi, Nadia Mushtaq, et al. "BERT applications in natural language processing: a review." Artificial Intelligence Review 58.6 (2025): 1-49.
[4] Meng, Jiana, et al. "Sarcasm detection based on BERT and attention mechanism." Multimedia Tools and Applications 83.10 (2024): 29159-29178.
[5] Javed, Tayyaba, Muhammad Asif Nouman, and Rushna Zahid. "BERT model adoption for sarcasm detection on Twitter data." VFAST Transactions on Software Engineering 12.3 (2024): 177-198.
[6] Bharati, Rajesh, et al. "SVM-based Sarcasm Detection System: NLP Using Heuristic Approach." 2024 8th International Conference on Computing, Communication, Control and Automation (ICCUBEA). IEEE, 2024.
[7] Starbuck, Craig. "Logistic regression." The fundamentals of people analytics: With applications in R. Cham: Springer International Publishing, 2023. 223-238.
[8] Sharma, Surbhi, and Nisheeth Joshi. "A fusion approach to detect sarcasm using NLTK models BERT and XG Boost."
[9] Bohacek, Matyas, and Michal Bravansky. "When XGBoost outperforms GPT-4 on text classification: A case study." Proceedings of the 4th Workshop on Trustworthy Natural Language Processing (TrustNLP 2024). 2024.
[10] Pandey, Rajnish, et al. "A hybrid convolutional neural network for sarcasm detection from multilingual social media posts." Multimedia Tools and Applications 84.16 (2025): 15867-15895.
[11] Pandey, Rajnish, and Jyoti Prakash Singh. "BERT-LSTM model for sarcasm detection in code-mixed social media post." Journal of Intelligent Information Systems 60.1 (2023): 235-254.
[12] Meng, Jiana, et al. "Sarcasm detection based on BERT and attention mechanism." Multimedia Tools and Applications 83.10 (2024): 29159-29178.
[13] Mienye, Ibomoiye Domor, Theo G. Swart, and George Obaido. "Recurrent neural networks: A comprehensive review of architectures, variants, and applications." Information 15.9 (2024): 517.
[14] Orvieto, Antonio, et al. "Resurrecting recurrent neural networks for long sequences." International Conference on Machine Learning. PMLR, 2023.
[15] Šandor, Daniel, and Marina Bagić Babac. "Sarcasm detection in online comments using machine learning." Information Discovery and Delivery 52.2 (2024): 213-226.
[16] Ali, Rasikh, et al. "Deep learning for sarcasm identification in news headlines." Applied Sciences 13.9 (2023): 5586.
[17] Anan, Ramisa, et al. "Interpretable bangla sarcasm detection using bert and explainable ai." 2023 IEEE 13th annual computing and communication workshop and conference (CCWC). IEEE, 2023.
[18] Aftan, Sulaiman, and Habib Shah. "A survey on bert and its applications." 2023 20th Learning and Technology Conference (L&T). IEEE, 2023.
[19] Szeghalmy, Szilvia, and Attila Fazekas. "A comparative study of the use of stratified cross-validation and distribution-balanced stratified cross-validation in imbalanced learning." Sensors 23.4 (2023): 2333.
[20] Ren, Yafeng, et al. "A knowledge-augmented neural network model for sarcasm detection." Information Processing & Management 60.6 (2023): 103521.
[21] Pal, Moumita, and Rajesh Prasad. "Sarcasm detection followed by sentiment analysis for Bengali language: neural network & supervised approach." 2023 International Conference on Advances in Intelligent Computing and Applications (AICAPS). IEEE, 2023.
[22] Ganganwar, Vaishali, et al. "Sarcasm and humor detection in code-mixed Hindi data: A survey." International Conference on Computing and Machine Learning. Singapore: Springer Nature Singapore, 2024.
[23] Indira Priyadarshini, Y., et al. "Sarcasm Detection in Hindi Social Media Text Using HindiBERT-LSTM and XGBoost Models." International Conference on Emerging Research in Computing, Information, Communication, Artificial Intelligence and Machine Learning. Singapore: Springer Nature Singapore, 2024.
[24] Biswas, Susmoy, et al. "BanglaSarc3: A benchmark dataset for Bangla sarcasm detection from social media to advance Bangla NLP." Data in Brief (2025): 111953.
[25] Lora, Sanzana Karim, et al. "Ben-Sarc: A self-annotated corpus for sarcasm detection from Bengali social media comments and its baseline evaluation." Natural Language Processing 31.2 (2025): 674-699.
[26] Chakraborty, Sovon, et al. "An Analytical Review of Preprocessing Techniques in Bengali Natural Language Processing." IEEE Access (2025).
[27] Prajapati, Abhinav, et al. "Sarcasm Detection in English-Hindi Code-Mixed Text Data." 2024 2nd International Conference on Advances in Computation, Communication and Information Technology (ICAICCIT). Vol. 1. IEEE, 2024.
[28] Tejasvi, Koti, et al. "Sarcasm Detection for Hindi English Code Mixed Twitter Data." International Journal for Research in Applied Science and Engineering Technology 11.3 (2023): 159-164.
[29] Kumar, Akshi, et al. "Hybrid deep learning model for sarcasm detection in Indian indigenous language using word-emoji embeddings." ACM Transactions on Asian and Low-Resource Language Information Processing 22.5 (2023): 1-20.
[30] Sultana, Zinnia, et al. "Multilingual Sarcasm Detection Using Deep Learning and Transformed Based Model." 2025 2nd International Conference on Next-Generation Computing, IoT and Machine Learning (NCIM). IEEE, 2025.
[31] Mercioni, Marina Adriana, and Stefan Holban. "A brief review of the most recent activation functions for neural networks." 2023 17th International Conference on Engineering of Modern Electric Systems (EMES). IEEE, 2023.
[32] Islam, Muhammad, and Muhammad Azhar. "Sarcasm Detection in Multilingual Text through Embedding-Enhanced Language Models: BERT Variants." 2024 26th International Multi-Topic Conference (INMIC). IEEE, 2024.
[33] Ghous, Hamid, et al. "Navigating sarcasm in multilingual text: An in-depth exploration and evaluation." Journal of Computing & Biomedical Informatics (2024).
[34] John-Otumu, A. M., et al. "A Machine Learning Approach to Irony Detection in Text Using TF-IDF and Random Forests."
[35] Fkih, Fethi, Delel Rhouma, and Hajar Alghofaily. "A semantic approach for sarcasm identification for preventing fake news spreading on social networks." International Journal of Information Technology (2024): 1-19.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
