Domain-Adaptive Transformer-Based Abstractive Summarization Of Social Media Discourse: A Comprehensive Evaluation Framework
DOI:
https://doi.org/10.29284/5xpz1d02Keywords:
Social Media Summarization, Transformers, Abstractive Summarization, Natural Language Generation, Reference-Free Evaluation.Abstract
The exponential growth of social media platforms has generated vast streams of unstructured, linguistically diverse user-generated content that, while rich in public sentiment and cultural insight, presents significant challenges for automated analysis. This study presents a domain-adaptive framework for abstractive summarization specifically engineered to address the linguistic complexity, structural fragmentation, and semantic irregularities characteristic of social media discourse. Leveraging the Text-to-Text Transfer Transformer (T5-small) architecture fine-tuned on Facebook news comment data comprising 1,781,576 records, our system generates concise, semantically faithful summaries that preserve contextual meaning while reducing cognitive overload. Rigorous empirical evaluation across reference-based (ROUGE variants), reference-free (SUPERT, Shannon metrics, SDC), and human-aligned quality indicators demonstrates that the proposed framework consistently outperforms ChatGPT-4.0, achieving a ROUGE-1 F1-score of 0.3292 (20% relative improvement), SUPERT score of 0.9990, and Shannon Coherency of 0.9519. The system's sophisticated preprocessing pipeline and attention-driven encoder-decoder architecture enable extraction of both factual content and affective, cultural, and pragmatic nuances embedded within informal text. This work makes three principal contributions: (1) demonstration that strategically adapted transformer architectures effectively accommodate unique properties of social media discourse while maintaining semantic faithfulness, (2) introduction of a comprehensive multidimensional evaluation framework integrating lexical, semantic, and human-centered quality metrics, and (3) empirical validation that domain-specific fine-tuning yields superior performance compared to zero-shot prompting of general-purpose large language models. These findings advance automated text summarization methodologies and provide critical tools for sentiment analysis, trend monitoring, public opinion mining, and strategic decision-making across governmental, commercial, and civic engagement domains.
References
[1] A. Papagiannopoulou, and C. Angeli, “Summarizing User Comments on Social Media Using Transformers”, In 11th European Conference Proceedings on Social Media (ECSM-2024), 198-205, ACI, Brighton, UK, 2024.
[2] H. P. Luhn, “The Automatic Creation of Literature Abstracts”, IBM Journal of Research and Development, vol. 2, no. 2, 1958, pp. 159-165.
[3] S. Gupta, and S. K. Gupta, “Abstractive Summarization: An Overview of the State of The Art”, Expert Systems with Applications 121, 2019, pp. 49–65.
[4] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, “Attention Is All You Need”, In 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA, 2017.
[5] Q. Li, and Q. Zhang, “Abstractive Event Summarization on Twitter”, In The Web Conference 2020 - Companion of the World Wide Web Conference, WWW 2020, Association for Computing Machinery: 22–23, 2020.
[6] Z. Kerui, H. Haichao, L. Yuxia, “Automatic text summarization on social media”, ACM International Conference Proceeding Series, Association for Computing Machinery, 2020.
[7] I. S. Blekanov, N. Tarasov and S. S. Bodrunova, “Transformer-Based Abstractive Summarization for Reddit and Twitter: Single Posts vs. Comment Pools in Three Languages”, Future Internet 14, 69, 2022.
[8] S. Gao, X. Chen, P. Li, Z. Ren, L. Bing, D. Zhao, and R. Yan, “Abstractive Text Summarization by Incorporating Reader Comments”, In The Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19), 2019, 33(01): 6399-6406.
[9] P. Bhandarkar, K. T. Thomas, “Text Summarization Using Combination of Sequence-To-Sequence Model with Attention Approach”, Springer Science and Business Media Deutschland GmbH, 2023, 283–293.
[10] Z. Liang, J. Du, and C. Li, “Abstractive social media text summarization using selective reinforced Seq2Seq attention model”, Neurocomputing, 410, 2020, 432–440.
[11] Q. Wang, and J. Ren, “Summary-aware attention for social media short text abstractive summarization”, Neurocomputing, 425, 2021, 290–299.
[12] A. Gupta, D. Chugh, and R. Katarya, “Automated News Summarization Using Transformers”, In Sustainable Advanced Computing, Volume 840, 2022. ISBN: 978-981-16-9011-2.
[13] J. Devlin, M-W. Chang, K. Lee, K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding”, Proceedings of NAACL-HLT, 2019, pp. 4171–4186.
[14] M. H. Su, C. H. Wu, H. T. Cheng, “A Two-Stage Transformer-Based Approach for Variable-Length Abstractive Summarization”, IEEE/ACM Transactions on Audio Speech and Language Processing, 28, 2020, 2061–2072.
[15] D. Singhal, K. Khatter, A. Tejaswini, R. Jayashree, “Abstractive Summarization of Meeting Conversations”, In 2020 IEEE International Conference for Innovation in Technology, INOCON 2020, Institute of Electrical and Electronics Engineers Inc.
[16] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narag, M. Matena, Y. Zhou, W. Li, and P. J. Liu, “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”, The Journal of Machine Learning Research, Volume 21, Issue 1, 2019, 140:1-140:67.
[17] A. Pal, L. Fan, V. Igodifo, “Text Summarization using BERT and T5”, https://anjali001.github.io/Project_Report.pdf, 2022.
[18] Nguyen, M. T., Nguyen, V. C., Vu, H. T., Nguyen, V. H. (2020) Transformer-based Summarization by Exploiting Social Information. in Proceedings - 2020 12th International Conference on Knowledge and Systems Engineering, KSE 2020, Institute of Electrical and Electronics Engineers Inc.: 2020, 25–30.
[19] R. Rawat, P. Rawat, V. Elahi, and A. Elahi, “Abstractive Summarization on Dynamically Changing Text”, In 5th International Conference on Computing Methodologies and Communication (ICCMC), Erode, India, 2021, pp. 1158-1163.
[20] P. Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing”, ACM Computing Surveys 55, 2021, 1-35.
[21] N. Ding, S. Hu, W. Zhao, Y. Chen, Z. Liu, H. Zheng, et al, “OpenPrompt: An Open-source Framework for Prompt-learning”, In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations; 2022, pp. 105-13.
[22] X. L. Li, and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation”, In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, 4582–4597.
[23] B. Lester, R. Al-Rfou, and N. Constant, “The Power of Scale for Parameter-Efficient Prompt Tuning”, In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021, 3045–3059.
[24] J. Wang, E. Shi, S. Yu, Z. Wu, C. Ma, H. Dai, Q. Yang, Y. Kang, J. Wu, H. Hu, C. Yue, H. Zhang, Y. Liu, X. Li, B. Ge, D. Zhu, Y. Yuan, D. Shen, T. Liu, and S. Zhang, “Prompt Engineering for Healthcare: Methodologies and Applications”, https://arxiv.org/html/2304.14670v2, 2024.
[25] A. Papagiannopoulou, and C. Angeli, “Designing a summarization system on social comments using transformers”, In 12th International Conference Proceedings on Computer Technologies and Development (TechDev), 2023, 1-5, IEEE, Rome, Italy.
[26] C. Y. Lin, “Rouge: A package for automatic evaluation of summaries”, In Text Summarization Branches Out, 2004, 74–81, Barcelona, Spain. Association for Computational Linguistics.
[27] J-P. Ng, and V. Abrecht, “Better Summarization Evaluation with Word Embeddings for ROUGE”, In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal. Association for Computational Linguistics, 2015, 1925–1930.
[28] Y. Gao, W. Zhao, S. Eger, “SUPERT: towards new frontiers in unsupervised evaluation metrics for multi-document summarization”, In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pages 1347–1354, Association for Computational Linguistics.
[29] N. Egan, O. Vasilyev, J. Bohannon, “Play the Shannon game with language models: a human-free approach to summary evaluation”, Proceedings of the AAAI conference on artificial intelligence, 2022, 10599–10607.
[30] Y. Liu, Q. Jia, K. Zhu, “Reference-free summarization evaluation via semantic correlation and compression ratio”. Proceedings of the 2022 conference of the North American Chapter of the Association for Computational Linguistics: human language technologies, 2022b, 2109–2115.
[31] W. Kryściński, B. McCann, C. Xiong, et al, “Evaluating the factual consistency of abstractive text summarization”, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020, pages 9332–9346, Association for Computational Linguistics.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
