Sentence Vectors Enable GRU And Multi-Headed Attention-Based Video Captioning Framework For Specifying Events In MSVD Video Dataset

K. Annapoorneshwari Shetty; Subrahmanya Bhat

doi:10.29284/mndwr104

Authors

K. Annapoorneshwari Shetty Research Scholar Srinivas University
Subrahmanya Bhat Professor Srinivas University

DOI:

https://doi.org/10.29284/mndwr104

Keywords:

Sentence Vector Branch, Skip Connection, Long Short-Term Memory (LSTM), Gated Recurrent Unit (GRU), Convolutional Neural Network (CNN).

Abstract

The fast growing digital culture has attracted people’s attention and motivated them to interact with fascinating multimedia materials, including images, videos, voice notes and messages. Video is the most often used digital material format globally; it is used for marketing for companies, education, proof of concern, memorial ceremonies, and amusement among other purposes. The primary goal of video/image captioning is to represent the dynamics of a video clip using simple and understandable conversational language. The motivation for captioning comes from its ability to enhance video accessibility for those who are deaf or hard-of-hearing, improve concentration and memory of information and allow viewing in sound-sensitive environments. In this proposed framework, we demonstrate how to create captions for videos using Gated Recurrent Unit (GRU) along with a Sentence Vector Branch (SVB) and skip connections employing two word embedding methods: Bidirectional Encoder Representations from Transformers (BERT) and Embeddings from Language Model (ELMO). The encoder section of the model uses skip connections to address the issue of vanishing gradients and the sentence vector branch is used to improve the efficiency of the encoder-decoder architecture. The decoder generates a suitable description using the sentence vector branch together with past generated words. All of our experiments make use of the Standard MSVD databases. BLEU@4, METEOR, ROUGE and CIDEr measures have been applied for the performance assessment. Comparative analysis of BERT and ELMO word embeddings reveals their respective strengths. Features are extracted from pre-trained CNNs NASNet-Large, VGG16, Inceptionv3, DenseNet-169 and ResNet50. In the MSVD dataset, BERT beats ELMO in the case of the GRU-based model, whereas ELMO beats BERT in the case of the LSTM-based model; among LSTM and GRU, LSTM outperforms GRU. DenseNet-169 beats every feature extraction method used in the MSVD dataset.

References

[1] J. Barnard, “Online video viewing to reach 100 minutes a day in 2021 – Zenith — zenithmedia.com,” https://www.zenithmedia.com/ online-video-viewing -to-reach-100-minutes-a-day-in-2021/, 2021, [Accessed 06-Jun-2023].

[2] D. Naik and C. Jaidhar, “A novel multi-layer attention framework for visual description prediction using bidirectional lstm,” journal of big data, vol. 9, 11 2022.

[3] M. Mohsin, “10 YouTube Statistics That You Need to Know in 2022 — oberlo.com,” https://www.oberlo.com/blog/youtube-statistics, 2022.

[4] N. Patwari and D. Naik, “En-de-cap: An encoder decoder model for image captioning,” in 2021 5th international conference on computing methodologies and communication (ICCMC). IEEE, 2021, pp. 1192–1196.

[5] D. Naik and C. Jaidhar, “A novel multi-layer attention framework for visual description prediction using bidirectional lstm,” journal of big data, vol. 9, no. 1, p. 104, 2022.

[6] R. M. R. Guddeti, “Human action recognition using multi-stream attention-based deep networks with heterogeneous data from overlapping sub-actions,” neural computing and applications, vol. 36, no. 18, pp. 10 681–10 697, 2024.

[7] J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,” CoRR, vol. abs/1810.04805, 2018.

[8] M. E. Peters, M. Neumann, M. Iyyer, M. Gardner, C. Clark, K. Lee, and L. Zettlemoyer, “Deep contextualized word representations,” CoRR, vol. abs/1802.05365, 2018. [Online]. Available: http://arxiv.org/abs/1802.05365.

[9] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” in the 3rd International Conference on Learning Representations (ICLR2015), 2015. [Online]. Available: https://arxiv.org/abs/1409.1556

[10] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.

[11] G. Huang, Z. Liu, L. van der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” 2018.

[12] H. Liu, F. Qu, Y. Yang, W. Li, and Z. Hao, “Soybean variety identification based on improved resnet18 hyperspectral image,” in journal of physics: conference series, vol. 2284, no. 1. IOP Publishing, 2022, p. 012017.

[13] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” in proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826.

[14] H. Mule and D. Naik, “Video to text generation using sentence vector and skip connections,” in international conference on machine learning and big data analytics. Springer, 2022, pp. 515–527.

[15] M. Rashmi and R. M. R. Guddeti, “Multi-stream multi-attention deep neural network for context-aware human action recognition,” in 2022 IEEE Region 10 Symposium (TENSYMP). IEEE, 2022, pp. 1–6.

[16] N. Yadav and D. Naik, “Generating short video description using deep-lstm and attention mechanism,” in 2021 6th international conference for convergence in technology (I2CT). IEEE, 2021, pp. 1–6.

[17] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” advances in neural information processing systems, vol. 27, 2014.

[18] R. M and R. M. R. Guddeti, “Human action recognition using multi-stream attention-based deep networks with heterogeneous data from overlapping sub-actions,” neural computing and applications, vol. 36, no. 18, pp. 10 681–10 697, June 2024. [Online]. Available: https://doi.org/10.1007/s00521-024-09630-0.

[19] S. Venugopalan, M. Rohrbach, J. Donahue, R. Mooney, T. Darrell, and K. Saenko, “Sequence to sequence - video to text,” in proceedings of the IEEE international conference on computer vision (ICCV), December 2015.

[20] M. M. Rahman, T. Abedin, K. S. S. Prottoy, A. Moshruba, and F. H. Siddiqui, “Video captioning with stacked attention and semantic hard pull,” peerj computer science, vol. 7, p. e664, 2021.

[21] D. Naik and C. Jaidhar, “Semantic context driven language descriptions of videos using deep neural network,” journal of big data, vol. 9, no. 1, p. 17, 2022.

[22] D. Naik and J. CD, “Video captioning using sentence vector-enabled convolutional framework with short-connected lstm,” multimedia tools and applications, vol. 83, no. 4, pp. 11 187–11 213, 2024.

[23] P. Shailendra, M. Rashmi, S. Ramu, and R. M. R. Guddeti, “Fake news detection in hindi using embedding techniques,” in 2022 IEEE Region 10 Symposium (TENSYMP). IEEE, 2022, pp. 1–5.

[24] H. Maru, T. Chandana, and D. Naik, “Comparitive study of gru and lstm cells based video captioning models,” in 2021 12th international conference on computing communication and networking technologies (ICCCNT). IEEE, 2021, pp. 1–5.

[25] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Jul. 2002, pp. 311–318.

[26] S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization. Ann Arbor, Michigan: Association for Computational Linguistics, Jun. 2005, pp. 65–72.

[27] C.-Y. Lin, “ROUGE: A package for automatic evaluation of summaries,” in text summarization branches out. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81. [Online]. Available: https://www.aclweb.org/anthology/W04-1013

[28] R. Vedantam, C. L. Zitnick, and D. Parikh, “CIDEr: Consensus-based image description evaluation,” CoRR, vol. abs/1411.5726, 2014.

[29] D. Chen and W. Dolan, “Collecting highly parallel data for paraphrase evaluation,” in proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Portland, Oregon, USA: Association for Computational Linguistics, Jun. 2011, pp. 190–200.

[30] B. Zoph, V. Vasudevan, J. Shlens, and Q. V. Le, “Learning transferable architectures for scalable image recognition,” in proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 8697–8710.

[31] B. Sun, H. Dai, J. Sun, and X. Wei, “Research on intrusion detection models with multi-head attention mechanism based on graph neural network.” engineering letters, vol. 33, no. 5, 2025.

[32] Z. Mahmoud, N. Kryvinska, M. Abdalsalm, A. Solyman, A. Alfatemi, and A. Musyafa, “Toward utilizing bidirectional multi-head attention technique for automatic correction of grammatical errors.” IAENG international journal of computer science, vol. 51, no. 11, pp. 1886–1897, 2024.

[33] L. Yao, A. Torabi, K. Cho, N. Ballas, C. Pal, H. Larochelle, and A. Courville, “Describing videos by exploiting temporal structure,” in proceedings of the IEEE international conference on computer vision, 2015, pp. 4507–4515.

[34] S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, “Self-critical sequence training for image captioning,” in proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7008–7024.

[35] L. Baraldi, C. Grana, and R. Cucchiara, “Hierarchical boundary-aware neural encoder for video captioning,” in 2017 IEEE conference on computer vision and pattern recognition (CVPR). IEEE, jul 2017. [Online]. Available: https://doi.org/10.1109%2Fcvpr.2017.339

[36] G. Li, S. Ma, and Y. Han, “Summarization-based video caption via deep neural networks,” in proceedings of the 23rd ACM international conference on multimedia, ser. MM ’15. New York, NY, USA: association for computing machinery, 2015, p. 1191–1194. [Online]. Available: https://doi.org/10.1145/2733373.2806314 [37] J. Xu, T. Mei, T. Yao, and Y. Rui, “Msr-vtt: A large video description dataset for bridging video and language,” in proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 5288–5296.