AI-Driven Accessibility: Automating Audio Descriptions For Video Content
DOI:
https://doi.org/10.29284/nta1vw77Keywords:
AI, videos, visual frame, multimedia content, transcripts.Abstract
Background: Traditional audio description generation for videos often relies solely on visual frame analysis, overlooking the potential of integrating advanced natural language generation and text-to-speech technologies into a unified automated pipeline. Existing solutions are either manual or semi-automated, making them time-consuming, inconsistent, and less scalable for real-world accessibility needs, especially for visually impaired audiences.
Objective: This study aims to develop a robust audio-visual captioning framework that leverages extracted video frames and corresponding audio transcripts to produce semantically coherent and contextually relevant descriptions. By evaluating the proposed method against established video captioning approaches, the study seeks to demonstrate its effectiveness in improving descriptive accuracy and contextual depth.
Methods: The system extracts frames from user-uploaded videos at one frame per second using OpenCV. Optionally, YOLOv8 is applied for object detection to improve contextual accuracy. Captions are generated for each frame using the BLIP-2 model, and Google Text-to-Speech converts these captions into high-quality narration. The processed frames and audio are then reconstructed into a final video. Evaluation is conducted using BLEU and ROUGE metrics, along with qualitative user feedback for clarity and usefulness.
Results: Experimental evaluation shows the proposed method achieves a BLEU score of 0.82 and ROUGE score of 0.84, outperforming multiple prior studies in both metrics. Generated captions demonstrate improved contextual relevance and richer descriptions, effectively integrating scene details from visual frames and semantic cues from audio transcripts for enhanced multimedia content understanding.
Conclusion: The proposed approach effectively fuses audio and visual features, generating highly accurate captions, and offers strong potential for multimedia content applications.
References
1. M. Monfort et al., “Spoken moments: Learning joint audio-visual representations from video descriptions,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 14871–14881.
2. A.-M. Oncescu, J. F. Henriques, Y. Liu, A. Zisserman, and S. Albanie, “Queryd: A video dataset with high-quality text and audio narrations,” in ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2021, pp. 2265–2269.
3. Y. Wang, W. Liang, H. Huang, Y. Zhang, D. Li, and L.-F. Yu, “Toward automatic audio description generation for accessible videos,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, 2021, pp. 1–12.
4. A. Nagrani et al., “Learning audio-video modalities from image captions,” in European Conference on Computer Vision, Springer, 2022, pp. 407–426.
5. J. Holsanova, “A cognitive approach to audio description: production and reception processes,” in The Routledge handbook of audio description, Routledge, 2022, pp. 57–77.
6. K. Lin et al., “Mm-vid: Advancing video understanding with gpt-4v (ision),” ArXiv Prepr. ArXiv231019773, 2023.
7. N. Talaván, J. Lertola, and A. I. Moreno, “Audio description and subtitling for the deaf and hard of hearing: Media accessibility in foreign language learning,” Transl. Translanguaging Multiling. Contexts, vol. 8, no. 1, pp. 1–29, 2022.
8. A.-M. Oncescu, A. Koepke, J. F. Henriques, Z. Akata, and S. Albanie, “Audio retrieval with natural language queries,” ArXiv Prepr. ArXiv210502192, 2021.
9. F. Shu, L. Zhang, H. Jiang, and C. Xie, “Audio-visual llm for video understanding,” ArXiv Prepr. ArXiv231206720, 2023.
10. S. Chen et al., “Vast: A vision-audio-subtitle-text omni-modality foundation model and dataset,” Adv. Neural Inf. Process. Syst., vol. 36, pp. 72842–72866, 2023.
11. G. Yariv, I. Gat, S. Benaim, L. Wolf, I. Schwartz, and Y. Adi, “Diverse and aligned audio-to-video generation via text-to-video model adaptation,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2024, pp. 6639–6647.
12. Y. Wang et al., “Internvideo2: Scaling foundation models for multimodal video understanding,” in European Conference on Computer Vision, Springer, 2024, pp. 396–416.
13. B. Zhu et al., “Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment,” ArXiv Prepr. ArXiv231001852, 2023.
14. K. Shimada et al., “STARSS23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events,” Adv. Neural Inf. Process. Syst., vol. 36, pp. 72931–72957, 2023.
15. N. Pellas, “The influence of sociodemographic factors on students’ attitudes toward AI-generated video content creation,” Smart Learn. Environ., vol. 10, no. 1, p. 57, 2023.
16. G. Li, Y. Wei, Y. Tian, C. Xu, J.-R. Wen, and D. Hu, “Learning to answer questions in dynamic audio-visual scenarios,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 19108–19118.
17. J. A. Portillo-Quintero, J. C. Ortiz-Bayliss, and H. Terashima-Marín, “A straightforward framework for video retrieval using clip,” in Mexican Conference on Pattern Recognition, Springer, 2021, pp. 3–12.
18. C. Chen, N. Hou, Y. Hu, H. Zou, X. Qi, and E. S. Chng, “Interactive audio-text representation for automated audio captioning with contrastive learning,” ArXiv Prepr. ArXiv220315526, 2022.
19. J. Bai et al., “Audiosetcaps: An enriched audio-caption dataset using automated generation pipeline with large audio and language models,” IEEE Trans. Audio Speech Lang. Process., 2025.
20. M. Haji-Ali, W. Menapace, A. Siarohin, G. Balakrishnan, and V. Ordonez, “Taming data and transformers for audio generation,” ArXiv Prepr. ArXiv240619388, 2024.
21. R. Indoriya, P. Aggarwal, C. Jindal, S. Gupta, and G. Singh, “AI Driven Approach for Description of Visual,” 2025.
22. P. A. Sadique and R. Aswiga, “Automatic summarization of cooking videos using transfer learning and transformer-based models,” Discov. Artif. Intell., vol. 5, no. 1, p. 7, 2025.
23. L. Agarwal and B. Verma, “Towards Explainable AI: Multi-Modal Transformer for Video-based Image Description Generation,” ArXiv Prepr. ArXiv250416788, 2025.
24. M. S. Zaoad, M. R. Mannan, A. B. Mandol, M. Rahman, M. A. Islam, and M. M. Rahman, “An attention-based hybrid deep learning approach for Bengali video captioning,” J. King Saud Univ.-Comput. Inf. Sci., vol. 35, no. 1, pp. 257–269, 2023.
25. K. Safiya and R. Pandian, “A real-time image captioning framework using computer vision to help the visually impaired,” Multimed. Tools Appl., vol. 83, no. 20, pp. 59413–59438, 2024.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
