Llm-Based Generative Error Correction For Hindi And Punjabi Asr Using In-Context Learning And Task-Activating Prompting
DOI:
https://doi.org/10.29284/w99v2389Keywords:
Generative Error Correction, In-Context Learning, Large Language Models, Low-resource Speech Processing, Task-Activating Prompting.Abstract
With the rapid development of Automatic Speech Recognition (ASR) technology, end-to-end (E2E) speech recognition systems have gained significant attention due to their ability to directly convert raw speech signals into text. However, such systems heavily rely on large amounts of labeled speech data, which severely limits model training performance and generalization. To address these limitations, Large Language Models (LLMs) have emerged as a powerful substrate for Generative Error Correction (GER), which provide the semantic and phonetic reasoning necessary to refine acoustic hypotheses. This paper proposes a novel E2E ASR framework that utilizes a deep-fusion strategy to tightly couple an LLM with an E2E decoder through contextualized prompting. Unlike traditional methods that require resource-intensive fine-tuning, our approach applies In-Context Learning (ICL) and Task-Activating Prompting (TAP) to guide the LLM through a step-by-step reasoning process. This allows the model to synthesize corrections for tokens entirely absent from the initial N-best list. It effectively surpassing the oracle WER upper bound that limits conventional re-ranking strategies. Experimental results demonstrate that proposed framework significantly improves recognition accuracy and stabilizes outputs under low-resource conditions. By shifting the paradigm from discriminatory rescoring to generative synthesis, we provide a reproducible evaluation benchmark and release as pre-trained models to facilitate future research in LLM-enhanced ASR.
References
1. D. Wang, X. Wang, and S. Lv, "An Overview of End-to-End Automatic Speech Recognition," Symmetry, vol. 11, no. 8, p. 1018, 2019.
2. R. Prabhavalkar, T. Hori, T. N. Sainath, R. Schlüter, and S. Watanabe, "End-to-End Speech Recognition: A Survey," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 32, pp. 325-351, 2024.
3. K. Deng et al., "Improving CTC-Based Speech Recognition Via Knowledge Transferring from Pre-Trained Language Models," in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 8517-8521.
4. Y. Nakagome, T. Komatsu, Y. Fujita, S. Ichimura, and Y. Kida, "InterAug: Augmenting Noisy Intermediate Predictions for CTC-based ASR," in Interspeech 2022, 2022, pp. 5140-5144.
5. T. Moriya et al., "Streaming End-to-End Speech Recognition for Hybrid RNN-T/Attention Architecture," in Interspeech 2021, Brno, Czechia, 2021.
6. D. Albesano, J. Andrés-Ferrer, N. Ferri, and P. Zhan, "On the Prediction Network Architecture in RNN-T for ASR," in Interspeech 2022, Incheon, Korea, 2022.
7. S. Kim, T. Hori, and S. Watanabe, "Joint CTC-attention based end-to-end speech recognition using multi-task learning," in 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2017, pp. 4835-4839.
8. S. Ling, G. Ye, R. Zhao, and Y. Gong, "Hybrid Attention-Based Encoder-Decoder Model for Efficient Language Model Adaptation," in 2024 IEEE Spoken Language Technology Workshop (SLT), 2024, pp. 49-55.
9. K. Rao, H. Sak, and R. Prabhavalkar, "Exploring architectures, data and units for streaming end-to-end speech recognition with rnn-transducer," in 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2017, pp. 193-199: IEEE.
10. S. Karita, N. E. Y. Soplin, S. Watanabe, M. Delcroix, A. Ogawa, and T. Nakatani, "Improving Transformer-Based End-to-End Speech Recognition with Connectionist Temporal Classification and Language Model Integration," in Interspeech 2019, 2019, pp. 1408-1412.
11. Y. C. Chen, S. F. Huang, H. y. Lee, Y. H. Wang, and C. H. Shen, "Audio Word2vec: Sequence-to-Sequence Autoencoding for Unsupervised Learning of Audio Segmentation and Representation," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 9, pp. 1481-1493, 2019.
12. A. Sriram, H. Jun, S. Satheesh, and A. Coates, "Cold Fusion: Training Seq2Seq Models Together with Language Models," in Interspeech 2018, 2018, pp. 387-391.
13. S. Toshniwal, A. Kannan, C.-C. Chiu, Y. Wu, T. N. Sainath, and K. Livescu, "A comparison of techniques for language model integration in encoder-decoder speech recognition," in 2018 IEEE spoken language technology workshop (SLT), 2018, pp. 369-375: IEEE.
14. L. Xu et al., "RescoreBERT: Discriminative Speech Recognition Rescoring With Bert," in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 6117-6121.
15. H. Futami, H. Inaguma, S. Ueno, M. Mimura, S. Sakai, and T. Kawahara, "Distilling the Knowledge of BERT for Sequence-to-Sequence ASR," in Interspeech 2020, Shanghai, China, 2020.
16. Y. Leng et al., "SoftCorrect: error correction with soft detection for automatic speech recognition," presented at the Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, 2023. Available: https://doi.org/10.1609/aaai.v37i11.26531
17. C. Chen, Y. Hu, C. H. H. Yang, S. M. Siniscalchi, P. Y. Chen, and E. S. Chng, "HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models," in Advances in Neural Information Processing Systems, 2023, pp. 31665-31688: Curran Associates, Inc.
18. C. H. H. Yang, Y. Gu, Y. C. Liu, S. Ghosh, I. Bulyko, and A. Stolcke, "Generative Speech Recognition Error Correction With Large Language Models and Task-Activating Prompting," in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2023, pp. 1-8.
19. S. Radhakrishnan et al., "Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition," Singapore, 2023, pp. 10007-10016: Association for Computational Linguistics.
20. E. J. Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models," in International Conference on Learning Representations (ICLR), 2022.
21. J. Wei et al., "Chain-of-thought prompting elicits reasoning in large language models," presented at the Proceedings of the 36th International Conference on Neural Information Processing Systems, New Orleans, LA, USA, 2022.
22. D. Kakwani et al., "IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages," Online, 2020, pp. 4948-4961: Association for Computational Linguistics.
23. E. Arisoy, A. Sethy, B. Ramabhadran, and S. Chen, "Bidirectional recurrent neural network language models for automatic speech recognition," in 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5421-5425.
24. T. Udagawa, M. Suzuki, G. Kurata, N. Itoh, and G. Saon, "Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systems," in Interspeech 2022, 2022, pp. 3919-3923.
25. J. Salazar, D. Liang, T. Q. Nguyen, and K. Kirchhoff, "Masked Language Model Scoring," Online, 2020, pp. 2699-2712: Association for Computational Linguistics.
26. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," Minneapolis, Minnesota, 2019, pp. 4171-4186: Association for Computational Linguistics.
27. L. Zhuang, L. Wayne, S. Ya, and Z. Jun, "A Robustly Optimized BERT Pre-training Approach with Post-training," Huhhot, China, 2021, pp. 1218-1227: Chinese Information Processing Society of China.
28. N. M. T. Anh and T. H. Sy, "Improving Speech Recognition with Prompt-based Contextualized ASR and LLM-based Re-predictor," in Interspeech 2024, 2024, pp. 737-741.
29. I. E. Kang, C. V. Gysel, and M. H. Siu, "Transformer-based Model for ASR N-Best Rescoring and Rewriting," in Interspeech 2024, 2024, pp. 3505-3509.
30. K. Abida, F. Karray, and W. Abida, "A Novel Voting Scheme for ROVER Using Automatic Error Detection," Berlin, Heidelberg, 2012, pp. 175-183: Springer Berlin Heidelberg.
31. K. Audhkhasi, A. M. Zavou, P. G. Georgiou, and S. S. Narayanan, "Theoretical Analysis of Diversity in an Ensemble of Automatic Speech Recognition Systems," IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 22, no. 3, pp. 711-726, 2014.
32. Y. Fang et al., "Fewer Hallucinations, More Verification: A Three-Stage LLM-Based Framework for ASR Error Correction," arXiv preprint, arXiv:2505.24347 no. cs.CL, 2025.
33. H. Touvron et al., "LLaMA: Open and Efficient Foundation Language Models," arXiv preprint, arXiv:2302.13971 no. cs.CL, 2023.
34. S. Zhang et al., "OPT: Open Pre-trained Transformer Language Models," ArXiv, vol. abs/2205.01068, 2022.
35. A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, "Robust speech recognition via large-scale weak supervision," presented at the Proceedings of the 40th International Conference on Machine Learning, Honolulu, Hawaii, USA, 2023.
36. S. C. Y. Chan et al., "Data distributional properties drive emergent in-context learning in transformers," presented at the Proceedings of the 36th International Conference on Neural Information Processing Systems, New Orleans, LA, USA, 2022.
37. A. Lampinen et al., "Can language models learn from explanations in context?," Abu Dhabi, United Arab Emirates, 2022, pp. 537-563: Association for Computational Linguistics.
38. O. Rubin, J. Herzig, and J. Berant, "Learning To Retrieve Prompts for In-Context Learning," Seattle, United States, 2022, pp. 2655-2671: Association for Computational Linguistics.
39. Y. Zhang, K. Zhou, and Z. Liu, "What Makes Good Examples for Visual In-Context Learning?," in Advances in Neural Information Processing Systems, 2023, pp. 17773-17794: Curran Associates, Inc.
40. S. Lu, I. Bigoulaeva, R. S. Sachdeva, H. Tayyar Madabushi, and I. Gurevych, "Are Emergent Abilities in Large Language Models just In-Context Learning?," vol. 1, L.-W. Ku, A. F. T. Martins, and V. Srikumar, Eds., ed. Bangkok, Thailand: Association for Computational Linguistics, 2024, pp. 5098-5139.
41. Z. Yang, P. Li, and Y. Liu, "Failures Pave the Way: Enhancing Large Language Models through Tuning-free Rule Accumulation," Singapore, 2023, pp. 1751-1777: Association for Computational Linguistics.
42. R. Ullah et al., "End-to-End Deep Convolutional Recurrent Models for Noise Robust Waveform Speech Enhancement," Sensors, vol. 22, no. 20, p. 7782, 2022.
43. D. Chen and K. Xu, "AReLU: Attention-based Rectified Linear Unit," arXiv preprint arXiv, vol. abs/2006.13858, 2020.
44. T. N. Sainath et al., "Two-Pass End-to-End Speech Recognition," in Proceedings of Interspeech 2019, 2019, pp. 2773-2777.
45. J. K. Sandhu, M. Kumar, and A. Singh, "Punjabi Speech Corpus with Tonal Characteristics," Singapore, 2026, pp. 129-141: Springer Nature Singapore.
46. B. Saglam, X. Hu, Z. Yang, D. Kalogerias, and A. Karbasi, "Learning Task Representations from In-Context Learning," Vienna, Austria, 2025, pp. 6634-6663: Association for Computational Linguistics.
47. Y. Ko, S. Li, C.-H. H. Yang, and T. Kawahara, "Benchmarking Japanese Speech Recognition on ASR-LLM Setups with Multi-Pass Augmented Generative Error Correction," ed, 2024.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
