Enhanced Speaker Embeddings With Adaptive Feature Aggregation For Robust Speaker Diarization
DOI:
https://doi.org/10.29284/0etf5d47Keywords:
Speaker Diarization, AFA-TDNN, BADA, DER, AMI Meeting Corpus.Abstract
Speaker diarization, is the process of identifying "who spoke when" in recordings of multiple speakers, is central to applications like meeting transcription, conversational AI, and speech analytics. Yet real-world diarization is hindered by overlapping speech, background noise, and channel variability that corrupt the performance of standard embedding-based systems. To tackle these challenges, they introduce an Enhanced Speaker Embedding approach based on an Adaptive Feature Aggregation Time-Delay Neural Network (AFA-TDNN) for speaker diarization that is robust. The method innovatively adds dynamic feature weighting, multi-scale temporal aggregation, and residual feature fusion to acquire discriminative embeddings. A Batch-Adaptive Data Augmentation (BADA) strategy, which includes spectral masking, noise injection, temporal perturbation, and reverberation simulation, is utilized to improve robustness in varying acoustic conditions. The embeddings are then aggregated with spectral clustering to obtain speaker segmentation with high accuracy even from noisy and overlapped conditions. Experimental evaluation on the AMI Meeting Corpus shows significant gain, with a Diarization Error Rate (DER) of 2.23% and a Jaccard Error Rate (JER) of 5.9%, which beats current baselines. The overall framework guarantees strong, noise-tolerant, and highly accurate speaker diarization and is applicable for real-world multi-speaker applications.
References
1 O’Shaughnessy, Douglas. "Speaker Diarization: A Review of Objectives and Methods." Applied Sciences 15, no. 4 (2025): 2002.
2 Wooters, Chuck, and MarijnHuijbregts. "The ICSI RT07s speaker diarization system." In International Evaluation Workshop on Rich Transcription, pp. 509-519. Berlin, Heidelberg: Springer Berlin Heidelberg, 2007.
3 Mao, Huanru Henry, Shuyang Li, Julian McAuley, and Garrison Cottrell. "Speech recognition and multi-speaker diarization of long conversations." arXiv preprint arXiv:2005.08072 (2020).
4 Anguera, Xavier, Simon Bozonnet, Nicholas Evans, Corinne Fredouille, Gerald Friedland, and OriolVinyals. "Speaker diarization: A review of recent research." IEEE Transactions on audio, speech, and language processing 20, no. 2 (2012): 356-370.
5 Stan, Adriana. "Residual information in deep speaker embedding architectures." Mathematics 10, no. 21 (2022): 3927.
6 Irum, Amna, and Ahmad Salman. "Speaker verification using deep neural networks: A." International Journal of Machine Learning and Computing 9, no. 1 (2019).
7 S. Horiguchi, Y. Takashima, P. García, S. Watanabe, and Y. Kawaguchi, “Multi-Channel End-To-End Neural Diarization with Distributed Microphones,” in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 7332–7336. doi: 10.1109/ICASSP43922.2022.9746749.
8 N. Zheng et al., “Multi-Channel Speaker Diarization Using Spatial Features for Meetings,” in ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 7337–7341. doi: 10.1109/ICASSP43922.2022.9747343.
9 Z. Chen, B. Han, S. Wang, and Y. Qian, “Attention-based encoder-decoder end-to-end neural diarization with embedding enhancer,” IEEEACM Trans. Audio Speech Lang. Process., vol. 32, pp. 1636–1649, 2024.
10 X. Anguera, S. Bozonnet, N. Evans, C. Fredouille, G. Friedland, and O. Vinyals, “Speaker diarization: A review of recent research,” IEEE TASLP, 2012.
11 M. Rybicka, J. Villalba, T. Thebaud, N. Dehak, and K. Kowalczyk, “End-to-End Neural Speaker Diarization With Non-Autoregressive Attractors,” IEEEACM Trans. Audio Speech Lang. Process., vol. 32, pp. 3960–3973, 2024, doi: 10.1109/TASLP.2024.3439993.
12 F. Landini, “From Modular to End-to-End Speaker Diarization,” ArXivPrepr. ArXiv240708752, 2024.
13 Serafini, Luca, Samuele Cornell, Giovanni Morrone, Enrico Zovato, AlessioBrutti, and Stefano Squartini. "An experimental review of speaker diarization methods with application to two-speaker conversational telephone speech recordings." Computer speech & language 82 (2023): 101534.
14 Ancilin, J., and A. Milton. "Improved speech emotion recognition with Mel frequency magnitude coefficient." Applied Acoustics 179 (2021): 108046.
15 Pudasaini, Anil, Muna Al-Hawawreh, Mohamed RedaBouadjenek, Hakim Hacid, and Sunil Aryal. "A comprehensive study of audio profiling: Methods, applications, challenges, and future directions." Neurocomputing (2025): 130334.
16 Y. Lei, N. Scheffer, L. Ferrer, and M. McLaren, “A novel scheme for speaker recognition using a phonetically aware deep neural network,” in ICASSP 2014, 2014.
17 L. Li, D. Wang, Z. Zhang, and T. F. Zheng, “Deep Speaker Vectors for Semi Text-independent Speaker Verification,” ArXivPrepr. ArXiv150506427, 2015.
18 G. Sun, C. Zhang, and P. Woodland, “Combination of Deep Speaker Embeddings for Diarisation,” ArXivPrepr. ArXiv201012025, 2021.
19 Kinoshita, Keisuke, Marc Delcroix, and NaohiroTawara. "Integrating end-to-end neural and clustering-based diarization: Getting the best of both worlds." In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7198-7202. IEEE, 2021.
20 Kinoshita, Keisuke, Marc Delcroix, and NaohiroTawara. "Advances in integration of end-to-end neural and clustering-based diarization for real conversational speech." arXiv preprint arXiv:2105.09040 (2021).
21 Anidjar, OrHaim, YannickEsteve, Chen Hajaj, AmitDvir, and ItshakLapidot. "Speech and multilingual natural language framework for speaker change detection and diarization." Expert Systems with Applications 213 (2023): 119238.
22 Chauhan, Neha, Tsuyoshi Isshiki, and Dongju Li. "Enhancing speaker recognition models with noise-resilient feature optimization strategies." In Acoustics, vol. 6, no. 2, pp. 439-469. MDPI, 2024.
23 Kwon, Youngki, Jee-weon Jung, Hee-SooHeo, You Jin Kim, Bong-Jin Lee, and Joon Son Chung. "Adapting speaker embeddings for speaker diarisation." arXiv preprint arXiv:2104.02879 (2021).
24 B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,” in Proceedings of Interspeech, 2020, pp. 3830–3834. doi: 10.21437/Interspeech.2020-2650.
25 Gao, Zhifu, Yan Song, Ian McLoughlin, Pengcheng Li, Yiheng Jiang, and Li-Rong Dai. "Improving Aggregation and Loss Function for Better Embedding Learning in End-to-End Speaker Verification System." In Interspeech, pp. 361-365. 2019.
26 Tu, Youzhi. "Deep speaker embedding for robust speaker verification." (2022).
27 Johannessen, Runar Ask. "Aggregation of speaker embeddings for speaker diarization." Master's thesis, NTNU, 2020.
28 G. Sell, D. Garcia-Romero, D. Snyder, A. McCree, D. Povey, and S. Khudanpur, “Diarization is Hard: Some Experiences and Lessons Learned,” in Proceedings of Interspeech, 2018, pp. 2808–2812. [Online]. Available: https://www.isca-speech.org/archive/Interspeech_2018/pdfs/1576.pdf
29 Ryant, Neville, Prachi Singh, VenkatKrishnamohan, RajatVarma, Kenneth Church, Christopher Cieri, Jun Du, SriramGanapathy, and Mark Liberman. "The third DIHARD diarization challenge." arXiv preprint arXiv:2012.01477 (2020).
30 Baghel, Shikha, ShreyasRamoji, Prachi Singh, Somil Jain, Pratik Roy Chowdhuri, KaustubhKulkarni, SwapnilPadhi, DeepuVijayasenan, and SriramGanapathy. "Displace challenge: Diarization of speaker and language in conversational environments." arXiv preprint arXiv:2303.00830 (2023).
31 Brown, Andrew, Jaesung Huh, Joon Son Chung, ArshaNagrani, Daniel Garcia-Romero, and Andrew Zisserman. "Voxsrc 2021: The third voxceleb speaker recognition challenge." arXiv preprint arXiv:2201.04583 (2022).
32 Y. Lei, N. Scheffer, L. Ferrer, and M. McLaren, “A novel scheme for speaker recognition using a phonetically‑aware deep neural network,” in ICASSP 2014, 2014.
33 Li, Lantian, Dong Wang, Zhiyong Zhang, and Thomas Fang Zheng. "Deep speaker vectors for semi text-independent speaker verification." arXiv preprint arXiv:1505.06427 (2015).
34 Sun, Guangzhi, Chao Zhang, and Philip C. Woodland. "Combination of deep speaker embeddings for diarisation." Neural Networks 141 (2021): 372-384.
35 Carletta, Jean, Simone Ashby, SebastienBourban, Mike Flynn, Mael Guillemot, Thomas Hain, JaroslavKadlec et al. "The AMI meeting corpus: A pre-announcement." In International workshop on machine learning for multimodal interaction, pp. 28-39. Berlin, Heidelberg: Springer Berlin Heidelberg, 2005.
36 Kwon, Youngki, Jee-weon Jung, Hee-SooHeo, You Jin Kim, Bong-Jin Lee, and Joon Son Chung. "Adapting speaker embeddings for speaker diarisation." arXiv preprint arXiv:2104.02879 (2021).
37 Li, Xiang, VivekGovindan, RohitPaturi, and SundararajanSrinivasan. "Speakers unembedded: Embedding-free approach to long-form neural diarization." arXiv preprint arXiv:2406.18679 (2024).
38 Li, Yaqian, Xiaolong Zhang, Haibin Li, and Wengming Zhang. "Enhancing Speaker Diarization with Deep Neural Network Embeddings and Spectral Clustering." Available at SSRN 4761198.
39 He, Mao-Kui, Jun Du, Qing-Feng Liu, and Chin-Hui Lee. "Ansd-ma-mse: Adaptive neural speaker diarization using memory-aware multi-speaker embedding." IEEE/ACM Transactions on Audio, Speech, and Language Processing 31 (2023): 1561-1573.
40 Cord-Landwehr, Tobias, ChristophBoeddeker, CătălinZorilă, Rama Doddipatla, and Reinhold Haeb-Umbach. "Frame-wise and overlap-robust speaker embeddings for meeting diarization." In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
41 Alvarez-Trejos, Juan Ignacio, Alicia Lozano-Diez, and Daniel Ramos. "Feature Integration Strategies for Neural Speaker Diarization in Conversational Telephone Speech." Applied Sciences 15, no. 9 (2025): 4842.
42 AMI Project, “The AMI Meeting Corpus.” University of Edinburgh, IDIAP Research Institute, University of Sheffield, TNO, 2005. [Online]. Available: https://groups.inf.ed.ac.uk/ami/corpus/
43 X. Anguera, C. Wooters, and J. Hernando, “BeamformIt: An Open-source Toolkit for Robust Multi-microphone Speech Processing,” in Proceedings of Interspeech, 2006. [Online]. Available:
https://www.iscaspeech.org/archive/interspeech_2006/i06_2601.html
44 D. Snyder, G. Chen, and D. Povey, “MUSAN: A Music, Speech, and Noise Corpus.” 2015. [Online]. Available: https://arxiv.org/abs/1510.08484
45 T. Ko, V. Peddinti, D. Povey, and S. Khudanpur, “A Study on Data Augmentation of Reverberant Speech for Robust Speech Recognition,” in Proceedings of ICASSP, 2017, pp. 5220–5224. doi: 10.1109/ICASSP.2017.7953075.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
