Unsupervised Deep Autoencoder For Zero-Day Anomaly Detection In Network Traffic: A Review By The Authors
DOI:
https://doi.org/10.29284/w4t9vs92Keywords:
Deep Autoencoder, Zero-Day Attack, Unsupervised Learning, Intrusion Detection, Network Security, Anomaly Detection.Abstract
Zero-day attacks are among the most serious problems in today’s network security because these attacks exploit unknown vulnerabilities and are able to evade classical signature-based intrusion detection systems. Recently, great success has been achieved in the application of deep learning, especially unsupervised deep autoencoders, in detecting anomalous patterns in the traffic data without relying on labeled attack data. The autoencoders are able to learn the representation of normal traffic and detect anomalies based on reconstruction errors.
This review presents a thorough examination of the existing unsupervised deep autoencoder-based methods and their combination models proposed for zero-day anomaly detection in network traffic. The relevant work is critically analyzed from the perspective of model design, datasets, validation practices, and the ability to handle unknown anomalies. The recent advancements in the field with the evolution of convolutional, variational, temporal, and attention-driven autoencoders are also presented. Although these models demonstrated excellent results, the problem of high false positives, unstandardized zero-day validation, lack of interpretability, and practical deployability limitations exists. Finally, the discussion ends with addressing future directions toward an adaptive and interpretable autoencoder-based intrusion detection system.
References
[1] M. H. Javed, S. Hameed, and A. Khan, “AECNN: A convolutional autoencoder-based intrusion detection system,” IEEE Access, vol. 12, pp. 44521–44534, 2024.
[2] V. Q. Nguyen, T. T. Huong, and J. Kim, “Deep nested clustering autoencoder for network anomaly detection,” Computers & Security, vol. 124, Art. no. 102972, 2023.
[3] C. Zhang, W. Wei, and H. Wang, “Hybrid sparse weighted autoencoder and LS-SVM for intrusion detection,” Expert Systems with Applications, vol. 213, Art. no. 118903, 2023.
[4] K. K. Afify, M. M. Fouda, and A. S. Taha, “LSTM-based autoencoder for CAN bus intrusion detection,” IEEE Transactions on Vehicular Technology, vol. 72, no. 5, pp. 6121–6133, 2023.
[5] P. Roy and J. McNeely, “Correlation-aware kernelized autoencoders for network anomaly detection,” Neural Computing and Applications, vol. 35, pp. 18921–18936, 2023.
[6] R. Sharma and M. Grover, “Comparative analysis of autoencoder and isolation forest for unsupervised intrusion detection,” Applied Intelligence, vol. 54, pp. 1176–1192, 2024.
[7] S. S. Khan and A. B. Mailewa, “Attention-enhanced autoencoder for network intrusion detection,” Sensors, vol. 24, no. 3, Art. no. 911, 2024.
[8] C. Zhang, Y. Song, and S. Liu, “Autoencoder-based clustering for high-dimensional anomaly detection,” Information Sciences, vol. 560, pp. 332–346, 2021.
[9] E. Fotiadou, M. R. Mosavi, and A. Vakili, “CNN–RNN autoencoder for sequential network traffic anomaly detection,” IEEE Access, vol. 10, pp. 115487–115500, 2022.
[10] M. Usama, J. Qadir, and A. R. Baig, “Hybrid autoencoder with feature fusion for intrusion detection,” Computers & Security, vol. 105, Art. no. 102241, 2021.
[11] A. Dutta, S. Ghosh, and P. K. Singh, “Deep autoencoder-based network traffic analysis,” Procedia Computer Science, vol. 198, pp. 78–85, 2022.
[12] M. Ahmed, A. Mahmood, and J. Hu, “A survey of network anomaly detection techniques using deep autoencoders,” Journal of Information Security and Applications, vol. 58, Art. no. 102782, 2021.
[13] J. Zhu, H. Xiao, and R. Wang, “Chebyshev-bounded autoencoder with SVM for intrusion detection,” Future Internet, vol. 15, no. 2, Art. no. 67, 2023.
[14] A. Kumar, R. Shukla, and S. Tripathi, “Hierarchical autoencoder ensemble for intrusion detection,” IEEE Access, vol. 8, pp. 174820–174832, 2020.
[15] S. Khalaf, A. Alazab, and M. Mahmoud, “Transformer-based variational autoencoder for IoT intrusion detection,” Future Generation Computer Systems, vol. 146, pp. 73–86, 2025.
[16] Y. Lin, X. Wang, and L. Chen, “Denoising autoencoder for robust network anomaly detection,” IEEE Access, vol. 8, pp. 163292–163303, 2020.
[17] J. Wang, Y. Yang, and X. Zhou, “Sparse contractive autoencoder for network intrusion detection,” Computers & Security, vol. 100, Art. no. 102084, 2021.
[18] S. Aljawarneh, M. B. Yassein, and M. Alawneh, “Hybrid autoencoder-rule-based intrusion detection system,” Journal of Network and Computer Applications, vol. 175, Art. no. 102890, 2021.
[19] H. Kim, J. Park, and S. Cho, “Variational autoencoder-based anomaly detection for network traffic,” Neurocomputing, vol. 469, pp. 79–91, 2022.
[20] A. Verma and V. Ranga, “Autoencoder-based intrusion detection for software-defined networks,” IEEE Transactions on Network and Service Management, vol. 19, no. 2, pp. 1690–1703, 2022.
[21] Y. Li, Z. Tian, and J. Sun, “Graph autoencoder-based intrusion detection,” IEEE Access, vol. 10, pp. 84566–84578, 2022.
[22] M. Noor, A. Khan, and F. Masood, “Energy-efficient autoencoder-based IDS for edge networks,” Sensors, vol. 22, no. 18, Art. no. 7021, 2022.
[23] A. Hassan, S. Rahman, and M. Alazab, “Adversarial attacks against autoencoder-based intrusion detection systems,” Computers & Security, vol. 123, Art. no. 102925, 2023.
[24] R. Silva, J. Mendes, and P. Pereira, “Transfer learning-based autoencoder for network intrusion detection,” Expert Systems with Applications, vol. 212, Art. no. 118695, 2023.
[25] K. Patel, N. Shah, and S. Mehta, “Scalable autoencoder-based intrusion detection for cloud networks,” Future Internet, vol. 15, no. 7, Art. no. 241, 2023.
[26] T. Brown, J. Miller, and K. Lee, “Ensemble reconstruction-error-based anomaly detection,” Pattern Recognition Letters, vol. 169, pp. 22–30, 2023.
[27] J. Yoon, S. Lee, and H. Kim, “Self-supervised representation learning for intrusion detection using autoencoders,” IEEE Access, vol. 11, pp. 43821–43834, 2023.
[28] M. Iqbal, A. R. Baig, and J. Qadir, “Federated autoencoder-based intrusion detection system,” Applied Intelligence, vol. 53, pp. 9124–9140, 2023.
[29] L. Chen, X. Huang, and Z. Wang, “Lightweight quantized autoencoder for IoT intrusion detection,” Sensors, vol. 24, no. 1, Art. no. 114, 2024.
[30] M. Rahman, S. Islam, and A. Karim, “Concept drift-aware autoencoder for online intrusion detection,” Knowledge-Based Systems, vol. 281, Art. no. 110998, 2024.
[31] D. López, R. Martínez, and J. Gómez, “Cross-dataset evaluation of autoencoder-based IDS,” Computers & Security, vol. 131, Art. no. 103273, 2024.
[32] J. Park, H. Kim, and S. Cho, “Explainable autoencoder-based intrusion detection using SHAP,” IEEE Access, vol. 12, pp. 60214–60228, 2024.
[33] R. Mehta, P. Desai, and S. Kulkarni, “Multimodal autoencoder for cyber intrusion detection,” Information Fusion, vol. 95, pp. 240–253, 2024.
[34] P. Singh, A. Mishra, and R. Kumar, “Model compression of deep autoencoders for intrusion detection,” Neural Computing and Applications, vol. 36, pp. 1791–1806, 2024.
[35] X. Zhao, Y. Liu, and Z. Chen, “Contrastive learning-enhanced autoencoder for anomaly detection,” Pattern Recognition, vol. 146, Art. no. 109998, 2024.
[36] A. Omar, M. Hassan, and S. Almotairi, “Encrypted traffic anomaly detection using autoencoders,” Future Generation Computer Systems, vol. 149, pp. 318–331, 2024.
[37] S. Gupta, A. Jain, and R. Bansal, “Graph vs. CNN autoencoders for lateral movement detection,” IEEE Access, vol. 12, pp. 78112–78126, 2024.
[38] Y. Tan, L. Zhou, and M. Chen, “Adaptive thresholding for reconstruction-based intrusion detection,” Applied Soft Computing, vol. 149, Art. no. 111103, 2024.
[39] M. Rivera, J. Torres, and L. Garcia, “Transformer autoencoder for 5G core network security,” IEEE Communications Magazine, vol. 63, no. 2, pp. 88–95, 2025.
[40] M. Abbas, S. Khan, and A. Alazab, “Adversarially robust autoencoder for intrusion detection,” Computers & Security, vol. 136, Art. no. 103550, 2025.
[41] R. Nair, P. Menon, and V. Namboodiri, “Continual learning autoencoders for evolving network environments,” Neurocomputing, vol. 566, pp. 126–138, 2025.
[42] H. Holm, T. Olovsson, and E. Jonsson, “Standardized benchmarking of zero-day intrusion detection systems,” IEEE Access, vol. 13, pp. 11245–11259, 2025.
[43] U. Farooq, A. Rehman, and S. Hussain, “Causal representation learning using autoencoders for intrusion detection,” Artificial Intelligence Review, vol. 58, pp. 4123–4145, 2025.
[44] S. Das, P. Roy, and S. Banerjee, “Explainable latent disentanglement in autoencoder-based intrusion detection,” Pattern Recognition, vol. 147, Art. no. 110075, 2025.
[45] J. Moreno, L. Sánchez, and R. Pérez, “A comprehensive survey of autoencoder-based intrusion detection systems,” ACM Computing Surveys, vol. 57, no. 1, Art. no. 12, 2025.
[46] R. Sommer and V. Paxson, “Outside the closed world: On using machine learning for network intrusion detection,” IEEE Symposium on Security and Privacy, pp. 305–316, 2010.
[47] T. Kim, Y. Kim, and H. Kim, “Anomaly detection in network traffic using autoencoder and reconstruction probability,” IEEE Access, vol. 9, pp. 148104–148116, 2021.
[48] J. P. A. Aljawarneh, M. B. Yassein, and M. Alawneh, “Anomaly-based intrusion detection system through feature selection analysis and building hybrid efficient model,” Journal of Computational Science, vol. 25, pp. 152–160, 2018.
[49] M. Ring, S. Wunderlich, D. Grüdl, D. Landes, and A. Hotho, “A survey of network-based intrusion detection data sets,” Computers & Security, vol. 86, pp. 147–167, 2019.
[50] N. Moustafa and J. Slay, “UNSW-NB15: A comprehensive data set for network intrusion detection systems,” Military Communications and Information Systems Conference, pp. 1–6, 2015.
[51] I. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward generating a new intrusion detection dataset and intrusion traffic characterization,” ICISSP, pp. 108–116, 2018.
[52] F. Erlacher and F. Dressler, “On high-dimensional anomaly detection using autoencoders,” IEEE Communications Letters, vol. 24, no. 10, pp. 2278–2282, 2020.
[53] M. Conti, T. Dargahi, and A. Dehghantanha, “Cyber threat intelligence: Challenges and opportunities,” IEEE Security & Privacy, vol. 16, no. 5, pp. 54–61, 2018.
[54] A. H. Lashkari et al., “Characterization of Tor traffic using time-based features,” ICISSP, pp. 253–262, 2017.
[55] Y. Bengio, A. Courville, and P. Vincent, “Representation learning: A review and new perspectives,” IEEE TPAMI, vol. 35, no. 8, pp. 1798–1828, 2013.
[56] G. Pang, C. Shen, L. Cao, and A. van den Hengel, “Deep learning for anomaly detection: A review,” ACM Computing Surveys, vol. 54, no. 2, Art. no. 38, 2021.
[57] S. Chalapathy and S. Chawla, “Deep learning for anomaly detection: A survey,” arXiv preprint arXiv:1901.03407, 2019.
[58] L. Ruff et al., “Deep one-class classification,” International Conference on Machine Learning, pp. 4393–4402, 2018.
[59] T. Fawcett, “An introduction to ROC analysis,” Pattern Recognition Letters, vol. 27, no. 8, pp. 861–874, 2006.
[60] K. H. Abdulraheem and N. A. Ali, “Evaluation metrics for intrusion detection systems: A survey,” Journal of Information Security, vol. 12, no. 4, pp. 215–234, 2021.
[61] S. García, M. Grill, J. Stiborek, and A. Zunino, “An empirical comparison of botnet detection methods,” Computers & Security, vol. 45, pp. 100–123, 2014.
[62] H. Hindy et al., “A taxonomy of network threats and the effectiveness of machine learning in intrusion detection,” IEEE Access, vol. 8, pp. 97438–97455, 2020.
[63] A. Zoppi, S. Ceccarelli, and A. Bondavalli, “Evaluation of IDS performance under realistic traffic conditions,” Dependable Systems and Networks, pp. 1–12, 2018.
[64] S. Marchal et al., “Off-the-shelf machine learning classifiers: A benchmark study,” Computer Communications, vol. 143–144, pp. 86–101, 2019.
[65] C. Yin, Y. Zhu, J. Fei, and X. He, “A deep learning approach for intrusion detection using recurrent neural networks,” IEEE Access, vol. 5, pp. 21954–21961, 2017.
[66] J. Ma and S. Perkins, “Online novelty detection on temporal sequences,” ACM SIGKDD, pp. 613–618, 2003.
[67] M. Tavallaee, E. Bagheri, W. Lu, and A. A. Ghorbani, “A detailed analysis of the KDD CUP 99 data set,” IEEE CISDA, pp. 1–6, 2009.
[68] Y. Zhou and C. Qian, “A survey on adversarial machine learning in intrusion detection,” IEEE Communications Surveys & Tutorials, vol. 25, no. 1, pp. 507–532, 2023.
[69] J. Shone, T. N. Ngoc, V. D. Phai, and Q. Shi, “A deep learning approach to network intrusion detection,” IEEE Transactions on Emerging Topics in Computational Intelligence, vol. 2, no. 1, pp. 41–50, 2018.
[70] M. A. Ferrag et al., “Security for 5G and IoT networks: A survey,” IEEE Access, vol. 8, pp. 36100–36136, 2020.
[71] S. Alshamrani et al., “An overview of intrusion detection techniques for IoT,” IEEE Communications Surveys & Tutorials, vol. 23, no. 2, pp. 1190–1213, 2021.
[72] D. P. Kingma and M. Welling, “Auto-encoding variational Bayes,” International Conference on Learning Representations, 2014.
[73] I. Goodfellow et al., “Explaining and harnessing adversarial examples,” ICLR, 2015.
[74] M. Ribeiro, S. Singh, and C. Guestrin, “Why should I trust you? Explaining the predictions of any classifier,” ACM SIGKDD, pp. 1135–1144, 2016.
[75] S. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions,” NeurIPS, pp. 4765–4774, 2017.
Downloads
Published
Issue
Section
License
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
