Deep Learning Strategies For Fileless Mal Ware Detection: A Critical Review Of Detection Models, Feature Constraints, And Learning-Based Enhancements

Dr. Sonali kothari; Vaishali Kumar

doi:10.29284/sbgx4294

Authors

Dr. Sonali kothari Department of Computer Science and Engineering, Symbiosis Institute of Technology, Symbiosis International (Deemed University), Pune, Maharashtra, India
Vaishali Kumar Department of Computer Science and Engineering, Symbiosis Institute of Technology, Symbiosis International (Deemed University), Pune, Maharashtra, India

DOI:

https://doi.org/10.29284/sbgx4294

Keywords:

Fileless malware detection; code injection techniques; deep learning architectures; memory forensics; feature optimization; process hollowing; reflective DLL injection; behavioral analysis; adversarial resilience; cybersecurity.

Abstract

Leveraging a combination of genuine system processes, temporary memory and advanced code injection techniques to evade common security practices, fileless malware has evolved into one of the most persistent and evasive forms of modern day cyber attacks. Researchers are leaning towards deep learning methods capable of understanding complex behavioral patterns and elements related to memory as attackers morph their tactics into using memory code execution, concealed scripts or injection techniques used in the wild. By comparing two classes of analysis features, which are due to transient memory structures and call patterns from the API, on its impact on learning-based detection success, this survey consolidates inconsistent findings across different parts of publications. All aspects of scalability, feature extensibility, clarity and robustness to adversarial attacks will be considered in our comparison among the existing detection frameworks. Injection strategies that include, among others, process hollowing, reflective DLL loading and persistence through registry are considered and their implications on the design of feature extraction systems are highlighted. This study synthesizes findings from empirical studies, datasets, and design trends to identify promising areas for future research (e.g., light neural encoders; memory-semantic based feature selection; hybrid behavior–forensics models) as well as major gaps that render the detection unreliable. The purpose of this survey is to supply researchers and practitioners with a comprehensive guide for creating more resilient, accurate, effective deep learning-based defense mechanisms against fileless malware detection and code injection attacks.

References

1. Aghakhani H, Gritti F, Mecca F, Lindorfer M, Ortolani S, Balzarotti D, Vigna G, Kruegel C (2020) When malware is packin’ heat; limits of machine learning classifiers based on static analysis features. In: Proceedings of Symposium on Network and Distributed System Security (NDSS). The Internet Society

2. Ahmed F, Hameed H, ZubairShafiq M, Farooq M (2009). Using spatio-temporal information in API calls with machine learning algorithms for malware detection. In: Proceedings of the 2nd ACM Workshop on Security and Artificial Intelligence, AISec ’09, pp 55–62

3. Bagher BM, Mahdi A, Asghar T (2019) HLMD: a signature-based approach to hardware-level behavioral malware detection and classification. J. Supercomput. 75(8):5551–5582 Article Google Scholar

4. Canzanese R, Mancoridis S, Kam M (2015) System call-based detection of malicious processes. In: 2015 IEEE International Conference on Software Quality, Reliability and Security, pp 119–124

5. Chen Xiaohui, HaoZhiyu, Li Lun, Cui Lei, Zhu Yiran, Ding Zhenquan, Liu Yongji (2022) Cruparamer: learning on parameter-augmented API sequences for malware detection. IEEE Tran Inf Forensics Secur 17:788–803 Article Google Scholar

6. Chen X, Tong Y, Du C, Liu Y, Ding Z, Ran Q, Zhang Y, Cui L, Hao Z (2022) Malpro: Learning on process-aware behaviors for malware detection. In: 2022 IEEE Symposium on Computers and Communications (ISCC), pp 01–07

7. Christodorescu M, Jha S, Seshia SA, Song D, Bryant RE (2005) Semantics-aware malware detection. In: 2005 IEEE Symposium on Security and Privacy (S P’05), pp 32–46

8. Darem A, Abawajy J, Makkar A, Alhashmi A, Alanazi S (2021) Visualization and deep-learning-based malware variant detection using opcode-level features. FutGenerComputSyst 125:314–323 Article Google Scholar

9. Han W, Xue J, Wang Y, Huang L, Kong Z, Limin M (2019) Maldae: detecting and explaining malware based on correlation and fusion of static and dynamic characteristics. ComputSecur 83:208–233 Article Google Scholar

10. Han W, Xue J, Wang Y, Liu Z, Kong Z (2019) Malinsight: a systematic profiling based malware detection framework. J NetwComputAppl 125:236–250 Article Google Scholar

11. Jindal C, Salls C, Aghakhani H, Long K, Kruegel C, Vigna G (2019) Neurlux: dynamic malware analysis without feature engineering. In: Proceedings of the 35th Annual Computer Security Applications Conference, ACSAC ’19, pp 444–455

12. Kalash M, Rochan M, Mohammed N, Bruce ND, Wang Y, Iqbal F (2018) Malware classification with deep convolutional neural networks. In: 2018 9th IFIP international conference on new technologies, mobility and security (NTMS), pp 1–5

13. Kolosnjaji B, Zarras A, Webster G, Eckert C (2016) Deep learning for classification of malware system call sequences. In: Kang BH, Bai Q, editors, AI 2016: Advances in Artificial Intelligence, pp 137–149

14. Kong D, Yan G (2013) Discriminant malware distance learning on structural information for automated malware classification. In: Proceedings of the 19th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 1357–1365

15. Lanzi A, Balzarotti D, Kruegel C, Christodorescu M, Kirda E (2010) Accessminer: Using system-centric models for malware protection. In: Proceedings of the 17th ACM Conference on Computer and Communications Security, CCS ’10, pp 399–412

16. Li C, Cheng Z, Zhu H, Wang L, Lv Q, Wang Y, Li N, Sun D (2022) DMalNet: dynamic malware analysis based on API feature engineering and graph learning. ComputSecur 122:102872 Article Google Scholar

17. Li D, Li Q (2020) Adversarial deep ensemble: evasion attacks and defenses for malware detection. IEEE Trans Inf Forensics Secur 15:3886–3900 Article Google Scholar

18. Lu R (2019) Malware detection with LSTM using opcode language. arxiv:1906.04593

19. Min D, Park D, Ahn J, Walker R, Lee J, Park S, Kim Y (2018) Amoeba: an autonomous backup and recovery SSD for ransomware attack defense. IEEE Comput Architect Lett 17(2):245–248 Article Google Scholar

20. Moskovitch R, Feher C, Tzachar N, Berger E, Gitelman M, Dolev S, Elovici Y (2008) Unknown malcode detection using opcode representation. In: Intelligence and Security Informatics, pp 204–215

21. Naeem H, Ullah F, Naeem MR, Khalid S, Vasan D, Jabbar S, Saeed S (2020) Malware detection in industrial internet of things based on hybrid image visualization and deep learning model. Ad Hoc Netw 105:102154 Article Google

22. Scholar Narouei M, Ahmadi M, Giacinto G, Takabi H, Sami A (2015) DLLMiner: structural mining for malware detection. SecurCommunNetw 8(18):3311–3322 Article Google Scholar

23. Qiang W, Yang L, Jin H (2022) Efficient and robust malware detection based on control flow traces using deep neural networks. ComputSecur 122:102871 Article Google Scholar

24. Sayadi H, Patel N, Sasan A, Rafatirad S, Homayoun H (2018) Ensemble learning for effective run-time hardware-based malware detection: a comprehensive analysis and classification. In: 2018 55th ACM/ESDA/IEEE Design Automation Conference (DAC), pp 1–6

25. Sebastio S, Baranov E, Biondi F, Decourbe O, Given-Wilson T, Legay A, Puodzius C, Quilbeuf J (2020) Optimizing symbolic execution for malware behavior classification. ComputSecur 93:101775 Article Google Scholar SL SD,

26. Sun G, Qian Q (2018) Deep learning and visualization for identifying malware families. In: IEEE Transactions on Dependable and Secure Computing

27. Tang A, Sethumadhavan S, Stolfo SJ (2014) Unsupervised anomaly-based malware detection using hardware features. In: Stavrou A, Bos H, Portokalidis G (eds) Research in attacks, intrusions and defenses. Springer International Publishing, Cham, pp 109–129 Chapter Google Scholar

28. Tang F, Ma B, Li J, Zhang F, Su J, Ma J (2020) Ransomspector: an introspection-based approach to detect crypto ransomware. ComputSecur 97:101997 Article Google Scholar

29. Tian D, Ying Q, Jia X, Ma R, Hu C, Liu W (2021) MDCHD: a novel malware detection method in cloud using hardware trace and deep learning. ComputNetw 198:108394 Article Google Scholar

30. Xiaofeng L, Fangshuo J, Xiao Z, Shengwei Y, Jing S, Lio P (2019) ASSCA: API sequence and statistics features combined architecture for malware detection. ComputNetw 157:99–111 Article Google Scholar

31. Alrawi, O., Ike, M., Pruett, M., Kasturi, R.P., Barua, S., Hirani, T., Hill, B., Saltaformaggio, B.: Forecasting malware capabilities from cyber attack memory images. In: Proceedings of the USENIX Security Symposium (2021)

32. Barabosch, T., Bergmann, N., Dombeck, A., Padilla, E.: Quincy: Detecting host based code injection attacks in memory dumps. In: Proceedings of the Conference on Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA) (2017)

33. Barabosch, T., Eschweiler, S., Gerhards-Padilla, E.: Bee Master: Detecting Host Based Code Injection Attacks. In: Proceedings of the Conference on Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA) (2014)

34. . Barabosch, T., Gerhards-Padilla, E.: Host-based code injection attacks: A popular technique used by malware. In: Proceedings of the International Conference on Malicious and Unwanted Software (MALWARE) (2014)

35. Bayer, U., Moser, A., Kruegel, C., Kirda, E.: Dynamic analysis of malicious code. Journal in Computer Virology (2006)

36. Biondi, F., Given-Wilson, T., Legay, A., Puodzius, C., Quilbeuf, J.: Tutorial: An overview of malware detection and evasion techniques. In: Proceedings of the In ternational Symposium on Leveraging Applications of Formal Methods (2018)

37. Borello, J.M., M´e, L.: Code Obfuscation Techniques for Metamorphic Viruses. Journal in Computer Virology (2008)

38. Botacin, M., de Geus, P.L., Gr´egio, A.: “VANILLA” Malware: Vanishing An tiviruses by Interleaving Layers and Layers of Attacks. Journal of Computer Vi rology and Hacking Techniques 15(4), 233–247 (Dec 2019)

39. Christodorescu, M., Jha, S., Kruegel, C.: Mining Specifications of Malicious Be havior. In: Proceedings of the Joint Meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on The Foundations of Software Engineering. Association for Computing Machinery (2007)

40. Galloro, N., Polino, M., Carminati, M., Continella, A., Zanero, S.: A systematical and longitudinal study of evasive behaviors in windows malware. Computers & Security (November 2021)

41. Griffin, K., Schneider, S., Hu, X., Chiueh, T.c.: Automatic generation of string signatures for malware detection. In: Proceedings of the International Workshop on Recent Advances in Intrusion Detection (RAID) (2009)

42. Kharaz, A., Arshad, S., Mulliner, C., Robertson, W., Kirda, E.: UNVEIL: A Large-Scale, automated approach to detecting ransomware. In: Proceedings of the USENIX Security Symposium (2016)

43. Korczynski, D., Yin, H.: Capturing Malware Propagations with Code Injections and Code-Reuse Attacks. In: Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS) (2017)

44. Gu, G., Liu, J.C.: Shadow Attacks: Automatically Evading System-Call-Behavior based Malware Detection. Journal in Computer Virology (2012)

45. Mantovani, A., Aonzo, S., Ugarte-Pedrero, X., Merlo, A., Balzarotti, D.: Prevalence and impact of low-entropy packing schemes in the malware ecosystem. In: Proceedings of the ISOC Network and Distributed System Security Symposium (NDSS) (2020)

46. Martignoni, L., Stinson, E., Fredrikson, M., Jha, S., Mitchell, J.C.: A Layered Architecture for Detecting Malicious Behaviors. In: Proceedings of the International Symposium on Recent Advances in Intrusion Detection (RAID) (2008)

47. ., Patnaik, M., Rebeiro, C.: D-TIME: Distributed threadlessindepen dent malware execution for runtime obfuscation. In: Proceedings of the USENIX Workshop on Offensive Technologies (WOOT 19) (2019)

48. Quarta, D., Salvioni, F., Continella, A., Zanero, S.: Toward Systematically Exploring Antivirus Engines. In: Proceedings of the Conference on Detection of Intrusions and Malware and Vulnerability Assessment (DIMVA) (2018)

49. Rossow, C., Dietrich, C.J., Grier, C., Kreibich, C., Paxson, V., Pohlmann, N., Bos, H., Van Steen, M.: Prudent practices for designing malware experiments: Status quo and outlook. In: Proceedings of the IEEE Symposium on Security & Privacy (S&P) (2012)

50. Wang, Q., Hassan, W.U., Li, D., Jee, K., Yu, X., Zou, K., Rhee, J., Chen, Z., Cheng, W., Gunter, C.A., et al.: You Are What You Do: Hunting Stealthy Malware via Data Provenance Analysis. In: Proceedings of the ISOC Network and Distributed System Security Symposium (NDSS) (2020)

51. Zhu, S., Shi, J., Yang, L., Qin, B., Zhang, Z., Song, L., Wang, G.: Measuring and Modeling the Label Dynamics of Online Anti-Malware Engines. In: Proceedings of the USENIX Security Symposium (2020)

52. Omar Alrawi, Moses Ike, Matthew Pruett, RanjitaPaiKasturi, SrimantaBarua, TalebHirani, Brennan Hill, and Bren danSaltaformaggio. 2021. Forecasting malware capabilities from cyber attack memory images. In Proceedings of the USENIX Security Symposium.

53. Thomas Barabosch, Niklas Bergmann, Adrian Dombeck, and Elmar Padilla. 2017. Quincy: Detecting host-based code injection attacks in memory dumps. In Proceedings of the Conference on Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA’17).

54. Thomas Barabosch, Sebastian Eschweiler, and ElmarGerhards-Padilla. 2014. Bee master: Detecting host-based code injection attacks. In Proceedings of the Conference on Detection of Intrusions and Malware, and Vulnerability Assessment (DIMVA’14).

55. Thomas Barabosch and ElmarGerhards-Padilla. 2014. Host-based code injection attacks: A popular technique used by malware. In Proceedings of the International Conference on Malicious and Unwanted Software (MALWARE’14).

56. Marcus Botacin, Paulo Lí–cio de Geus, and André Grégio. 2019. “VANILLA” malware: Vanishing antiviruses by in terleaving layers and layers of attacks. Journal of Computer Virology and Hacking Techniques 15, 4 (01 Dec. 2019), 233–247.

57. Min Du, Wenjun Hu, and William Hewlett. 2021. AutoCombo: Automatic malware signature generation through combination rule mining. In Proceedings of the ACM International Conference on Information & Knowledge Management.

58. Clemens Kolbitsch, Paolo MilaniComparetti, Christopher Kruegel, EnginKirda, Xiao-yong Zhou, Xiao Feng Wang, et al. 2009. Effective and efficient malware detection at the end host.. In USENIX Security Symposium,Vol.4.

59. David Korczynski and Heng Yin. 2017. Capturing malware propagations with code injections and code-reuse attacks. In Proceedings of the ACM SIGSAC Conference on Computer and Communications Security (CCS’17).

60. Alexander Küchler, Alessandro Mantovani, Yufei Han, Leyla Bilge, and DavideBalzarotti. 2021. Does every second count? Time-based evolution of malware behavior in sandboxes. In Proceedings of the ISOC Network and Distributed System Security Symposium (NDSS’21).

61. Tamas K. Lengyel, Steve Maresca, Bryan D. Payne, George D. Webster, Sebastian Vogl, and AggelosKiayias. 2014. Scalability, fidelity and stealth in the DRAKVUF dynamic malware analysis system. In Proceedings of the Annual Computer Security Applications Conference (ACSAC’14).

62. Lorenzo Martignoni, Elizabeth Stinson, Matt Fredrikson, SomeshJha, and John C. Mitchell. 2008. A layered architecture for detecting malicious behaviors. In Proceedings of the International Symposium on Recent Advances in Intrusion Detection (RAID’08).

63. Mohammad N. Olaimat, MohdAizainiMaarof, and Bander Ali S. Al-Rimy. 2021. Ransomware anti-analysis and evasion techniques: A survey and research directions. In Proceedings of the International Cyber Resilience Conference (CRC’21).

64. JithinPavithran, Milan Patnaik, and Chester Rebeiro. 2019. D-TIME: Distributed threadless independent malware execution for runtime obfuscation. In Proceedings of the USENIX Workshop on Offensive Technologies (WOOT’19).

65. CERT Polska. 2012. More Human than Human- Flame’s Code Injection Techniques. https://cert.pl/en/posts/2012/08/ more-human-than-human-flames-code-injection-techniques