MAnTra: A Transformer-Based Approach for Malware Anomaly Detection in Network Traffic Classification
DOI:
https://doi.org/10.52436/1.jutif.2025.6.6.5462Keywords:
Classification, Malware Anomaly, Malware Detection, Network Traffic, TransformerAbstract
Cybersecurity is a critical priority in the ever-evolving digital era, particularly with the emergence of increasingly sophisticated and difficult to detect malware. Traditional detection techniques, such as static and dynamic analysis, are often limited in their ability to recognize novel and concealed malware that poses a threat to security systems. Consequently, this study investigates the potential of Transformer models for network traffic classification to detect anomalies associated with malware activity. The proposed approach emphasizes retrospective analysis, wherein the model is evaluated across various platforms and datasets encompassing different virus variants. By incorporating diverse types of malwares into the training data, the model is better equipped to identify a range of attack patterns. The Transformer model employed in this study was trained over 30 epochs. The evaluation results demonstrated excellent performance, achieving a training accuracy of 99.16% and a test accuracy of 99.32%. The very low average loss value of 0.01 indicates that the model effectively reduces classification errors. These findings underscore the potential of Transformer models as an efficient method for malware detection, offering greater accuracy and speed compared to traditional approaches. The results further reveal that the Transformer exhibits strong capabilities in handling sequential data, which is highly relevant to the dynamic nature of network traffic. For future research, it is recommended to explore the scalability of this method in larger network environments and assess its effectiveness in real-time detection scenarios. Expanding its application could establish the Transformer model as a more reliable and efficient solution for identifying evolving malware threats, thereby enhancing overall network security. This approach presents a robust framework for protecting systems and data against increasingly complex cyber threats.
Downloads
References
M. F. Safitra, M. Lubis, and H. Fakhrurroja, “Counterattacking Cyber Threats: A Framework for the Future of Cybersecurity,” Sustainability, vol. 15, no. 18, p. 13369, Sep. 2023. DOI: 10.3390/su151813369
S. Poornima and R. Mahalakshmi, “Automated malware detection using machine learning and deep learning approaches for android applications,” Measurement: Sensors, vol. 32, p. 100955, Apr. 2024. DOI: 10.1016/j.measen.2023.100955
P. Maniriho, A. N. Mahmood, and M. J. M. Chowdhury, “MeMalDet: A memory analysis-based malware detection framework using deep autoencoders and stacked ensemble under temporal evaluations,” Computers & Security, vol. 142, p. 103864, Jul. 2024. DOI: 10.1016/j.cose.2024.103864
G. Aceto et al., “Synthetic and privacy-preserving traffic trace generation using generative AI models for training Network Intrusion Detection Systems,” Journal of Network and Computer Applications, vol. 229, p. 103926, Sep. 2024. DOI: 10.1016/j.jnca.2024.103926
M. Alshomrani, A. Albeshri, B. Alturki, F. S. Alallah, and A. A. Alsulami, “Survey of Transformer-Based Malicious Software Detection Systems,” Electronics, vol. 13, no. 23, p. 4677, Nov. 2024. DOI: 10.3390/electronics13234677
O. A. Madamidola, F. Ngobigha, and A. Ez-zizi, “Detecting new obfuscated malware variants: A lightweight and interpretable machine learning approach,” Intelligent Systems with Applications, vol. 25, p. 200472, Mar. 2025. DOI: 10.1016/j.iswa.2024.200472
S. Liu, P. Feng, S. Wang, K. Sun, and J. Cao, “Enhancing malware analysis sandboxes with emulated user behavior,” Computers & Security, vol. 115, p. 102613, Apr. 2022. DOI: 10.1016/j.cose.2022.102613
S. N. A. Sherazi and A. Qureshi, “Hybrid Analysis Model for Detecting Fileless Malware,” Electronics, vol. 14, no. 15, p. 3134, Aug. 2025. DOI: 10.3390/electronics14153134
S. Singh, D. Krishnan, V. Vazirani, V. Ravi, and S. A. Alsuhibany, “Deep hybrid approach with sequential feature extraction and classification for robust malware detection,” Egyptian Informatics Journal, vol. 27, p. 100539, Sep. 2024. DOI: 10.1016/j.eij.2024.100539
G. M. and S. C. Sethuraman, “A comprehensive survey on deep learning based malware detection techniques,” Computer Science Review, vol. 47, p. 100529, Feb. 2023. DOI: 10.1016/j.cosrev.2022.100529
J. Song et al., “A study of the relationship of malware detection mechanisms using Artificial Intelligence,” ICT Express, vol. 10, no. 3, pp. 632–649, Jun. 2024. DOI: 10.1016/j.icte.2024.03.005
A. Bensaoud, J. Kalita, and M. Bensaoud, “A survey of malware detection using deep learning,” Machine Learning with Applications, vol. 16, p. 100546, Jun. 2024. DOI: 10.1016/j.mlwa.2024.100546
G. Karat, J. M. Kannimoola, N. Nair, A. Vazhayil, S. V G, and P. Poornachandran, “CNN-LSTM Hybrid Model for Enhanced Malware Analysis and Detection,” Procedia Computer Science, vol. 233, pp. 492–503, 2024. DOI: 10.1016/j.procs.2024.03.239
G. Kale, G. E. Bostancı, and F. V. Çelebi, “Evolutionary feature selection for machine learning based malware classification,” Engineering Science and Technology, an International Journal, vol. 56, p. 101762, Aug. 2024. DOI: 10.1016/j.jestch.2024.101762
D. Zhan et al., “Enhancing reinforcement learning based adversarial malware generation to evade static detection,” Alexandria Engineering Journal, vol. 98, pp. 32–43, Jul. 2024. DOI: 10.1016/j.aej.2024.04.024
S. R. Choi and M. Lee, “Transformer Architecture and Attention Mechanisms in Genome Data Analysis: A Comprehensive Review,” Biology, vol. 12, no. 7, p. 1033, Jul. 2023. DOI: 10.3390/biology12071033
J. Liu, Y. Zhao, Y. Feng, Y. Hu, and X. Ma, “SeMalBERT: Semantic-based malware detection with bidirectional encoder representations from transformers,” Journal of Information Security and Applications, vol. 80, p. 103690, Feb. 2024. DOI: 10.1016/j.jisa.2023.103690
P. Maniriho, A. N. Mahmood, and M. J. M. Chowdhury, “API-MalDetect: Automated malware detection framework for windows based on API calls and deep learning techniques,” Journal of Network and Computer Applications, vol. 218, p. 103704, Sep. 2023. DOI: 10.1016/j.jnca.2023.103704
J. T. Santoso, B. Hartono, F. D. Silalahi, and M. Muthohir, “Transformers in Cybersecurity: Advancing Threat Detection and Response through Machine Learning Architectures,” Journal of Technology Informatics and Engineering, vol. 3, no. 3, pp. 382–396, Dec. 2024. DOI: 10.51903/jtie.v3i3.211
S. Berrios, D. Leiva, B. Olivares, H. Allende-Cid, and P. Hermosilla, “Systematic Review: Malware Detection and Classification in Cybersecurity,” Applied Sciences, vol. 15, no. 14, p. 7747, Jul. 2025. DOI: 10.3390/app15147747
K. Lee, J. Lee, and K. Yim, “Classification and Analysis of Malicious Code Detection Techniques Based on the APT Attack,” Applied Sciences, vol. 13, no. 5, p. 2894, Feb. 2023. DOI: 10.3390/app13052894
P. Maniriho, A. N. Mahmood, and M. J. M. Chowdhury, “MeMalDet: A memory analysis-based malware detection framework using deep autoencoders and stacked ensemble under temporal evaluations,” Computers & Security, vol. 142, p. 103864, Jul. 2024. DOI: 10.1016/j.cose.2024.103864
Dr. K. Chhillar, Dr. D. Tomar, and Prof. A. Verma, “A Hybrid Static–Dynamic Malware Analysis Framework Using Interpretable Neural Network,” International Journal of Scientific Research in Engineering and Management, vol. 09, no. 09, pp. 1–9, Sep. 2025. DOI: 10.55041/IJSREM52505
P. Schummer, A. del Rio, J. Serrano, D. Jimenez, G. Sánchez, and Á. Llorente, “Machine Learning-Based Network Anomaly Detection: Design, Implementation, and Evaluation,” AI, vol. 5, no. 4, pp. 2967–2983, Dec. 2024. DOI: 10.3390/ai5040143
Kamdan, Y. Pratama, R. S. Munzi, A. B. Mustafa, and I. L. Kharisma, “Static Malware Detection and Classification Using Machine Learning: A Random Forest Approach,” in The 7th International Global Conference Series on ICT Integration in Technical Education & Smart Society, 2025, p. 76. DOI: 10.3390/engproc2025107076
L. Alzubaidi et al., “Review of deep learning: concepts, CNN architectures, challenges, applications, future directions,” Journal of Big Data, vol. 8, no. 1, p. 53, Mar. 2021. DOI: 10.1186/s40537-021-00444-8
G. Karat, J. M. Kannimoola, N. Nair, A. Vazhayil, S. V G, and P. Poornachandran, “CNN-LSTM Hybrid Model for Enhanced Malware Analysis and Detection,” Procedia Computer Science, vol. 233, pp. 492–503, 2024. DOI: 10.1016/j.procs.2024.03.239
M. M. Rahman et al., “CNN vs Transformer Variants: Malware Classification Using Binary Malware Images,” in 2023 IEEE International Conference on Communication, Networks and Satellite (COMNETSAT), 2023, pp. 308–315. DOI: 10.1109/COMNETSAT59769.2023.10420585
L. Zou, X. Luo, Y. Zhang, X. Yang, and X. Wang, “HC-DTTSVM: A Network Intrusion Detection Method Based on Decision Tree Twin Support Vector Machine and Hierarchical Clustering,” IEEE Access, vol. 11, pp. 21404–21416, 2023. DOI: 10.1109/ACCESS.2023.3251354
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Muhammad Abdhi Priyatama, Dodon Turianto Nugrahadi, Irwan Budiman, Andi Farmadi, Mohammad Reza Faisal, Bedy Purnama, Puput Dani Prasetyo Adi, Luu Duc Ngo
This work is licensed under a Creative Commons Attribution 4.0 International License.
