Anomaly Detection of Denial-of-Service Network Traffic Attacks using Autoencoders and Isolation Forest
DOI:
https://doi.org/10.31436/ijpcc.v12i1.680Abstract
This paper presents an unsupervised network-based anomaly detection framework that integrates deep autoencoders with the Isolation Forest algorithm. The framework analyzes extracted traffic features, including packet length and IP address patterns, to detect deviations from normal behaviour without requiring labelled data. Autoencoders reconstruct benign traffic to highlight subtle deviations, while Isolation Forest efficiently assigns anomaly scores to identify statistical outliers in large-scale, unlabelled datasets. Experimental evaluation shows that the Isolation Forest model achieves a low mean squared error (MSE) of 0.0065 with an accuracy of 9.79%, indicating stable anomaly score separation, whereas the standalone autoencoder records a substantially higher reconstruction error (MSE = 3.92 × 10¹?) and an accuracy of 6.09%, reflecting the difficulty of modelling complex and highly variable network traffic patterns. By combining both approaches, the proposed framework improves overall detection performance, achieving a higher accuracy of 13.55%, and demonstrates enhanced capability in detecting both volumetric and stealthy attacks, such as application-layer denial-of-service (DoS) traffic. Visualization of traffic behaviour further supports the analysis, revealing clearer separation between normal and anomalous flows when both models are integrated. These findings highlight the complementary strengths of statistical outlier detection and deep learning-based reconstruction, providing a practical foundation for adaptive and real-time anomaly monitoring in dynamic network environments.
References
C. Jha and C. S. Dash, "Real-Time Slowloris Attack Detection and Mitigation with Machine Learning Techniques," Int. J. Eng. Res. Technol., vol. 13, no. 9, Sep. 2024.
W. Chua et al., "Web Traffic Anomaly Detection Using Isolation Forest," Future Internet, vol. 11, no. 4, p. 83, 2023.
M. A. Rassam, "Autoencoder-Based Neural Network Model for Anomaly Detection in Wireless Body Area Networks," Electronics, vol. 5, no. 4, p. 39, 2021.
G. Geng et al., "Enhanced Isolation Forest-Based Algorithm for Unsupervised Anomaly Detection in Lidar SLAM Localization," World Electr. Veh. J., vol. 16, no. 4, p. 209, 2025.
F. Farahnakian and J. Heikkonen, "A Deep Auto-Encoder Based Approach for Intrusion Detection System," Proc. 2018 Int. Conf. Adv. Commun. Tech. (ICACT), 2018, pp. 603-611.
M. K. M. Almansoori and M. Telek, "Anomaly Detection Using Combination of Autoencoder and Isolation Forest," Proc. 2023 IEEE Global Workshop on Information Security and Privacy (WISP), 2023, pp. 48-53.
T. Smolen and L. Benova, "Comparing Autoencoder and Isolation Forest in Network Anomaly Detection," Proc. 2023 33rd Conf. Open Innovations Assoc. (FRUCT), 2023, pp. 89-96.
S. A. Elsaid and A. Binbusayyis, "An Optimized Isolation Forest Based Intrusion Detection System for Heterogeneous and Streaming Data in the Industrial Internet of Things (IIoT) Networks," Discover Appl. Sci., vol. 6, p. 483, Sept. 2024.
F. Moomtaheen et al., "Extended Isolation Forest for Intrusion Detection in Zeek Data," Information, vol. 15, no. 7, p. 404, 2024.
S. A. Hussein and S. R. Répás, "Enhancing Network Security through Machine Learning-Based Anomaly Detection Systems," Int. J. Intell. Syst. Appl. Eng., vol. 12, no. 21S, 2024.
S. Dev and A. D. Jurcut, "Network Anomaly Detection Using LSTM Based Autoencoder," Proc. 16th ACM Symp. QoS & Security Wireless Mobile Netw., 2020, pp. 37-45.
H. Huang et al., "Deep Learning Advancements in Anomaly Detection: A Comprehensive Survey," arXiv preprint arXiv:2503.13195, 2025.
H. Rhachi et al., "Enhanced Anomaly Detection in IoT Networks Using Deep Autoencoders with Feature Selection Techniques," Sensors, vol. 25, no. 10, p. 3150, 2025.
E. Krzyszto? et al., "A Comparative Analysis of Anomaly Detection Methods in IoT Networks: An Experimental Study," Appl. Sci., vol. 14, no. 24, p. 11545, 2024.
Y. Feng et al., "An Improved X-means and Isolation Forest Based Methodology for Network Traffic Anomaly Detection," PLoS ONE, vol. 17, no. 1, Jan. 2022, Art. no. e0263423
R. Chalapathy and S. Chawla, "Deep learning for anomaly detection: A survey," ACM Computing Surveys (CSUR), vol. 52, no. 1, pp. 1 - 38, Feb. 2019.”
J. An and S. Cho, “Variational autoencoder based anomaly detection using reconstruction probability,” in Proc. 2021 Int. Conf. Computer and Information Sciences (ICCIS), 2021, pp. 1-6
G. Kim, S. Lee, and S. Kim “A novel hybrid intrusion detection method integrating anomaly detection with misuse detection,” Expert Syst. Appl., vol 41, no. 4, pp. 1690-1700, 2018.
S. Khan and A. Mailewa, ‘Predicting anomalies in computer networks using autoencoder-based representation learning’, International Journal of Informatics and Communication Technology (IJ-ICT), vol. 13, p. 9, 04 2024.
S. Hore, Q. H. Nguyen, Y. Xu, A. Shah, N. D. Bastian, and T. Le, “Empirical evaluation of autoencoder models for anomaly detection in packet-based NIDS,” in Proc. IEEE Conf. Dependable and Secure Computing (DSC), Nov. 2023, pp. 1–8.
T. P. Nguyen, J. Cho, and D. Kim, “Semi-supervised intrusion detection system for in-vehicle networks based on variational autoencoder and adversarial reinforcement learning,” Knowledge-Based Systems, vol. 304, p. 112563, 2024.
R. Sharma and M. Grover, “Enhancing Cybersecurity with Machine Learning: Evaluating the Efficacy of Isolation Forests and Autoencoders in Anomaly Detection,” vol. 11, pp. 1017–1021, Aug. 2024, doi: 10.1109/iccpct61902.2024.10673338
