Improving Lateral-Movement Intrusion Detection in Virtualized Networks using SHAP Feature Selection, SMOTE, and a Voting Ensemble Classifier

Abstract

Modern virtualized networks, such as those using VXLAN (Virtual eXtensible LAN), generate heavy east–west traffic, which can conceal the lateral movement of attackers. Detecting such infiltration attacks is challenging due to overlay encapsulation (e.g., VXLAN) and flat subnet architectures create blind spots for traditional IDS.  This study aims to evaluate a robust methodology for addressing class imbalance in intrusion detection by integrating SHAP-driven feature selection with SMOTE in a voting ensemble. We conducted an ablation study on the CICIDS2017 Thursday-WorkingHours-Afternoon-Infiltration subset, which is highly imbalanced (36 infiltration flows vs. 288,566 benign flows), varying SHAP feature sets (Top-5 vs. Top-30), classification thresholds , and SMOTE (Synthetic Minority Over-sampling Technique) balancing. The ensemble combined XGBoost, Random Forest, and Logistic Regression, and was evaluated with ROC-AUC, precision, recall, and F1-score. Results indicate that using more SHAP‑important features improves ROC‑AUC and recall, while SMOTE substantially enhances minority‑class detection. The best configuration is Top‑30 SHAP features with SMOTE at , achieved ROC‑AUC = 0.976 and F1‑score = 0.78, whereas using fewer features or omitting SMOTE significantly reduced recall and F1‑score. This synergy of interpretable feature selection and synthetic oversampling establishes a practical methodology for intrusion detection in highly imbalanced, modern virtualized environments. The novelty lies in demonstrating that SHAP + SMOTE integration yields both transparency and resilience, directly addressing encapsulation challenges in detecting stealthy lateral movement.