Prediction of Indonesian Banking Stock Prices Using a Hybrid LSTM and XGBoost Model with Optuna Based Hyperparameter Optimization

Abstract

Stock price prediction is a critical task in investment decision-making, particularly in highly volatile financial markets such as the Indonesian banking sector. While Long Short-Term Memory (LSTM) networks are effective in modeling temporal dependencies, they often fail to capture nonlinear residual patterns in financial time-series data, and their performance is highly sensitive to hyperparameter selection. To address these limitations, this study proposes a residual learning–based hybrid LSTM–XGBoost framework optimized using Optuna for predicting stock prices of major Indonesian banking stocks, namely BBCA, BBNI, BBRI, and BMRI. LSTM is employed as the base learner to model log-return sequences, while XGBoost is used to learn nonlinear residual structures from LSTM predictions. Latent embeddings extracted from the LSTM are further refined using Principal Component Analysis (PCA) to reduce redundancy and improve generalization. Hyperparameters of the LSTM, PCA, XGBoost, and calibration components are jointly optimized using Optuna with a Tree-structured Parzen Estimator (TPE) strategy. Experimental results demonstrate that the Optuna-optimized hybrid model consistently outperforms the baseline hybrid model across all datasets, achieving lower Mean Absolute Percentage Error (MAPE) values of 1.196% for BBCA, 1.67% for BBNI, 1.53% for BBRI, and 1.70% for BMRI. Additional stability analyses confirm that the proposed framework delivers stable and reliable predictions on unseen data. These findings provide a scalable hybrid forecasting framework that contributes to the development of intelligent financial decision-support systems and demonstrates the effectiveness of adaptive hybrid deep learning optimization techniques in real-world time-series prediction problems within the field of informatics.