Be Wary of the Overfitting Trap: A Comparative Study between XGBoost and LSTM to Predict Stock Rise and Fall
Main Article Content
Keywords
overfitting, stock prediction, LSTM, XGBoost, literature review
Abstract
With the development of artificial intelligence technology and the transformation of financial research methods, machine learning is widely used in financial prediction, and overfitting is the most critical factor that impacts the ability of financial prediction model to generalize. The paper primarily studies the two main models (tree ensemble model XGBoost and neural network model LSTM) and their over-fitting problem in the stock rise and fall prediction scenario. This paper claims that LSTM can be over-fitted more easily than XGBoost, but can still overcome this by its use of large data sets and effective regularization. But the problem is that, XGBoost has low risk while market drift has a great impact upon its output results. The mixed model of the two can be effectively learned from each other and is currently the best solution. Hopefully, the conclusions of this paper can give some reference for the related scholars to understand the pitfalls of financial machine learning.
References
- [1] Adiyarta K, Sugihartono T, Sulaiman R, et al. XGBoost vs. LSTM: A Comparative Performance Analysis for Global Stock Market Prediction[C]//2025 7th International Conference on Cybernetics and Intelligent System (ICORIS). 2025: 1-6
- [2] Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pp. 785-794.
- [3] Hochreiter, S., & Schmidhuber, J. (1997). Long Short-Term Memory. Neural Computation, 9(8), 1735-1780.
- [4] Fischer, T., & Krauss, C. (2018). Deep learning with long short-term memory networks for financial market predictions. European Journal of Operational Research, 270(2), 654-669.
- [5] Krauss, C., Do, X. A., & Huck, N. (2017). Deep neural networks, gradient-boosted trees, random forests: Statistical arbitrage on the S&P 500. European Journal of Operational Research, 259(2), 689-702.
- [6] Siami-Namini, S., Tavakoli, N., & Namin, A. S. (2019). A Comparison of ARIMA and LSTM in Forecasting Time Series. In 2019 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 1394-1401.
- [7] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: A Simple Way to Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15(56), 1929-1958.
- [8] Gal, Y., & Ghahramani, Z. (2016). A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. In Advances in Neural Information Processing Systems 29 (NIPS 2016), pp. 1019-1027.
- [9] Dezhkam, A., Manzuri, M. T., Aghapour, A., Karimi, A., Rabiee, A., & Shalmani, S. M. (2023). A Bayesian-based classification framework for financial time series trend prediction. The Journal of Supercomputing, 79(4), 4622–4659.
- [10] Wen, Q., Sun, L., Yang, F., Song, X., Gao, J., Wang, X., & Xu, H. (2021). Time series data augmentation for deep learning: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (pp. 4653–4660).
- [11] Zaremba, W., & Sutskever, i. (2014). Learning to execute. ArXiv: 1410.4615. https://doi.org/10.48550/arXiv.1410.4615
- [12] Ardana, A. (2023). Performance analysis of XGBoost algorithm to determine the most optimal parameters and features in predicting stock price movement. Telematika: Jurnal Informatika dan Teknologi Informasi, 20(1), 91–102.
- [13] Fawaz, H. I., Forestier, G., Weber, J., Idoumghar, L., & Muller, P. A. (2019). Deep learning for time series classification: A review. Data Mining and Knowledge Discovery, 33(4), 917–963.
- [14] Nabipour, M., Nayyeri, P., Jabani, H., Shamshirband, S., & Mosavi, A. (2020). Predicting stock market trends using machine learning and deep learning algorithms via continuous and binary data; a comparative analysis. IEEE Access, 8, 150199-150212.
- [15] Peng Y, de Moraes Souza J G. Chaos, overfitting and equilibrium: To what extent can machine learning beat the financial market?[J]. International Review of Financial Analysis, 2024, 95: 103474.
- [16] Gumelar, A. B., Setyorini, H., Adi, D. P., Nilowardono, S., Latipah, Widodo, A., Wibowo, A. T., Sulistyono, M. T., & Christine, E. (2020). Boosting the accuracy of stock market prediction using XGBoost and Long Short-Term Memory. In Proceedings of the 2020 International Seminar on Application for Technology of Information and Communication (iSemantic) (pp. 609-613). IEEE.
- [17] Jiang, M., Liu, J., Zhang, L., & Liu, C. (2020). An improved Stacking framework for stock index prediction by leveraging tree-based ensemble models and deep learning algorithms. Physica A: Statistical Mechanics and its Applications, 541, 122272.
- [18] Fang Z, Ma X, Pan H F, et al. Movement Forecasting of Financial Time Series Based on Adaptive LSTM-BN Network[J]. Expert Systems with Applications, 2023, 213(Part): 119207.
- [19] Stock Market Prediction: XGBoost and LSTM Comparative Analysis[C]//2024 3rd International Conference on Artificial Intelligence for Internet of Things (AIIoT). Vellore: IEEE, 2024: 1-6.
- [20] Stocks and Models: A Comparative Benchmark of Machine Learning and Deep Learning Approaches for Stock Market Prediction[J]. Procedia Computer Science, 2025, 269: 752-760.
- [21] Liang L C. ARIMA with Attention-based CNN-LSTM and XGBoost Hybrid Model for Stock Prediction in the US Stock Market[J]. SHS Web of Conferences, 2024, 196: 02001.
- [22] Din R U, et al. A Novel Decision Ensemble Framework: Customized Attention-BiLSTM and XGBoost for Speculative Stock Price Forecasting[J/OL]. arXiv preprint, arXiv:2401.11621, 2024.
- [23] Stock Price Prediction Based on Multifactorial Linear Models and Machine Learning Approaches[C]//2023 IEEE 7th Conference on Information and Communication Technology (CICT). Jabalpur: IEEE, 2023: 1-6.
- [24] Chen W, Chen B, Cai X. Forecasting China‘s Stock Market Risk Under the Background of the Stock Connect Programs[J]. Soft Computing, 27
- [25] Oukhouya H, El Himdi K. Comparing Machine Learning Methods—SVR, XGBoost, LSTM, and MLP—For Forecasting the Moroccan Stock Market[J]. Computer Science and Mathematics Forum, 2023, 7(1): 39.
- [26] Galegale, N. V., & Shimabukuro, C. I. (2024). Deep learning applied to stock prices: Epoch adjustment in training an LSTM neural network. International Journal of Business and Management, 19(4), 80.
- [27] Xia, Y., & Wang, Y. (2021). Predicting stock price movement with multiple data sources and machine learning models. In J. Zeng, P. Qin, W. Jing, X. Song, & Z. Lu (Eds.), Data Science: 7th International Conference of Pioneering Computer Scientists, Engineers and Educators, ICPCSEE 2021, Taiyuan, China, September 17–20, 2021, Proceedings, Part I (Vol. 1451, pp. 90-105). Springer.
- [28] Kim, H. Y., & Won, C. H. (2018). Forecasting the volatility of stock price index: A hybrid model integrating LSTM with multiple GARCH-type models. Expert Systems with Applications, 103, 25-37.
- [29] Moghar, A., & Hamiche, M. (2020). Stock market prediction using LSTM recurrent neural network. Procedia Computer Science, 170, 1168–1173.
- [30] Kapoor, S. (2021). A simple and fast baseline for tuning large XGBoost models. arXiv.
Article Sidebar
Full Text
PDF
Information
Copy Publication Information
Published
Oct 19, 2026
Conference Proceedings Volume
Section
Articles
This work is licensed under a Creative Commons Attribution 4.0 International License.
How to Cite
Zhang, Jiahao. “Be Wary of the Overfitting Trap: A Comparative Study Between XGBoost and LSTM to Predict Stock Rise and Fall”. Exploring Science Academic Conference Series, vol. 17, Oct. 2026, pp. 41-50, https://doi.org/10.70267/icrafm.2026174150.