Performance Analysis of Traditional Machine Learning Classifiers on LSTM-Extracted Features for Indonesian Sign Language System Recognition
DOI:
https://doi.org/10.52436/1.jutif.2026.7.2.5210Keywords:
Deep Learning, Hand Gesture Recognition, Indonesian Sign Language System (SIBI), Machine Learning, MediaPipeAbstract
Recognizing affix gestures in the Indonesian Sign Language System (SIBI) remains challenging due to subtle visual differences in hand shape and movement, often resulting in lower classification accuracy compared to other categories. This study aims to evaluate whether lightweight traditional and hybrid classifiers can provide competitive performance to deep learning models for SIBI recognition. Using a dataset of 21,351 gesture videos covering four categories (Affix, Alphabet, Number, and Word), features were extracted from MediaPipe keypoints and processed as frozen LSTM embeddings. Six classifiers (Random Forest, K-Nearest Neighbors, Naïve Bayes, Multilayer Perceptron, Support Vector Machine, and Hidden Markov Model) were evaluated with 5-fold stratified cross-validation using accuracy, precision, recall, and F1-score, with statistical significance tested through Friedman and Nemenyi analyses. Results show that MLP and RF achieved high performance in Alphabet, Number, and Word categories (above 96 percent accuracy), while Affix remained the most difficult, with MLP reaching 81.17 percent, outperforming the 68.17 percent from a prior BiLSTM model. This study provides a benchmark for hybrid model implementation in sign language recognition, showing that while traditional classifiers on deep features are effective and computationally lighter for general gestures, deep architectures remain superior for capturing the fine-grained temporal nuances critical for complex categories like affixes.
Downloads
References
P. Ho and H. Santoso, “LSTM-Based Hand Gesture Recognition for Indonesian Sign Language System (SIBI) on Affix, Alphabet, Number, and Word,” Journal of Applied Informatics and Computing (JAIC), vol. 9, no. 3, pp. 928–937, Jun. 2025, doi: 10.30871/jaic.v9i3.9607.
K. Gajurel, C. Zhong, and G. Wang, “A Fine-Grained Visual Attention Approach for Fingerspelling Recognition in the Wild,” in 2021 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Melbourne: IEEE, May 2021, pp. 3266–3271. doi: 10.1109/SMC52423.2021.9658982.
I. D. M. B. A. Darmawan, Linawati, G. Sukadarmika, N. M. A. E. D. Wirastuti, and R. Pulungan, “Temporal Action Segmentation in Sign Language System for Bahasa Indonesia (SIBI) Videos Using Optical Flow-Based Approach,” Jurnal Ilmu Komputer dan Informasi, vol. 17, no. 2, pp. 195–202, Jun. 2024, doi: 10.21609/jiki.v17i2.1284.
I. D. M. B. A. Darmawan et al., “Advancing Total Communication in SIBI: A Proposed Conceptual Framework for Sign Language Translation,” in Proceedings - International Conference on Smart-Green Technology in Electrical and Information Systems, Badung: Institute of Electrical and Electronics Engineers Inc., Nov. 2023, pp. 23–28. doi: 10.1109/ICSGTEIS60500.2023.10424020.
F. Bigand, E. Prigent, B. Berret, and A. Braffort, “Machine Learning of Motion Statistics Reveals the Kinematic Signature of the Identity of a Person in Sign Language,” Frontiers in Bioengineering and Biotechnology, vol. 9, pp. 1–10, Jul. 2021, doi: 10.3389/fbioe.2021.710132.
A. Safonova, G. Ghazaryan, S. Stiller, M. Main-Knorn, C. Nendel, and M. Ryo, “Ten deep learning techniques to address small data problems with remote sensing,” International Journal of Applied Earth Observation and Geoinformation, vol. 125, p. 103569, Dec. 2023, doi: 10.1016/J.JAG.2023.103569.
I. H. Rather, S. Kumar, and A. H. Gandomi, “Breaking the data barrier: a review of deep learning techniques for democratizing AI with small datasets,” Artificial Intelligence Review, vol. 57, no. 9, p. 226, Aug. 2024, doi: 10.1007/S10462-024-10859-3.
W. Chen, K. Yang, Z. Yu, Y. Shi, and C. L. P. Chen, “A Survey on Imbalanced Learning: Latest Research, Applications and Future Directions,” Artificial Intelligence Review, vol. 57, no. 6, p. 137, Jun. 2024, doi: 10.1007/S10462-024-10759-6/FIGURES/11.
O. Khattach, O. Moussaoui, and M. Hassine, “Enhancing Machine Failure Prediction with a Hybrid Model Approach,” IAES International Journal of Artificial Intelligence (IJ-AI), vol. 13, no. 3, pp. 2946–2955, Sep. 2024, doi: 10.11591/ijai.v13.i3.pp2946-2955.
M. A. Metu, N. Akhter, S. Nasrin, T. Anzum, A. Khatun, and R. Mazumder, “Hybrid SVM-Bidirectional Long Short-Term Memory Model for Fine-Grained Software Requirement Classification,” Journal of Advances in Information Technology, vol. 15, no. 8, pp. 914–922, Aug. 2024, doi: 10.12720/jait.15.8.914-922.
S. Kamble, “SLRNet: A Real-Time LSTM-Based Sign Language Recognition System,” Jun. 2025. doi: 10.48550/arXiv.2506.11154.
S. Djaballah, L. Saidi, K. Meftah, A. Hechifa, M. Bajaj, and I. Zaitsev, “A Hybrid LSTM Random Forest Model with Grey Wolf Optimization for Enhanced Detection of Multiple Bearing Faults,” Scientific Reports, vol. 14, p. 23997, Dec. 2024, doi: 10.1038/s41598-024-75174-x.
M. Haidarh, C. Mu, Y. Liu, and X. He, “Exploring Traditional, Deep Learning and Hybrid Methods for Hyperspectral Image Classification: A Review,” Journal of Information and Intelligence, Apr. 2025, doi: 10.1016/J.JIIXD.2025.04.002.
C. Haibin and H. Yongliang, “A Hybrid LSTM and Decision Tree Model: A Novel Machine Learning Architecture for Complex Data Classification,” in 2023 IEEE International Conference on Sensors, Electronics and Computer Engineering, Jinzhou: IEEE, Aug. 2023, pp. 1441–1446. doi: 10.1109/ICSECE58870.2023.10263317.
L. Li, Y. Wu, Y. Ou, Q. Li, Y. Zhou, and D. Chen, “Research on Machine Learning Algorithms and Feature Extraction for Time Series,” in IEEE International Symposium on Personal, Indoor and Mobile Radio Communications, Montreal: IEEE, Jul. 2017, pp. 1–5. doi: 10.1109/PIMRC.2017.8292668.
E. Rakun, A. M. Arymurthy, L. Y. Stefanus, A. F. Wicaksono, and I. W. W. Wisesa, “Recognition of Sign Language System for Indonesian Language Using Long Short-Term Memory Neural Networks,” Advanced Science Letters, vol. 24, no. 2, pp. 999–1004, Mar. 2018, doi: 10.1166/ASL.2018.10675.
C. A. Sari, E. H. Rachmawanto, Z. Saifullah, C. Jatmoko, D. Sinaga, and S. Program, “Real-Time Detection of Indonesian Sign Language (ISL) Gestures Based on Long Short-Term Memory,” Journal of Soft Computing Exploration, vol. 5, no. 3, pp. 251–262, Sep. 2024, doi: 10.52465/JOSCEX.V5I3.452.
H. Joshi, V. Golhar, J. Gundawar, A. Gangurde, A. Yenkikar, and N. P. Sable, “Real-Time Sign Language Recognition and Sentence Generation,” in Proceedings of the International Conference on Innovative Computing & Communication (ICICC 2024), Elsevier BV, Oct. 2024. doi: 10.2139/SSRN.4992818.
J. P. Sahoo, S. Ari, and S. K. Patra, “Hand Gesture Recognition using PCA based Deep CNN Reduced Features and SVM Classifier,” in Proceedings - 2019 IEEE International Symposium on Smart Electronic Systems, iSES 2019, Rourkela: IEEE, Dec. 2019, pp. 221–224. doi: 10.1109/ISES47678.2019.00056.
N. Zerrouki, A. Houacine, F. Harrou, R. Bouarroudj, M. Y. Cherifi, and Y. Sun, “Exploiting Deep Learning-Based LSTM Classification for Improving Hand Gesture Recognition to Enhance Visitors’ Museum Experiences,” in 2022 International Conference on Innovation and Intelligence for Informatics, Computing, and Technologies, 3ICT 2022, Sakheer: Institute of Electrical and Electronics Engineers Inc., Nov. 2022, pp. 451–456. doi: 10.1109/3ICT56508.2022.9990722.
R. E. Nogales and M. E. Benalcázar, “Hand Gesture Recognition Using Automatic Feature Extraction and Deep Learning Algorithms with Memory,” Big Data and Cognitive Computing, vol. 7, no. 2, p. 102, May 2023, doi: 10.3390/BDCC7020102.
S. Wu, “A Compact LSTM-SVM Fusion Model for Long-Duration Cardiovascular Diseases Detection,” Michigan, Nov. 2023. doi: 10.48550/arXiv.2312.09442.
Z. F. Jailani and D. Nurmadewi, “Hybrid Machine Learning Predicts Flooding Using LSTM And Random Forests On Geodata,” INTECOMS: Journal of Information Technology and Computer Science, vol. 8, no. 1, pp. 35–41, Jan. 2025, doi: 10.31539/INTECOMS.V8I1.13991.
I. Simatupang, D. S. Pamungkas, and S. K. Risandriya, “Naïve Bayes Classifier for Hand Gestures Recognition,” in Proceedings of the 3rd International Conference on Applied Engineering - ICAE, Batam: Scitepress, Jul. 2021, pp. 110–114. doi: 10.5220/0010352601100114.
H. Ma’, A. Yudo Husodo, and B. Irmawati, “Performance Comparison of Naive Bayes and Bidirectional LSTM Algorithms In BSI Mobile Review Sentiment Analysis,” Jurnal Teknik Informatika (JUTIF), vol. 6, no. 1, pp. 159–172, Feb. 2025, doi: 10.52436/1.jutif.2025.6.1.4178.
S. Widodo, H. Brawijaya, and S. Samudi, “Stratified K-fold Cross Validation Optimization on Machine Learning for Prediction,” Sinkron : Jurnal dan Penelitian Teknik Informatika, vol. 6, no. 4, pp. 2407–2414, Oct. 2022, doi: 10.33395/SINKRON.V7I4.11792.
I. D. M. B. A. Darmawan, L. Linawati, G. Sukadarmika, N. M. A. E. D. Wirastuti, and R. Pulungan, “Indonesian Sign Language System (SIBI) Dataset,” Aug. 13, 2024, Mendeley Data. doi: 10.17632/44PBRBSNKH.3.
A. R. Isnain, J. Supriyanto, and M. P. Kharisma, “Implementation of K-Nearest Neighbor (K-NN) Algorithm For Public Sentiment Analysis of Online Learning,” IJCCS (Indonesian Journal of Computing and Cybernetics Systems), vol. 15, no. 2, pp. 121–130, Apr. 2021, doi: 10.22146/IJCCS.65176.
S. Shivani and S. B. Gupta, “A Comprehensive Analysis of Recognition of Hand Gestures using Machine Learning,” Makara Journal of Technology, vol. 29, no. 1, p. 5, Apr. 2025, doi: 10.7454/mst.v29i1.1679.
A. Desiani et al., “Penerapan Metode Support Vector Machine Dalam Klasifikasi Bunga Iris,” IJAI (Indonesian Journal of Applied Informatics), vol. 7, no. 1, pp. 12–18, Apr. 2023, doi: 10.20961/IJAI.V7I1.61486.
S. Sharma, S. Modi, P. S. Rana, and J. Bhattacharya, “Hand Gesture Recognition Using Gaussian Threshold and Different SVM Kernels,” in International Conference on Advances in Computing and Data Sciences, M. Singh, P. K. Gupta, V. Tyagi, J. Flusser, and T. Ören, Eds., Dehradun: Springer, Singapore, Oct. 2018, pp. 138–147. doi: 10.1007/978-981-13-1813-9_14.
K. Nguyen-Trong and T. T. T. Nguyen, “Optimization of Multi-Layer Perceptron Deep Neural Networks using Genetic Algorithms for Hand Gesture Recognition,” Journal of Computer Science, vol. 18, no. 2, pp. 57–66, Feb. 2022, doi: 10.3844/jcssp.2022.57.66.
D. Pardede, B. H. Hayadi, and Iskandar, “Kajian Literatur Multi Layer Perceptron: Seberapa Baik Performa Algoritma Ini,” Journal of ICT Aplication and System, vol. 1, no. 1, pp. 23–35, Jun. 2022, doi: 10.56313/jictas.v1i1.127.
N. Neverova, C. Wolf, G. W. Taylor, and F. Nebout, “Multi-scale Deep Learning for Gesture Detection and Localization,” in Lecture Notes in Computer Science (including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), L. Agapito, M. M. Bronstein, and C. Rother, Eds., Zurich: Springer, Cham, Jan. 2015, pp. 474–490. doi: 10.1007/978-3-319-16178-5_33.
N. S. O’Connell, B. C. Jaeger, G. S. Bullock, and J. L. Speiser, “A Comparison of Random Forest Variable Selection Methods for Regression Modeling of Continuous Outcomes,” Briefings in Bioinformatics, vol. 26, no. 2, pp. 1–15, Mar. 2025, doi: 10.1093/BIB/BBAF096.
A. Chauhan and E. Shyni, “Affordable Real-Time Hand Gesture Detection Using Random Forest,” International Journal of Innovative Science and Research Technology, vol. 10, no. 2, pp. 183–190, Feb. 2025, doi: 10.5281/zenodo.14898697.
S. Pavan and Y. Gowtham, “Hand Gesture Recognition System Using Random Forest,” International Journal of Advance Research in Science and Engineering, vol. 12, no. 4, pp. 125–129, Apr. 2023, Accessed: Jul. 29, 2025. [Online]. Available: https://www.ijarse.com/images/fullpdf/1682502367_737.pdf
O. Peretz, M. Koren, and O. Koren, “Naive Bayes Classifier – An Ensemble Procedure for Recall and Precision Enrichment,” Engineering Applications of Artificial Intelligence, vol. 136B, p. 108972, Jul. 2024, doi: 10.1016/j.engappai.2024.108972.
H. J. Escalante, E. F. Morales, and L. E. Sucar, “A Naïve Bayes Baseline for Early Gesture Recognition,” Pattern Recognition Letters, vol. 73, pp. 91–99, Apr. 2016, doi: 10.1016/j.patrec.2016.01.013.
Y. Ma et al., “The Hidden Markov Model and Its Applications in Bioinformatics Analysis,” Genes & Diseases, p. 101729, Jun. 2025, doi: 10.1016/J.GENDIS.2025.101729.
P. Morguet and M. Lang, “Spotting Dynamic Hand Gestures in Video Image Sequences Using Hidden Markov Models,” in Proceedings 1998 International Conference on Image Processing, Chicago: IEEE, Oct. 1998, pp. 1–6. doi: 10.1109/ICIP.1998.999009.
S. Sathyanarayanan and B. R. Tantri, “Confusion Matrix-Based Performance Evaluation Metrics,” African Journal of Biomedical Research, vol. 27, no. 4S, pp. 4023–4031, Nov. 2024, doi: 10.53555/AJBR.V27I4S.4345.
R. W. Mellyssa, A. F. Dewi, M. Misriana, S. Suryati, and R. Rachmawati, “Pengaruh Algoritma Deep Learning dalam Meningkatkan Akurasi Sistem Pendeteksian Kondisi Jalan Raya,” in Proceeding Seminar Nasional Politeknik Negeri Lhokseumawe, Lhokseumawe: Politeknik Negeri Lhokseumawe, Nov. 2022, pp. 12–16. Accessed: Aug. 14, 2025. [Online]. Available: https://e-jurnal.pnl.ac.id/semnaspnl/article/download/3426/2747
J. S. Akosa, “Predictive Accuracy: A Misleading Performance Measure for Highly Imbalanced Data,” in Proceedings of the SAS Global Forum, Orlando: SAS Institute Inc., Apr. 2017, pp. 1–4. Accessed: Jul. 30, 2025. [Online]. Available: https://support.sas.com/resources/papers/proceedings17/0942-2017.pdf
B. Karaman, A. Güven, A. Öner, and N. S. Kahraman, “Classification of Retinitis Pigmentosa Stages Based on Machine Learning by Fusion of Image Features of VF and MfERG Maps,” Electronics (Switzerland), vol. 14, no. 9, p. 1867, May 2025, doi: 10.3390/electronics14091867.
S. R. Bauskar, C. R. Madhavaram, E. P. Galla, J. R. Sunkara, H. K. Gollangi, and S. K. Rajaram, “Predictive Analytics for Project Risk Management Using Machine Learning,” Journal of Data Analysis and Information Processing, vol. 12, no. 4, pp. 566–580, Sep. 2024, doi: 10.4236/JDAIP.2024.124030.
J. Demšar, “Statistical Comparisons of Classifiers over Multiple Data Sets,” Journal of Machine Learning Research, vol. 7, pp. 1–30, 2006.
O. Rainio, J. Teuho, and R. Klén, “Evaluation metrics and statistical tests for machine learning,” Scientific Reports, vol. 14, no. 1, p. 6086, Mar. 2024, doi: 10.1038/s41598-024-56706-x.
Suharjito, N. Thiracitta, and H. Gunawan, “SIBI Sign Language Recognition Using Convolutional Neural Network Combined with Transfer Learning and non-trainable Parameters,” Procedia Computer Science, vol. 179, no. 1, pp. 72–80, Feb. 2021, doi: 10.1016/J.PROCS.2020.12.011.
P. D. Bormane and S. D. Shirbahadurkar, “Indian Sign Language Recognition: Support Vector Machine Approach,” Advances in Nonlinear Variational Inequalities, vol. 27, no. 3, pp. 716–727, Aug. 2024, doi: 10.52783/ANVI.V27.1438.
Additional Files
Published
How to Cite
Issue
Section
License
Copyright (c) 2026 Patricia Ho, Handri Santoso
This work is licensed under a Creative Commons Attribution 4.0 International License.
