LEVERAGING DEEP LEARNING APPROACH FOR ACCURATE ALPHABET RECOGNITION THROUGH HAND GESTURES IN SIGN LANGUAGE
Abstract
Sign language is one way of communication used by people who cannot speak or hear (deaf and speech impaired), so not everyone can understand sign language. Therefore, to facilitate communication between normal people and deaf and speech-impaired people, many systems have been created to translate gestures and signs in sign language into understandable words. Artificial intelligence and computer vision-based technologies, such as YOLOv9 offer solutions to recognize hand gestures more quickly, accurately, and efficiently. This research aims to develop a hand gesture detection system for alphabetic sign language using YOLOv9 architecture, with the aim of improving the accuracy and speed of hand gesture detection. The data used consists of 6500 sign language alphabet hand gesture images that have been labeled with bounding boxes and processed using image augmentation techniques. The model was trained on the Kaggle platform and evaluated using performance metrics such as Accuracy, Precision, Recall, F1-Score, and Intersection over Union (IoU). The results show that the YOLOv9 model achieves an average detection accuracy of 97%, with precision and recall above 90% for most classes. In addition, YOLOv9 shows advantages over other algorithms such as SSD MobileNet v2 and Faster RCNN, both in terms of speed and accuracy. In conclusion, YOLOv9 proved to be very effective in detecting sign language hand gestures, thereby speeding up and facilitating communication. This research is expected to contribute to the development of more inclusive technologies in various fields, such as education, public services, and employment opportunities, which support better communication between sign language users and the general public.
Downloads
References
S. Dwijayanti, Hermawati, S. I. Taqiyyah, H. Hikmarika, and B. Y. Suprapto, “Indonesia Sign Language Recognition using Convolutional Neural Network,” Int. J. Adv. Comput. Sci. Appl., vol. 12, no. 10, pp. 415–422, 2021, doi: 10.14569/IJACSA.2021.0121046.
S. C. Mesbahi, M. A. Mahraz, J. Riffi, and H. Tairi, “Hand Gesture Recognition Based on Various Deep Learning YOLO Models,” Int. J. Adv. Comput. Sci. Appl., vol. 14, no. 4, pp. 307–319, 2023, doi: 10.14569/IJACSA.2023.0140435.
S. Al Ahmadi, F. Mohammad, and H. Al Dawsari, “Efficient YOLO-Based Deep Learning Model for Arabic Sign Language Recognition,” J. Disabil. Res., vol. 3, no. 4, 2024, doi: 10.57197/jdr-2024-0051.
M. P. Geetha, S. Swetha, M. Subitsha, and K. V. Visnupriya, “Gesture Based Sign Language Recognition for Specially Challenged Using Yolov5,” Proc. 2024 Int. Conf. Sci. Technol. Eng. Manag. ICSTEM 2024, pp. 1–4, 2024, doi: 10.1109/ICSTEM61137.2024.10560764.
M. R. Ningsih et al., “Sign Language Detection System Using YOLOv5 Algorithm to Promote Communication Equality People with Disabilities,” Sci. J. Informatics, vol. 11, no. 2, pp. 549–558, 2024, doi: 10.15294/sji.v11i2.6007.
A. N. Sihananto, E. M. Safitri, Y. Maulana, F. Fakhruddin, and M. E. Yudistira, “Indonesian Sign Language Image Detection Using Convolutional Neural Network (CNN) Method,” Inspir. J. Teknol. Inf. dan Komun., vol. 13, no. 1, pp. 13–21, 2023, doi: 10.35585/inspir.v13i1.37.
S. Sharma, R. Sreemathy, M. Turuk, J. Jagdale, and S. Khurana, “Real-Time Word Level Sign Language Recognition Using YOLOv4,” 2022 Int. Conf. Futur. Technol. INCOFT 2022, pp. 1–7, 2022, doi: 10.1109/INCOFT55651.2022.10094530.
M. Alaftekin, I. Pacal, and K. Cicek, “Real-time sign language recognition based on YOLO algorithm,” Neural Comput. Appl., vol. 36, no. 14, pp. 7609–7624, 2024, doi: 10.1007/s00521-024-09503-6.
A. Mujahid et al., “Real-time hand gesture recognition based on deep learning YOLOv3 model,” Appl. Sci., vol. 11, no. 9, 2021, doi: 10.3390/app11094164.
M. Bhavadharshini, J. Josephine Racheal, M. Kamali, S. Sankar, and M. Bhavadharshini, “Sign language translator using YOLO algorithm,” Adv. Parallel Comput., vol. 39, pp. 159–166, 2021, doi: 10.3233/APC210136.
P. Battistoni, M. Di Gregorio, M. Romano, M. Sebillo, G. Vitiello, and G. Solimando, Sign language interactive learning - measuring the user engagement, vol. 12206 LNCS. Springer International Publishing, 2020. doi: 10.1007/978-3-030-50506-6_1.
L. Chandwani et al., “Gesture based Sign Language Recognition system using Mediapipe,” 2023, [Online]. Available: https://doi.org/10.21203/rs.3.rs-3106646/v1
A. A. Sonkamble, R. D. Chavhan, B. S. Jadhao, and S. M. Rathod, “Real-Time Indian Sign Language Detection using SSD-Mobilenet,” 2022 Sardar Patel Int. Conf. Ind. 4.0 - Nascent Technol. Sustain. “Make India” Initiat. SPICON 2022, pp. 1–5, 2022, doi: 10.1109/SPICON56577.2022.10180839.
T. Diwan, G. Anirudh, and J. V. Tembhurne, “Object detection using YOLO: challenges, architectural successors, datasets and applications,” Multimed. Tools Appl., vol. 82, no. 6, pp. 9243–9275, 2023, doi: 10.1007/s11042-022-13644-y.
N. F. Attia, M. T. F. S. Ahmed, and M. A. M. Alshewimy, “Efficient deep learning models based on tension techniques for sign language recognition,” Intell. Syst. with Appl., vol. 20, no. September, p. 200284, 2023, doi: 10.1016/j.iswa.2023.200284.
M. Shashishekhara, H. Hamza, and I. Etit-kit, “Real Time American Sign Language Detection Using Yolo-v9 Amna Imran”, doi: 10.48550/arXiv.2407.17950.
A. Al-Shaheen, M. Çevik, and A. Alqaraghuli, “American Sign Language Recognition using YOLOv4 Method,” Int. J. Multidiscip. Stud. Innov. Technol., vol. 6, no. 1, p. 61, 2022, doi: 10.36287/ijmsit.6.1.61.
D. Ma, K. Hirota, Y. Dai, and Z. Jia, “Dynamic Sign Language Recognition Based on Improved Residual-LSTM network,” Proc. 7th Int. Work. Adv. Comput. Intell. Intell. Informatics , pp. 1–6, 2021, [Online]. Available: https://iwaciii2021.bit.edu.cn/docs/2021-12/9caf45c507224bdca4933ad139fff639.pdf
S. A. Khan, Z. A. Ansari, R. Singh, M. S. Rawat, F. Z. Khan, and S. K. Yadav, “Sign Translation Via Natural Language Processing,” Asian J. Res. Comput. Sci., vol. 11, no. 1, pp. 1–7, 2021, doi: 10.9734/ajrcos/2021/v11i130251.
L. S. Teja Mangamuri, L. Jain, and A. Sharmay, “Two Hand Indian Sign Language dataset for benchmarking classification models of Machine Learning,” IEEE Int. Conf. Issues Challenges Intell. Comput. Tech. ICICT 2019, pp. 2–6, 2019, doi: 10.1109/ICICT46931.2019.8977713.
J. Zhao, X. H. Li, J. C. D. Cruz, M. S. Verdadero, J. C. Centeno, and J. M. Novelero, “Hand Gesture Recognition Based on Deep Learning,” 2023 Int. Conf. Digit. Appl. Transform. Econ. ICDATE 2023, vol. 14, no. 4, pp. 307–320, 2023, doi: 10.1109/ICDATE58146.2023.10248500.
T. F. Dima and M. E. Ahmed, “Using YOLOv5 Algorithm to Detect and Recognize American Sign Language,” 2021 Int. Conf. Inf. Technol. ICIT 2021 - Proc., pp. 603–607, 2021, doi: 10.1109/ICIT52682.2021.9491672.
D. A. Abdurrafi, M. T. Alawiy, and B. M. Basuki, “Deteksi Klasifikasi Dan Menghitung Kendaraan Berbasis Algoritma You Only Look Once (YOLO) Menggunakan Kamera CCTV,” Sci. Electro, no. 1, pp. 1–9, 2023, [Online]. Available: https://jim.unisma.ac.id/index.php/jte/article/viewFile/21551/16069
M. P. K. Putra and Wahyono, “A Novel Method for Handling Partial Occlusion on Person Re-identification using Partial Siamese Network,” Int. J. Adv. Comput. Sci. Appl., vol. 12, no. 7, pp. 313–321, 2021, doi: 10.14569/IJACSA.2021.0120735.
D. D. Aboyomi and C. Daniel, “A Comparative Analysis of Modern Object Detection Algorithms: YOLO vs. SSD vs. Faster R-CNN,” ITEJ (Information Technol. Eng. Journals), vol. 8, no. 2, pp. 96–106, 2023, doi: 10.24235/itej.v8i2.123.
R. H. Abiyev, M. Arslan, and J. B. Idoko, “Sign language translation using deep convolutional neural networks,” KSII Trans. Internet Inf. Syst., vol. 14, no. 2, pp. 631–653, 2020, doi: 10.3837/tiis.2020.02.009.
S. Tyagi, P. Upadhyay, I. Hoor Fatima, and A. Kumar Sharma, “American Sign Language Detection using YOLOv5 and YOLOv8,” pp. 1–16, 2023, [Online]. Available: https://doi.org/10.21203/rs.3.rs-3126918/v1.
Copyright (c) 2025 Nadiyan Syah Wahyu Nugroho, Muhammad Pajar Kharisma Putra
This work is licensed under a Creative Commons Attribution 4.0 International License.
