A STUDY OF WORLDWIDE PATTERNS IN ALPHABET SIGN LANGUAGE RECOGNITION USING CONVOLUTIONAL AND RECURRENT NEURAL NETWORKS

Aris Rakhmadi; Anton Yudhana; Sunardi Sunardi

doi:10.52436/1.jutif.2025.6.1.4202

Aris Rakhmadi Informatics, Universitas Ahmad Dahlan, Indonesia
Anton Yudhana Informatics Engineering, Universitas Muhammadiyah Surakarta, Indonesia
Sunardi Electrical Engineering, Universitas Ahmad Dahlan, Indonesia

DOI: https://doi.org/10.52436/1.jutif.2025.6.1.4202

Keywords: Sign Language Recognition, Convolutional Neural Networks, Recurrent Neural Networks, Deep Learning

Abstract

Sign Language Recognition (SLR) has become an essential area of research due to its potential to promote understanding between the deaf and hearing communities through communication. This paper provides an in-depth study of various methodologies and models employed in SLR, focusing on Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN). We analyze their application to datasets from various sign languages, such as Arabic Sign Language (ArSL), American Sign Language (ASL), and British Sign Language (BSL), and explore how these models improve the recognition of dynamic, multi-dimensional hand gestures. This research not only advances the understanding of deep learning applications in sign language recognition but also addresses critical challenges in data processing and real-time applications, paving the way for inclusive technologies in informatics and human-computer interaction. Despite the progress in applying deep learning techniques to SLR, several challenges remain, particularly in dataset limitations, handling large vocabularies, and ensuring consistent performance across diverse environments and signers. The paper also investigates the broader applications of SLR, such as virtual reality, healthcare, education, and accessibility, and discusses the integration of SLR with human-computer interaction systems. Furthermore, it highlights current limitations in the field, such as difficulties with video data handling, the need for standard datasets, and issues related to training computational models. Finally, the paper outlines future research directions, including developing more robust SLR systems that can function effectively in uncontrolled environments, improving data collection methodologies, and creating real-time, user-friendly applications to assist the community of deaf and hard-of-hearing individuals.

Downloads

Download data is not yet available.

References

M. D. Meitantya, C. A. Sari, E. H. Rachmawanto, and R. R. Ali, “VGG-16 Architecture on CNN for American Sign Language Classification,” Jurnal Teknik Informatika (Jutif), vol. 5, no. 4, pp. 1165–1171, Jul. 2024, doi: 10.52436/1.jutif.2024.5.4.2160.

A. Yudhana, J. Rahmawan, and S. A. Akbar, “EonTex Conductive Strechtable Sensor Response on Smart Glove for Sign Language,” 2019.

G. Tharwat, A. M. Ahmed, and B. Bouallegue, “Arabic Sign Language Recognition System for Alphabets Using Machine Learning Techniques,” Journal of Electrical and Computer Engineering, vol. 2021, 2021, doi: 10.1155/2021/2995851.

A. Duarte et al., “How2Sign: A Large-scale Multimodal Dataset for Continuous American Sign Language,” 2021. [Online]. Available: http://how2sign.github.io/

C. Kwenda, M. Gwetu, and J. V. Fonou-Dombeu, “Ontology with Deep Learning for Forest Image Classification,” Applied Sciences (Switzerland), vol. 13, no. 8, Apr. 2023, doi: 10.3390/app13085060.

S. Luong, “Video Sign Language Recognition using Pose Extraction and Deep Learning Models,” San Jose State University, San Jose, CA, USA, 2023. doi: 10.31979/etd.jm4c-myd4.

Z. R. Saeed, Z. B. Zainol, B. B. Zaidan, and A. H. Alamoodi, “A Systematic Review on Systems-Based Sensory Gloves for Sign Language Pattern Recognition: An Update from 2017 to 2022,” 2022, Institute of Electrical and Electronics Engineers Inc. doi: 10.1109/ACCESS.2022.3219430.

G. Latif, N. Mohammad, J. Alghazo, R. AlKhalaf, and R. AlKhalaf, “ArASL: Arabic Alphabets Sign Language Dataset,” Data Brief, vol. 23, Apr. 2019, doi: 10.1016/j.dib.2019.103777.

A. A. J. Jim, I. Rafi, M. Z. Akon, U. Biswas, and A. Al Nahid, “KU-BdSL: An open dataset for Bengali sign language recognition,” Data Brief, vol. 51, Dec. 2023, doi: 10.1016/j.dib.2023.109797.

I. D. A. Rachmawati, R. Yunanda, M. F. Hidayat, and P. Wicaksono, “Deep Transfer Learning for Sign Language Image Classification: A Bisindo Dataset Study,” Engineering, MAthematics and Computer Science Journal (EMACS), vol. 5, no. 3, pp. 175–180, Sep. 2023, doi: 10.21512/emacsjournal.v5i3.10621.

M. A. Hossain, D. Bala, B. Sarkar, and I. Abdullah, “American Sign Language Alphabets Recognition using Convolutional Neural Network.” [Online]. Available: https://www.researchgate.net/publication/352878275

B. Alsharif, A. S. Altaher, A. Altaher, M. Ilyas, and E. Alalwany, “Deep Learning Technology to Recognize American Sign Language Alphabet,” Sensors, vol. 23, no. 18, Sep. 2023, doi: 10.3390/s23187970.

I. Irvanizam, I. Horatius, and H. Sofyan, “Applying Artificial Neural Network Based on Backpropagation Method for Indonesian Sign Language Recognition,” International Journal of Computing and Digital Systems, vol. 14, no. 1, pp. 975–985, 2023, doi: 10.12785/ijcds/140176.

Z. Zhang et al., “Artificial intelligence in cyber security: research advances, challenges, and opportunities,” Artif Intell Rev, vol. 55, no. 2, pp. 1029–1053, Feb. 2022, doi: 10.1007/s10462-021-09976-0.

A. W. Salehi et al., “A Study of CNN and Transfer Learning in Medical Imaging: Advantages, Challenges, Future Scope,” Apr. 01, 2023, MDPI. doi: 10.3390/su15075930.

B. Joksimoski et al., “Technological Solutions for Sign Language Recognition: A Scoping Review of Research Trends, Challenges, and Opportunities,” 2022, Institute of Electrical and Electronics Engineers Inc. doi: 10.1109/ACCESS.2022.3161440.

M. S. Amin, S. T. H. Rizvi, A. Mazzei, and L. Anselma, “Assistive Data Glove for Isolated Static Postures Recognition in American Sign Language Using Neural Network,” Electronics (Switzerland), vol. 12, no. 8, Apr. 2023, doi: 10.3390/electronics12081904.

A. Hekmat, M. Ali, H. H. Abbas, and I. Shahadi, “Sign Language Recognition and Hand Gestures Review,” vol. 02, no. 04, 2022, [Online]. Available: https://kjes.uokerbala.edu.iq/

A. Rahagiyanto, A. Basuki, R. Sigit, A. Anwar, and M. Zikky, “Hand Gesture Classification for Sign Language Using Artificial Neural Network,” in ICSEC 2017 - 21st International Computer Science and Engineering Conference 2017, Proceeding, Institute of Electrical and Electronics Engineers Inc., Aug. 2018, pp. 205–209. doi: 10.1109/ICSEC.2017.8443898.

M. Hocine, B. Mohammed Kamel, and B. Mohamed, “Hand gesture and sign language recognition based on deep learning.”

C. Lu, M. Kozakai, and L. Jing, “Sign Language Recognition with Multimodal Sensors and Deep Learning Methods,” Electronics (Switzerland), vol. 12, no. 23, Dec. 2023, doi: 10.3390/electronics12234827.

A. Muis, S. Sunardi, and A. Yudhana, “CNN-based Approach for Enhancing Brain Tumor Image Classification Accuracy,” International Journal of Engineering, Transactions B: Applications, vol. 37, no. 5, pp. 984–996, May 2024, doi: 10.5829/ije.2024.37.05b.15.

B. Awaji et al., “Hybrid Techniques of Facial Feature Image Analysis for Early Detection of Autism Spectrum Disorder Based on Combined CNN Features,” Diagnostics, vol. 13, no. 18, Sep. 2023, doi: 10.3390/diagnostics13182948.

J. Shin, A. Matsuoka, M. A. M. Hasan, and A. Y. Srizon, “American sign language alphabet recognition by extracting feature from hand pose estimation,” Sensors, vol. 21, no. 17, Sep. 2021, doi: 10.3390/s21175856.

M. S. Abdallah, E. Hemayed, M. S. Abdalla, and E. E. Hemayed, “Dynamic Hand Gesture Recognition of Arabic Sign Language using Hand Motion Trajectory Features Dynamic Hand Gesture Recognition of Arabic Sign Language using Hand Motion Trajectory Features,” 2013. [Online]. Available: https://www.researchgate.net/publication/258172682

S. Katoch, V. Singh, and U. S. Tiwary, “Indian Sign Language recognition system using SURF with SVM and CNN,” Array, vol. 14, Jul. 2022, doi: 10.1016/j.array.2022.100141.

A. Z. Nugraha, R. F. Salsabila, A. N. Handayani, A. P. Wibawa, E. Hitipeuw, and K. Arai, “Decision tree based algorithms for Indonesian Language Sign System (SIBI) recognition,” Applied Engineering and Technology, vol. 3, no. 2, pp. 86–101, Aug. 2024, doi: 10.31763/aet.v3i2.1536.

S. Oulad-Naoui, H. Ben-Abderrahmane, A. Chagha, and A. Cherif, “An LSTM-based System for Dynamic Arabic Sign Language Recognition,” in Proceedings ofthe International Conference on Emerging Intelligent Systems for Sustainable Development (ICEIS 2024), 2024, pp. 313–323. doi: 10.2991/978-94-6463-496-9_24.

B. Sundar and T. Bagyammal, “American Sign Language Recognition for Alphabets Using MediaPipe and LSTM,” in Procedia Computer Science, Elsevier B.V., 2022, pp. 642–651. doi: 10.1016/j.procs.2022.12.066.

A. M. Buttar, U. Ahmad, A. H. Gumaei, A. Assiri, M. A. Akbar, and B. F. Alkhamees, “Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs,” Mathematics, vol. 11, no. 17, Sep. 2023, doi: 10.3390/math11173729.

N. Al Mudawi et al., “Innovative healthcare solutions: robust hand gesture recognition of daily life routines using 1D CNN,” Front Bioeng Biotechnol, vol. 12, 2024, doi: 10.3389/fbioe.2024.1401803.

N. Adaloglou et al., “A Comprehensive Study on Deep Learning-Based Methods for Sign Language Recognition,” IEEE Trans Multimedia, vol. 24, pp. 1750–1762, 2022, doi: 10.1109/TMM.2021.3070438.

A. M. Buttar, U. Ahmad, A. H. Gumaei, A. Assiri, M. A. Akbar, and B. F. Alkhamees, “Deep Learning in Sign Language Recognition: A Hybrid Approach for the Recognition of Static and Dynamic Signs,” Mathematics, vol. 11, no. 17, 2023, doi: 10.3390/math11173729.

P. T. Nguyen, T. H. Nguyen, N. X. N. Hoang, H. T. B. Phan, H. S. H. Vu, and H. N. Huynh, “Exploring MediaPipe optimization strategies for real-time sign language recognition,” CTU Journal of Innovation and Sustainable Development, vol. 15, no. ISDS, pp. 142–152, Oct. 2023, doi: 10.22144/ctujoisd.2023.045.

A. Tayade and A. Halder, “Real-time Vernacular Sign Language Recognition using MediaPipe and Machine Learning,” International Journal of Research Publication and Reviews, vol. 2, no. 5, 2021, doi: 10.13140/RG.2.2.32364.03203.

S. K. Hussin, O. Mohamed, ; Mustafa Mohamed, E. Ahmed, and O. Mahmoud, “Real-time Arabic sign language translator Using media pipe and LSTM.” [Online]. Available: https://plomscience.com/journals/index.php/PLOMSAI/index

M. Papatsimouli, P. Sarigiannidis, and G. F. Fragulis, “A Survey of Advancements in Real-Time Sign Language Translators: Integration with IoT Technology,” Aug. 01, 2023, Multidisciplinary Digital Publishing Institute (MDPI). doi: 10.3390/technologies11040083.

M. S. Abdallah, G. H. Samaan, A. R. Wadie, F. Makhmudov, and Y. I. Cho, “Light-Weight Deep Learning Techniques with Advanced Processing for Real-Time Hand Gesture Recognition,” Sensors, vol. 23, no. 1, Jan. 2023, doi: 10.3390/s23010002.

R. S. Abdul Ameer, M. A. Ahmed, Z. T. Al-Qaysi, M. M. Salih, and M. L. Shuwandy, “Empowering Communication: A Deep Learning Framework for Arabic Sign Language Recognition with an Attention Mechanism,” Computers, vol. 13, no. 6, Jun. 2024, doi: 10.3390/computers13060153.

A. Rakhmadi and A. Yudhana, “Virtual Reality and Augmented Reality in Sign Language Recognition: A Review of Current Approaches,” International Journal of Informatics and Computation (IJICOM), vol. 6, no. 2, 2024, doi: 10.35842/ijicom.

T. H. Noor et al., “Real-Time Arabic Sign Language Recognition Using a Hybrid Deep Learning Model,” Sensors, vol. 24, no. 11, Jun. 2024, doi: 10.3390/s24113683.

I. Puspitasari, A. Yudhana, D. E. Wati, and S. Al Irfan, “Recognizing Micro Expression Pattern Using Convolutional Neural Networks (CNN) Method During Emotion Regulation Training for Parents in The Pandemic Era,” 2023. [Online]. Available: https://icistech.org/index.php/icistech/

N. Nurhadi, E. A. Winanto, R. M. Said, J. Jasmir, and L. Afuan, “PATTERN CLASSIFICATION SIGN LANGUAGE USING FEATURES DESCRIPTORS AND MACHINE LEARNING,” Jurnal Teknik Informatika (Jutif), vol. 5, no. 2, pp. 349–356, Apr. 2024, doi: 10.52436/1.jutif.2024.5.2.1228.

H. M. Al-Barhamtoshy, K. M. Jambi, M. A. Rashwan, and S. M. Abdou, “An Arabic Manuscript Regions Detection, Recognition and Its Applications for OCRing,” ACM Transactions on Asian and Low-Resource Language Information Processing, vol. 22, no. 1, Feb. 2023, doi: 10.1145/3532609.

B. Y. Al-Khuraym and M. M. Ben Ismail, “Arabic Sign Language Recognition using Lightweight CNN-based Architecture,” International Journal of Advanced Computer Science and Applications, vol. 13, no. 4, pp. 319–328, 2022, doi: 10.14569/IJACSA.2022.0130438.

B. Svendsen and S. Kadry, “Comparative Analysis of Image Classification Models for Norwegian Sign Language Recognition,” Technologies (Basel), vol. 11, no. 4, Aug. 2023, doi: 10.3390/technologies11040099.

I. Rodríguez-Moreno, J. M. Martínez-Otzeta, I. Goienetxea, and B. Sierra, “Sign language recognition by means of common spatial patterns: An analysis,” PLoS One, vol. 17, no. 10 October, Oct. 2022, doi: 10.1371/journal.pone.0276941.

T. J. Sánchez-Vicinaiz, E. Camacho-Pérez, A. A. Castillo-Atoche, M. Cruz-Fernandez, J. R. García-Martínez, and J. Rodríguez-Reséndiz, “MediaPipe Frame and Convolutional Neural Networks-Based Fingerspelling Detection in Mexican Sign Language,” Technologies (Basel), vol. 12, no. 8, Aug. 2024, doi: 10.3390/technologies12080124.

A. Kinanti and D. Maulana, “Convolutional Neural Network Implementation in BISINDO Alphabet Sign Language Recognition System Using Flask,” International Journal of New Media Technology), vol. 11, no. 1, p. 16, 2024.

Z. Alsaadi, E. Alshamani, M. Alrehaili, A. A. D. Alrashdi, S. Albelwi, and A. O. Elfaki, “A Real Time Arabic Sign Language Alphabets (ArSLA) Recognition Model Using Deep Learning Architecture,” Computers, vol. 11, no. 5, May 2022, doi: 10.3390/computers11050078.

F. S. Alamri, A. Rehman, S. B. Abdullahi, and T. Saba, “Intelligent Real-Life Key-Pixel Image Detection System for Early Arabic Sign Language Learners,” PeerJ Comput Sci, vol. 10, 2024, doi: 10.7717/PEERJ-CS.2063.

R. R. Agustin, H. Maulana, and E. P. Mandyartha, “Detection of Actions BISINDO (Indonesian Sign Language) into Text-to-Speech using Long Short-Term Memory with MediaPipe Holistics,” Jurnal Teknik Informatika (Jutif), vol. 5, no. 4, pp. 1051–1061, Nov. 2023, doi: 10.52436/1.jutif.2024.5.4.1492.

R. Cahuantzi, X. Chen, and S. Güttel, “A Comparison of LSTM and GRU Networks for Learning Symbolic Sequences,” in Intelligent Computing, K. Arai, Ed., Cham: Springer Nature Switzerland, 2023, pp. 771–785.

U. Farooq, M. S. Mohd Rahim, and A. Abid, “A multi-stack RNN-based neural machine translation model for English to Pakistan sign language translation,” Neural Comput Appl, vol. 35, no. 18, pp. 13225–13238, Jun. 2023, doi: 10.1007/s00521-023-08424-0.

Noura Alshareef, Rema Abobake, and Asma Abd Aljalil, “Arabic Sign Language Recognition in Real Time Using Transfer Deep Learning,” AlQalam Journal of Medical and Applied Sciences, pp. 730–739, Sep. 2024, doi: 10.54361/ajmas.247338.

A. M. J. AL Moustafa et al., “Integrated Mediapipe with a CNN Model for Arabic Sign Language Recognition,” Journal of Electrical and Computer Engineering, vol. 2023, pp. 1–15, Aug. 2023, doi: 10.1155/2023/8870750.

M. Mustafa, “Retraction Note to: A study on Arabic sign language recognition for differently abled using advanced machine learning classifiers,” J Ambient Intell Humaniz Comput, vol. 14, no. 1, p. 381, 2023, doi: 10.1007/s12652-022-04142-y.

R. Subandi, . H., and A. Yudhana, “Pneumonia Medical Image Classification Using Convolution Neural Network Model AlexNet and GoogleNet,” International Journal of Computing and Digital Systems, vol. 16, no. 1, pp. 1675–1684, Oct. 2024, doi: 10.12785/ijcds/1601124.

X. Wu et al., “CTransCNN: Combining transformer and CNN in multilabel medical image classification,” Knowl Based Syst, vol. 281, Dec. 2023, doi: 10.1016/j.knosys.2023.111030.

S. Saifullah et al., “Nondestructive Chicken Egg Fertility Detection Using CNN-Transfer Learning Algorithms,” Jurnal Ilmiah Teknik Elektro Komputer dan Informatika (JITEKI), vol. 9, no. 3, pp. 854–871, 2023, doi: 10.26555/jiteki.v9i3.26722.

D. O. Melinte and L. Vladareanu, “Facial Expressions Recognition for Human–Robot Interaction Using Deep Convolutional Neural Networks with Rectified Adam Optimizer,” Sensors, vol. 20, no. 8, 2020, doi: 10.3390/s20082393.

P. Singh, M. Kansal, R. Singh, S. Kumar, and C. Sen, “A Hybrid Approach based on Haar Cascade, Softmax, and CNN for Human Face Recognition,” J Sci Ind Res (India), vol. 83, no. 4, pp. 414–423, Apr. 2024, doi: 10.56042/jsir.v83i4.3167.

M. T. Hoang, B. Yuen, X. Dong, T. Lu, R. Westendorp, and K. Reddy, “Recurrent Neural Networks for Accurate RSSI Indoor Localization,” IEEE Internet Things J, vol. 6, no. 6, pp. 10639–10651, 2019, doi: 10.1109/JIOT.2019.2940368.

G. H. Samaan et al., “MediaPipe’s Landmarks with RNN for Dynamic Sign Language Recognition,” Electronics (Switzerland), vol. 11, no. 19, Oct. 2022, doi: 10.3390/electronics11193228.

K. K. Podder et al., “Signer-Independent Arabic Sign Language Recognition System Using Deep Learning Model,” Sensors, vol. 23, no. 16, Aug. 2023, doi: 10.3390/s23167156.

W. N. Waluyo, R. Rizal Isnanto, and Adian Fatchur Rochim, “Comparison of Mycobacterium Tuberculosis Image Detection Accuracy Using CNN and Combination CNN-KNN,” Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), vol. 7, no. 1, pp. 80–87, Feb. 2023, doi: 10.29207/resti.v7i1.4626.

C. C. Wang, C. Te Chiu, and J. Y. Chang, “EfficientNet-eLite: Extremely Lightweight and Efficient CNN Models for Edge Devices by Network Candidate Search,” J Signal Process Syst, vol. 95, no. 5, pp. 657–669, May 2023, doi: 10.1007/s11265-022-01808-w.

D. Tresnawati, R. Algani, and S. Mubaraq, “The Introduction of Hijaiyah Letters in Sign Languages using Augmented Reality Technology,” Jurnal Teknik Informatika (Jutif), vol. 3, no. 4, pp. 907–913, Aug. 2022, doi: 10.20884/1.jutif.2022.3.4.368.

M. M. Taye, “Theoretical Understanding of Convolutional Neural Network: Concepts, Architectures, Applications, Future Directions,” Mar. 01, 2023, MDPI. doi: 10.3390/computation11030052.